# Kubernetes Insights

Cloud Ctrl supports Kubernetes cost insights. This is achieved by extracting metrics from the cluster and exporting them to Cloud Ctrl.

Cloud Ctrl will allocate a cost to each pod based on its CPU and memory usage on a particular day. The namespace and any labels from the pod you wish to include in Cloud Ctrl will be tagged on the cost, allowing you to report on spend by namespace, and labels.

# Metrics Agent

Deploy the Cloud Ctrl metrics agent to your Kubernetes cluster(s):

Create a file agent-cronjob.yaml with these contents:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: metrics-exporter-job
spec:
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1
  concurrencyPolicy: Forbid
  schedule: "0 */8 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: metrics-exporter
              image: cloudctrlofficial/metrics-exporter:latest
              imagePullPolicy: Always
              env:
                - name: "CloudCtrl__ClusterName"
                  value: "<K8s_CLUSTER_NAME_HERE>"
                - name: "CloudCtrl__ApiKey"
                  value: "<CC_API_KEY>"
                - name: "CloudCtrl__TenantId"
                  value: "<CC_TENANT_ID>"
                - name: "CloudCtrl__CloudAccountId"
                  value: "<CC_CLOUD_ACCOUNT_ID>"
                - name: "Metrics__Host"
                  value: "prometheus-kube-prometheus-prometheus"
                - name: "Metrics__Port"
                  value: "9090"
          restartPolicy: OnFailure

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

After you have supplied values for the environment variables, run kubectl apply -f agent-cronjob.yaml

The agent runs within a CronJob in the default namespace. It runs 3 times a day to support resiliency to failures. You can change the schedule and namespace, but make sure the agent runs at least once a day.

# Environment Variables

CloudCtrl__ClusterName identifies which cluster the agent is running in. Use the cluster name as it appears in the cloud provider panel, just make sure it's unique for each cluster where you run the agent.

CloudCtrl_ApiKey Generate a Cloud Ctrl API key: instructions. Take note of how to find your Tenant Id and Cloud Account Id from those same instructions and enter them into the following variables:

CloudCtrl__TenantId
CloudCtrl__CloudAccountId

Metrics__Host is the Prometheus API service created when you deploy the kube-prometheus-stack (see below). If the agent and Prometheus stack are not running in the same namespace you will need to add the namespace to the end of the host. For example if Prometheus was running in a monitoring namespace you would use "prometheus-kube-prometheus-prometheus.monitoring"

The name of the service will depend on the name you gave the Prometheus stack when you deployed it. The default name is prometheus-kube-prometheus-prometheus. To verify the name of the service run kubectl get svc -n <namespace>, and check the name of the Prometheus service that runs on port 9090.

e.g. prometheus-operator-kube-p-prometheus ClusterIP 10.100.63.189 <none> 9090/TCP,8080/TCP 6d16h

Metrics_Port is the port of the Prometheus API service. The default port is 9090.

# Kubernetes Prometheus Stack

The agent queries the Prometheus API to extract various Kubernetes metrics. Deploy the kube-prometheus-stack (opens new window) to your cluster(s).

Create a values.yaml and paste the following contents:

Azure

# values.yaml
kube-state-metrics:
  metricLabelsAllowlist:
    - pods=[app],nodes=[kubernetes.azure.com/agentpool,kubernetes.azure.com/cluster,kubernetes.azure.com/nodepool-type]

1
2
3
4

AWS

# values.yaml
kube-state-metrics:
  metricLabelsAllowlist:
    - pods=[app],nodes=[eks.amazonaws.com/nodegroup]

1
2
3
4

# Exposing labels to Prometheus

Cloud Ctrl depends on certain node labels to be able to allocate metrics to cloud usage. You need to configure the Prometheus stack to extract these labels from the nodes.

You can also specify what pod labels to export to Cloud Ctrl. This is recommended so that cost reports can be created using the deployment labels. Teams deploying to Kubernetes typically use their own labels to identify their apps so it makes sense to report costs using those same labels.

pods=[app] exports the app label to Prometheus. The agent will export any pod labels to Cloud Ctrl and associate them as regular tags to the cost. Multiple labels can be exported, by separating them with a comma: pods=[app,app.deployment.team].

# AWS customers: Enable tags in billing reports

AWS requires that tags are activated for billing reports. You will need to do this for the eks.amazonaws.com/nodegroup tag in the Billing console. See the AWS Setup Guide for how to enable tags in the billing reports.

# All customers: tagging your clusters

Recommendation

If you notice that some Kubernetes spend is not being allocated, try the following recommendation.

Often the cloud provider will deploy extra resources to support your cluster, and these extra resources only carry the user tags applied to the cluster.

Cloud Ctrl can allocate costs to these extra resources with no extra config. However, it's not guaranteed to find every cost.

To address this, you can add a user tag to your cluster that carries the name of the Kubernetes cluster:

K8s-Cluster-Name = your_cluster_name. The value for the tag is the same one you specified in CloudCtrl__ClusterName when you deployed the agent.

# Deploy Prometheus

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack -f values.yaml

1
2
3

If you already have a Prometheus stack deployed, you will need to ensure the metricLabelsAllowlist is configured with at least the node labels listed above. To upgrade the deployment run:

helm upgrade prometheus prometheus-community/kube-prometheus-stack -f values.yaml

← Amazon Cloud Ctrl API →