Announcing Tetrate Agent Operations Director for GenAI Runtime Visibility and Governance

Learn more
< Back

How to configure Prometheus-Operator & scrape metrics from Istio 1.6

How%20to%20configure%20Prometheus-Operator%20%26%20scrape%20metrics%20from%20Istio%201.6

How do you get Istio metrics in Prometheus

The way that observability metrics are created, exchanged, and scraped has changed for Istio versions Istio 1.4 and up.

Here is how I configure Prometheus-Operator resources to scrape metrics from Istio 1.6 and install the latest Grafana Dashboards.

Tetrate offers an enterprise-ready, 100% upstream distribution of Istio, Tetrate Istio Subscription (TIS). TIS is the easiest way to get started with Istio for production use cases. TIS+, a hosted Day 2 operations solution for Istio, adds a global service registry, unified Istio metrics dashboard, and self-service troubleshooting.

Learn more

Install Prometheus and create a ServiceMonitor for Prometheus-operator

Prometheus-Operator is far more dynamic than the default Prometheus install. It adds some CRD to dynamically and transparently re-configure your Prometheus cluster.

ServiceMonitor is a resource describing which pods to scrape based on a Service.

In Istio 1.6+ we have two types of things to monitor: Istio control-plane resources and Istio-proxy data-plane.

For that we create 2 different ServiceMonitor resources:

Configuring Prometheus operator to monitor Istio control plane

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-oper-istio-controlplane
  labels:
    release: prometheus
spec:
  jobLabel: istio
  selector:
    matchExpressions:
      - {key: istio, operator: In, values: [mixer,pilot,galley,citadel,sidecar-injector]}
  namespaceSelector:
    any: true
  endpoints:
  - port: http-monitoring
    interval: 15s
  - port: http-policy-monitoring
    interval: 15s

If you know a bit of Prometheus, this is pretty easy to read:

  • look for any service with label istio equals to mixer , pilot
  • scrape port named http-monitoring and http-policy-monitoring every 15s

The only thing to be careful about are the labels at the beginning: they are selectors that MUST match the Prometheus install serviceMonitorSelector. If you fail to do so, Prometheus will not consider this resource. You can check how yours is configured by looking at the prometheus resource:

kubectl get prometheus  -o yaml | grep -A4 serviceMonitorSelector
serviceMonitorSelector:
      matchLabels:
        release: prometheus

In my case, it is release: prometheus

As you can see from my example, this Prom Operator was installed using Helm. I know… sorry…

Configuring Prometheus operator to monitor Istio data plane

The Data-Plane resource is quite the same but is targeting all the Istio-Proxy containers and adds some relabeling:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-oper-istio-dataplane
  labels:
    monitoring: istio-dataplane
    release: prometheus
spec:
  selector:
    matchExpressions:
      - {key: istio-prometheus-ignore, operator: DoesNotExist}
  namespaceSelector:
    any: true
  jobLabel: envoy-stats
  endpoints:
  - path: /stats/prometheus
    targetPort: http-envoy-prom
    interval: 15s
    relabelings:
    - sourceLabels: [__meta_kubernetes_pod_container_port_name]
      action: keep
      regex: '.*-envoy-prom'
    - action: labelmap
      regex: "__meta_kubernetes_pod_label_(.+)"
    - sourceLabels: [__meta_kubernetes_namespace]
      action: replace
      targetLabel: namespace
    - sourceLabels: [__meta_kubernetes_pod_name]
      action: replace
      targetLabel: pod_name

Again, pure Prom config. Just make sure you have the right label so the Operator will take care of the resource.

Add a label istio-prometheus-ignore=”true” to your deployments in case you don’t want Prometheus to scrape the proxy’s metrics.

Service discovery result

After few seconds for the whole thing to settle, you can connect to your Prom frontend, using Port-Forward on port 9090 or using the Istio Ingress-Gateway that you configured with SSL cert using SDS (check my older posts).

How to enable Istio dashboard in Grafana

Now that you have Istio Telemetry V2 into your Prometheus cluster, you maybe want to see the graphs with Grafana.

Glad you’ve read this far. I know this blog is missing some pictures and colors… but who cares, we are engineers right?

Istio Dashboards for Grafana are stored in many places. You can find the latest in the Istio Github repo, but the best solution for you is to grab the one that matches your Istio install from the Istio install zip (or tar) where you grabbed istioctl !

From Istio docs, get it with:

curl -L [https://istio.io/downloadIstio](https://istio.io/downloadIstio) | sh -

This will create a folder with all the Istio stuffs. Note that Addons (Grafana, Kiali, Prometheus..) will NOT be managed by istioctl quite soon. You can find all the deployment scripts in this folder.

Dashboards are also located in this folder (istio-1.6.7 as the time of this writing) at manifests/charts/istio-telemetry/grafana/dashboards/

For them to be used by Grafana (the one installed by Prom Operator), you need to copy them inside a secret. Here’s the script I use for that (do a cd istio-<your-version>before using it):

#!/bin/bash

# go into the dashboards folder
pushd manifests/charts/istio-telemetry/grafana/dashboards

# create the basic command to create the configmap
ISTIO_DASHBOARD_SECRET="kubectl -n monitoring create cm prometheus-oper-istio-dashboards "

# append each file to the secret
for i in *.json ; do
  echo $i
  ISTIO_DASHBOARD_SECRET="${ISTIO_DASHBOARD_SECRET} --from-file=${i}=${i}"
done

# run the secret creation command
eval $ISTIO_DASHBOARD_SECRET

# label the configmap so it is used by Grafana
kubectl label -n monitoring --overwrite cm prometheus-oper-istio-dashboards grafana_dashboard=1

popd

Restart the Grafana pod and you should see the Dashboards in Grafana:

Product background Product background for tablets
New to service mesh?

Get up to speed with free online courses at Tetrate Academy and quickly learn Istio and Envoy.

Learn more
Using Kubernetes?

Tetrate Enterprise Gateway for Envoy (TEG) is the easiest way to get started with Envoy Gateway for production use cases. Get the power of Envoy Proxy in an easy-to-consume package managed via the Kubernetes Gateway API.

Learn more
Getting started with Istio?

Tetrate Istio Subscription (TIS) is the most reliable path to production, providing a complete solution for running Istio and Envoy securely in mission-critical environments. It includes:

  • Tetrate Istio Distro – A 100% upstream distribution of Istio and Envoy.
  • Compliance-ready – FIPS-verified and FedRAMP-ready for high-security needs.
  • Enterprise-grade support – The ONLY enterprise support for 100% upstream Istio, ensuring no vendor lock-in.
  • Learn more
    Need global visibility for Istio?

    TIS+ is a hosted Day 2 operations solution for Istio designed to streamline workflows for platform and support teams. It offers:

  • A global service dashboard
  • Multi-cluster visibility
  • Service topology visualization
  • Workspace-based access control
  • Learn more
    Decorative CTA background pattern background background
    Tetrate logo in the CTA section Tetrate logo in the CTA section for mobile

    Ready to enhance your
    network

    with more
    intelligence?