Observe Service Mesh with SkyWalking and Envoy Access Log Service
If you are looking for a more efficient solution to observe your service mesh instead of using a Mixer-based solution, this is exactly what you need.
If you are looking for a more efficient solution to observe your service mesh instead of using a Mixer-based solution, this is exactly what you need.
Apache SkyWalking, the observability platform, and open-source application performance monitor (APM) project, today announced the general availability of its 8.2 release. The release extends Apache SkyWalking’s functionalities and monitoring boundary to the browser side.
One of the most repeated pieces of advice for anyone getting started with microservices is to make sure you can see everything that’s going on inside your services. Leverage the power of observability. However, observability is a loaded term – so it’s valuable to understand what that terms mean, and what’s involved.
SkyWalking, a top-level Apache project, is the open source APM and observability analysis platform that is solving the problems of 21st-century systems that are increasingly large, distributed, and heterogenous. It’s built for the struggles system admins face today: To identify and locate needles in a haystack of interdependent services, to get apples-to-apples metrics across polyglot apps, and to get a complete and meaningful view of performance.
SkyWalking is a holistic platform that can observe microservices on or off a mesh, and can provide consistent monitoring with a lightweight payload.
Let’s take a look at how SkyWalking evolved to address the problem of observability at scale, and grew from a pure tracing system to a feature-rich observability platform that is now used to analyze deployments that collect tens of billions of traces per day.
The way that observability metrics are created, exchanged, and scraped has changed for Istio versions Istio 1.4 and up.
Here is how I configure Prometheus-Operator resources to scrape metrics from Istio 1.6 and install the latest Grafana Dashboards.
Prometheus-Operator is far more dynamic than the default Prometheus install. It adds some CRD to dynamically and transparently re-configure your Prometheus cluster.
A ServiceMonitor is a resource describing which pods to scrape based on a Service.
In Istio 1.6+ we have two types of things to monitor: Istio control-plane resources and Istio-proxy data-plane.
For that we create 2 different ServiceMonitor resources:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: prometheus-oper-istio-controlplane
labels:
release: prometheus
spec:
jobLabel: istio
selector:
matchExpressions:
- {key: istio, operator: In, values: [mixer,pilot,galley,citadel,sidecar-injector]}
namespaceSelector:
any: true
endpoints:
- port: http-monitoring
interval: 15s
- port: http-policy-monitoring
interval: 15s
If you know a bit of Prometheus, this is pretty easy to read:
istio
equals to mixer
, pilot
…http-monitoring
and http-policy-monitoring
every 15sThe only thing to be careful about are the labels
at the beginning: they are selectors that MUST match the Prometheus install serviceMonitorSelector
. If you fail to do so, Prometheus will not consider this resource.
You can check how yours is configured by looking at the prometheus
resource:
kubectl get prometheus -o yaml | grep -A4 serviceMonitorSelector serviceMonitorSelector: matchLabels: release: prometheus
In my case, it is release: prometheus
As you can see from my example, this Prom Operator was installed using Helm. I know… sorry…
The Data-Plane resource is quite the same but is targeting all the Istio-Proxy containers and adds some relabeling:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: prometheus-oper-istio-dataplane
labels:
monitoring: istio-dataplane
release: prometheus
spec:
selector:
matchExpressions:
- {key: istio-prometheus-ignore, operator: DoesNotExist}
namespaceSelector:
any: true
jobLabel: envoy-stats
endpoints:
- path: /stats/prometheus
targetPort: http-envoy-prom
interval: 15s
relabelings:
- sourceLabels: [__meta_kubernetes_pod_container_port_name]
action: keep
regex: '.*-envoy-prom'
- action: labelmap
regex: "__meta_kubernetes_pod_label_(.+)"
- sourceLabels: [__meta_kubernetes_namespace]
action: replace
targetLabel: namespace
- sourceLabels: [__meta_kubernetes_pod_name]
action: replace
targetLabel: pod_name
Again, pure Prom config. Just make sure you have the right label so the Operator will take care of the resource.
Add a label istio-prometheus-ignore=”true”
to your deployments in case you don’t want Prometheus to scrape the proxy’s metrics.
After few seconds for the whole thing to settle, you can connect to your Prom frontend, using Port-Forward on port 9090 or using the Istio Ingress-Gateway that you configured with SSL cert using SDS (check my older posts).
Now that you have Istio Telemetry V2 into your Prometheus cluster, you maybe want to see the graphs with Grafana.
Glad you’ve read this far. I know this blog is missing some pictures and colors… but who cares, we are engineers right?
Istio Dashboards for Grafana are stored in many places. You can find the latest in the Istio Github repo, but the best solution for you is to grab the one that matches your Istio install from the Istio install zip (or tar) where you grabbed istioctl
!
From Istio docs, get it with:
curl -L https://istio.io/downloadIstio | sh -
This will create a folder with all the Istio stuffs. Note that Addons (Grafana, Kiali, Prometheus..) will NOT be managed by istioctl
quite soon. You can find all the deployment scripts in this folder.
Dashboards are also located in this folder (istio-1.6.7
as the time of this writing) at manifests/charts/istio-telemetry/grafana/dashboards/
For them to be used by Grafana (the one installed by Prom Operator), you need to copy them inside a secret. Here’s the script I use for that (do a cd istio-<your-version>
before using it):
#!/bin/bash # go into the dashboards folder pushd manifests/charts/istio-telemetry/grafana/dashboards # create the basic command to create the configmap ISTIO_DASHBOARD_SECRET="kubectl -n monitoring create cm prometheus-oper-istio-dashboards " # append each file to the secret for i in *.json ; do echo $i ISTIO_DASHBOARD_SECRET="${ISTIO_DASHBOARD_SECRET} --from-file=${i}=${i}" done # run the secret creation command eval $ISTIO_DASHBOARD_SECRET # label the configmap so it is used by Grafana kubectl label -n monitoring --overwrite cm prometheus-oper-istio-dashboards grafana_dashboard=1 popd
Restart the Grafana pod and you should see the Dashboards in Grafana:
Apache SkyWalking, the observability platform, and open-source application performance monitor (APM) project, today announced the general availability of its 8.1 release that extends its functionalities and provides a transport layer to maintain the lightweight of the platform that observes data continuously.
Starting with release 1.15.0 Envoy proxy supports decoding of Postgres messages for statistics purposes. This feature allows for an aggregated view of the types of Postgres transactions happening in the network. That aggregated view instantly provides a breakdown of types of Postgres operations and the number and severity of errors. Presented in a time series format allows for a clear overview of how the error rate of composition of queries changed over time.
Apache SkyWalking, the observability platform and open source application performance monitor (APM) project, today announced the general availability of its 8.0 release. Known for its powerful metrics, tracing and service mesh capabilities, SkyWalking extends its functionality with the new release by addressing user demand for metrics integration with other metrics collection systems, including Prometheus.
This post originally appears on The New Stack
This post introduces a way to automatically profile code in production with Apache SkyWalking. We believe the profile method helps reduce maintenance and overhead while increasing the precision in root cause analysis.