Background

This article will show how to use Apache SkyWalking with eBPF to make network troubleshooting easier in a service mesh environment.

Apache SkyWalking is an application performance monitor tool for distributed systems. It observes metrics, logs, traces, and events in the service mesh environment and uses that data to generate a dependency graph of your pods and services. This dependency graph can provide quick insights into your system, especially when there’s an issue.

However, when troubleshooting network issues in SkyWalking’s service topology, it is not always easy to pinpoint where the error actually is. There are two reasons for the difficulty:

• Traffic through the Envoy sidecar is not easy to observe. Data from Envoy’s Access Log Service (ALS) shows traffic between services (sidecar-to-sidecar), but not metrics on communication between the Envoy sidecar and the service it proxies. Without that information, it is more difficult to understand the impact of the sidecar.
• There is a lack of data from transport layer (OSI Layer 4) communication. Since services generally use application layer (OSI Layer 7) protocols such as HTTP, observability data is generally restricted to application layer communication. However, the root cause may actually be in the transport layer, which is typically opaque to observability tools.

Access to metrics from Envoy-to-service and transport layer communication can make it easier to diagnose service issues. To this end, SkyWalking needs to collect and analyze transport layer metrics between processes inside Kubernetes pods—a task well suited to eBPF. We investigated using eBPF for this purpose and present our results and a demo below.

Istio recently announced “ambient mesh”—an experimental, “sidecar-less” deployment model for Istio. We’ve written about sidecar vs. sidecar-less recently in the context of getting the most performance and resiliency out of the service mesh. In this article, we’ll present our take on ambient mesh in particular.

Deploying Kubernetes clusters across availability zones can offer significant reliability benefits, especially when you use Istio for application routing and load balancing. If you have built redundant failure domains in separate zones, the mesh can automatically shift traffic to another zone should one zone fail. Istio’s locality-aware load balancing can also help reduce latency and cross-zone traffic charges from your cloud provider by keeping traffic within the same zone as much as possible.

One of Istio’s core capabilities is to facilitate a zero trust network architecture by managing identity for services in the mesh. To retrieve valid certificates for mTLS communication in the mesh, individual workloads issue a certificate signing request (CSR) to istiod. Istiod, in turn, validates the request and uses a certificate authority (CA) to sign the CSR to generate the certificate. By default, Istio uses its own self-signed CA for this purpose, but best practice is to integrate Istio into your existing PKI by creating an intermediate CA for each Istio deployment.

Tetrate is excited to announce and welcome Brian Dussault to the team! Brian is joining as the Head of Engineering. He will lead and scale the Engineering organization owning TSB and open source initiatives that offer rich and highly performant solutions empowering multiple personas across the enterprise in their Service Mesh journey.