In the early stages of the Istio project, configurations were pushed to Envoy proxies using the global State of the World (SotW) method. Whenever a service changed, the global configuration had to be pushed to all sidecars, causing significant network load and performance loss on the control plane. The Istio community began developing Incremental xDS a few years ago to address this issue and has supported Incremental xDS in recent Istio versions. In the recent Istio 1.22 release, Incremental xDS—also known as “Delta xDS”—has become the default. This article will introduce you to xDS, Incremental xDS, and the method of configuration distribution in Istio.
What Is xDS?
xDS (Extensible Discovery Service) is a communication protocol used for managing service discovery and dynamic configuration in a microservices architecture. This mechanism is widely used in Envoy proxies and Istio service meshes to manage various types of resource configurations, such as routing, service discovery, load balancing settings, etc.
What Discovery Services are Included in xDS?
xDS includes the following main discovery services, each responsible for different types of network resource configurations:
- LDS (Listener Discovery Service): Manages the configuration of Envoy listeners, which defines how to receive and handle inbound connections.
- RDS (Route Discovery Service): Provides routing information, defining how to route requests to different services based on specified rules.
- CDS (Cluster Discovery Service): Manages cluster information, where a cluster represents a group of logically similar backend service instances.
- EDS (Endpoint Discovery Service): Provides the network addresses of specific service instances that make up the clusters defined in CDS.
- SDS (Secret Discovery Service): Manages security-related configurations, such as TLS certificates and private keys.
- VHDS (Virtual Host Discovery Service): Provides virtual host configurations for RDS, allowing dynamic updates of virtual hosts without restarting connections.
- SRDS (Scoped Route Discovery Service): Manages routing scopes, providing dynamic route selection based on different conditions (such as request headers).
- RTDS (Runtime Discovery Service): Provides runtime configurations, which can be used for experimental features or fine-tuning system behavior.
These services collectively support the distribution and update of dynamic configurations, enabling Envoy-based application architectures to adapt in real time to changes, enhancing scalability and flexibility. Each service can be implemented independently or managed collectively through an aggregated approach (such as ADS). CNCF has also established an xDS API working group to promote the xDS API as the de facto standard for L4/L7 data plane configuration, similar to the role OpenFlow plays in L2/L3/L4 in SDN.
For a detailed introduction to the xDS protocol, such as xDS RPC services and variant methods, as well as the xDS request process, please refer to the Envoy proxy documentation.
Variants of the xDS Protocol
The xDS protocol primarily includes the following variants:
- State of the World (SotW): A separate gRPC stream provides complete data for each type of resource, typically used during the initial startup of an Envoy proxy, and was the first type of xDS protocol used by Istio.
- Incremental xDS (Delta xDS): Provides only the changed parts of the data for each type of resource, developed starting in 2021 and enabled by default in Istio 1.22.
- Aggregated Discovery Service (ADS): A single gRPC stream aggregates all types of resource data.
- Incremental ADS (Delta ADS): A single gRPC stream aggregates all types of incremental data.
The table below summarizes the four variants of the xDS protocol, including explanations, usage scenarios, and a comparison of their advantages and disadvantages. These variants provide multiple options for different network environments and service needs, allowing you to choose the most suitable protocol variant to optimize service performance and resource use.
| Variant Type | Explanation | Usage Scenario | Advantages | Disadvantages |
|---|---|---|---|---|
| SotW | Sends all configuration data each time, regardless of changes. | Suitable for stable environments with little configuration change. | Simple to implement, easy to understand, and maintain. | Large data transmission, is not suitable for environments with frequent configuration updates. |
| Delta xDS | Transmits only changed configuration data, not all data. | Suitable for environments with frequent changes needing a quick response to updates. | Reduces unnecessary data transmission, increasing efficiency. | Complex implementation requires the client and server to manage configuration states. |
| ADS | Manages all configuration data through a single gRPC stream, eliminating the need for separate connections for each resource type. | Suitable for complex service architectures that need to manage multiple types of resources simultaneously. | Reduces the number of network connections, simplifying resource management. | In cases of poor network or service quality, a single point of failure could cause all configuration updates to fail. |
| Delta ADS | Combines the advantages of ADS and Incremental xDS, aggregating and transmitting only changes in resources through a single gRPC stream. | Suitable for highly dynamic environments that need to manage multiple types of resources and require frequent updates. | Provides maximum flexibility and efficiency, suitable for large-scale and highly dynamic service architectures. | Most complex implementations have high requirements for managing configuration logic and precise control over resource changes and transmission. |
Service meshes using the xDS protocol can more flexibly manage communication and configurations between microservices, reducing the latency of configuration changes and improving system response speed and reliability.
In Istio, the DiscoveryServer acts as the implementation of Envoy’s xDS API, responsible for listening to the gRPC interface and dynamically pushing configurations according to Envoy’s needs. It can handle requests for various resource types and update Envoy configurations in real time based on service changes. Additionally, it supports security features, such as verifying client certificates, and ensuring only legitimate service instances can receive configuration data.
Configuration Examples for xDS Variants
Configuring xDS variants typically involves specifying the xDS server details in the configuration of the Envoy proxy or a similar service mesh. While the configuration details of different service meshes and proxy servers may vary, here are some common YAML configuration examples that illustrate how to specify xDS servers and how to use these protocol variants.
State of the World (SotW)
In Envoy’s configuration, you can use SotW either through static resources or by dynamically obtaining resources via API. Here’s a simple Envoy configuration example showing how to statically define clusters and listeners:
static_resources:
listeners:
- address:
socket_address: { address: 0.0.0.0, port_value: 80 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
stat_prefix: ingress_http
codec_type: auto
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/" }
route: { cluster: local_service }
http_filters:
- name: envoy.router
clusters:
- name: local_service
connect_timeout: 0.25s
type: STATIC
lb_policy: ROUND_ROBIN
hosts: [{ socket_address: { address: 127.0.0.1, port_value: 80 } }]
Incremental xDS
Incremental xDS configuration requires the xDS server to support the incremental protocol, and the client configuration must specify the use of Incremental xDS. The Envoy startup configuration needs to add an API version to enable Incremental xDS:
dynamic_resources:
cds_config:
api_config_source:
api_type: DELTA_GRPC
grpc_services:
envoy_grpc:
cluster_name: xds_cluster
lds_config:
api_config_source:
api_type: DELTA_GRPC
grpc_services:
envoy_grpc:
cluster_name: xds_cluster
Aggregated Discovery Service (ADS)
When using ADS, the configuration of all resource types is aggregated through a single API endpoint. This is specified in the Envoy configuration:
dynamic_resources:
cds_config:
ads: {}
lds_config:
ads: {}
ads_config:
api_type: GRPC
grpc_services:
envoy_grpc:
cluster_name: xds_cluster
Incremental ADS
Incremental ADS achieves more fine-grained updates by specifying the incremental API type in the ADS configuration:
dynamic_resources:
cds_config:
ads: {}
lds_config:
ads: {}
ads_config:
api_type: GRPC
grpc_services:
envoy_grpc:
cluster_name: xds_cluster
These configuration examples need to be adjusted according to your specific environment and requirements. For more details and advanced configurations, you can refer to the Envoy documentation.
How Does Istio Send Configurations to Envoy Sidecars?
Thanks to the xDS protocol, tools like Istio and Envoy Gateway can dynamically distribute configurations to Envoy proxies via API. The diagram below shows the configuration distribution process in Istio (Sidecar mode).
The main steps of the configuration distribution process in Istio are as follows:
- Declarative Configuration: Users define the service mesh’s configuration using YAML files or other configuration management tools. These configurations can include routing rules, security policies, telemetry settings, etc.
- Kubernetes: Istio configuration files are submitted to the Kubernetes cluster, usually through the kubectl apply command or other CI/CD tools. Kubernetes receives the configuration files and stores them in the etcd database.
- Istiod: Istiod is the control plane component of Istio, responsible for managing and distributing configurations. It listens for events coming from the Kubernetes API server, obtains relevant configuration changes, and processes them. Istiod parses the configuration files, generates corresponding routing rules and policies, and distributes these configurations to the data plane (Envoy proxies) via the xDS API.
- xDS API: Istiod uses the xDS API to send configurations to each Envoy proxy.
- Envoy Proxy: Envoy is the data plane component of Istio, running in a sidecar container alongside each service, intercepting and managing all inbound and outbound traffic. Envoy proxies receive configurations from Istiod via the xDS API and manage traffic, enforce policies, and collect telemetry data based on these configurations.
- Pod: Each service instance runs in a Pod, which contains an application container and an Envoy proxy container. The Envoy proxy intercepts all network traffic to and from the application container and processes it according to the configurations.
This configuration distribution process ensures that Istio can dynamically manage and configure all service instances in the service mesh, providing consistent traffic management and policy enforcement.
The Evolution of xDS and the Implementation of Delta xDS in Istio
Initially, xDS adopted a “global state” (State of the World, abbreviated as SotW) design, which meant that any configuration change required sending the complete state of all configurations to Envoy. This approach created a huge burden on the network and control plane, especially in large-scale service deployments.
At EnvoyCon 2021, Aditya Prerepa and John Howard shared How Istio Implements Delta xDS, an incremental implementation of xDS. Compared to the traditional SotW xDS, Delta xDS sends only the changed configurations, significantly reducing the amount of configuration data that needs to be sent over the network, thus improving efficiency and performance. This method is particularly suitable for environments where configurations change frequently, as it updates only the changed parts rather than the entire configuration.
During the implementation of Delta xDS, the Istio team faced several challenges, including ensuring the correctness of configuration updates and avoiding potential resource leaks. They addressed these challenges by adopting a Dry-run mode to run the SotW and Delta generators in parallel, gradually identifying and fixing flaws in the implementation. Additionally, they introduced new Envoy types, such as the Virtual Host Discovery Service, to support more fine-grained configuration distribution.
Delta xDS Incremental Configuration
The diagram below illustrates the process of Delta xDS incremental configuration.
The Delta xDS configuration process is as follows:
- Initial Complete Configuration: The control plane sends the initial complete configuration to the proxy, using the StoW mode at this time.
- Subscribe to Configuration Changes: The proxy subscribes to configuration changes from the control plane.
- Check Configuration Changes: The control plane checks for configuration changes relative to the proxy’s known state.
- Calculate Differences: The control plane calculates the differences (increments) between the current configuration and the previous configuration held by the proxy.
- Send Only Differences: The control plane sends only the changed configuration (differences) to the proxy, which applies these incremental updates to its configuration.
This process ensures that only necessary changes are transmitted and applied, improving efficiency and reducing the load on network and proxy resources.
SotW vs Delta xDS
While Delta xDS solves the performance issues of configuration distribution in large-scale networks, the SotW mode still has its place, such as in the initial configuration delivery. The table below compares the two configuration distribution methods in Istio: SotW (State of the World) and Delta xDS.
| SotW | Delta XDS | |
|---|---|---|
| Data Volume | Transmits complete configuration data each time, regardless of whether there are changes. | Transmits only the changed configuration data, reducing data volume. |
| Efficiency | Acceptable efficiency in small or less changing environments. | Higher efficiency in large or frequently changing environments. |
| Complexity | Simple implementation, easy to understand and maintain. | More complex implementation requires fine tracking and management of changes. |
| Resource Usage | This may increase server and network load due to repeatedly sending large amounts of unchanged data. | Lower resource usage, as only changed parts are handled. |
| Timeliness | High immediacy as the full configuration is sent immediately after updates. | Faster response by sending only changed parts, reducing processing time. |
| Applicability | Suitable for small to medium deployments with infrequent configuration changes. | Suitable for environments with frequent configuration changes or large-scale deployments. |
This table compares SotW and Delta XDS from multiple perspectives, including data volume, efficiency, complexity, resource usage, timeliness, and applicability, helping you make the appropriate choice in different usage environments.
Conclusion
In this article, I shared the components of xDS and the process of configuration distribution in Istio, as well as the two modes of xDS, SotW and Delta xDS. With Delta xDS becoming the default configuration in Istio version 1.22, this will help users easily use Istio in large-scale network environments.
References
###
If you’re new to service mesh, Tetrate has a bunch of free online courses available at Tetrate Academy that will quickly get you up to speed with Istio and Envoy.
Are you using Kubernetes? Tetrate Enterprise Gateway for Envoy (TEG) is the easiest way to get started with Envoy Gateway for production use cases. Get the power of Envoy Proxy in an easy-to-consume package managed by the Kubernetes Gateway API. Learn more ›
Getting started with Istio? If you’re looking for the surest way to get to production with Istio, check out Tetrate Istio Subscription. Tetrate Istio Subscription has everything you need to run Istio and Envoy in highly regulated and mission-critical production environments. It includes Tetrate Istio Distro, a 100% upstream distribution of Istio and Envoy that is FIPS-verified and FedRAMP ready. For teams requiring open source Istio and Envoy without proprietary vendor dependencies, Tetrate offers the ONLY 100% upstream Istio enterprise support offering.
Need global visibility for Istio? TIS+ is a hosted Day 2 operations solution for Istio designed to simplify and enhance the workflows of platform and support teams. Key features include: a global service dashboard, multi-cluster visibility, service topology visualization, and workspace-based access control.
Get a Demo