Envoy AI Gateway Reaches v1.0, Establishing the Open Source Standard for Enterprise AI Traffic

Envoy: The Standard for Internet-Scale Traffic

Created at Lyft in 2015, Envoy has been used in production for nearly a decade. By September 2016, Envoy was processing over 2 million requests per second across Lyft’s infrastructure, and today handles millions of machine learning predictions daily. Netflix processes billions of API requests daily, while Airbnb handles over 1 million user events per second, with both organizations relying on Envoy for mission-critical traffic management. At least 44% of enterprises now use Envoy in production environments or are evaluating Envoy for production, including AWS, Docker, SAP, Atlassian, and LY Corporation. Leading AI providers including Databricks, Google, Lyft, Netflix, and Spotify have documented their reliance on Envoy to manage AI inference traffic at scale.

Envoy AI Gateway extends this production-proven foundation with native support for agentic traffic. Where Envoy became the standard for internet traffic, Envoy AI Gateway is its modern adaptation for AI workloads, bringing the same reliability, extensibility and governance to a new class of infrastructure demands.

v1.0: A Major Milestone Backed by Industry Partners

The path to v1.0 began when Bloomberg proposed the effort through the Envoy Gateway community to address its own challenge: routing generative AI workloads with the same reliability and governance expected of Envoy. The project was first released in February 2025. Over the past 16 months, sustained collaboration among Tetrate, Bloomberg, Nutanix, and the open source community has produced a mature, stable codebase ready for broad adoption.

At Bloomberg, Envoy AI Gateway is in production, demonstrating that the same open source code available to the community is stable and scalable enough to power demanding enterprise workloads.

“We see the Envoy AI Gateway as a key element toward standardizing how enterprises securely and reliably serve AI workloads,” said Dan Sun, Envoy AI Gateway and KServe co-founder/maintainer. “Bloomberg engineers have made hundreds of contributions to this project — in the spirit of our firm’s commitment to scalable, open source AI infrastructure that brings vendor neutrality, consistency, observability and control to AI inference at scale.”

“Envoy has become the foundational layer for internet traffic at the world’s most demanding organizations. With v1.0, Envoy AI Gateway brings that same trust to AI workloads,” said Varun Talwar, co-founder and CTO at Tetrate. “The code in the public repo is the same code running in production at Bloomberg and Tetrate. That level of transparency is rare in open source, and it’s what enterprises need as they scale AI.”

Nutanix is taking Envoy AI Gateway into production, including bug fixing and more. The company is incorporating the project into its Nutanix Agent Gateway and Nutanix Enterprise AI solutions.

“Nutanix is proud to be a maintainer and an active contributor in the Envoy AI Gateway community, helping bring the same transparency and production-grade reliability that powers enterprise Internet traffic to the next wave of AI workloads,” said Debo Dutta, chief AI officer at Nutanix. “We are using the project’s capabilities to bring transparent, multiprovider flexibility and production-ready AI infrastructure to our customers.”

“At LY Corporation, we utilize the Envoy AI Gateway to manage our multi-tenant, self-hosted LLM traffic,” said Shingo Omura, principal architect of AI infrastructure at LY Corporation. “It provides a unified API for flexible routing, token-based rate limiting, authentication and authorization, monitoring and extensibility, aligned with open standards like the Kubernetes Gateway API Inference Extension. This allows us to maximize operational efficiency of our LLM platform. We believe version 1.0 is a major milestone for production-grade AI infrastructure engineering.”

New capabilities in v1.0 are made possible by open source community collaboration among engineers working across a diversity of organizations and use cases:

• Unified API across every major AI provider — A single OpenAI-compatible interface in front of OpenAI, Anthropic (direct, AWS Bedrock, and GCP Vertex), Google Gemini, Azure OpenAI, AWS Bedrock, and the long tail of OpenAI-compatible providers (Groq, Together, Mistral, Cohere, DeepSeek, SambaNova).
• Native MCP Gateway for the agentic era — Production-grade Model Context Protocol routing — server multiplexing behind a single endpoint, tool filtering with include/exclude rules, OAuth 2.0 with JWT claim forwarding to backends, CEL-based fine-grained authorization, and per-tool observability.
• Token-aware traffic management — Rate limiting, budgets, and quotas that understand AI workloads — separate cost attribution for input, output, cached, and reasoning tokens; route-scoped costs with fleet-wide defaults; quota-aware routing primitives (QuotaPolicy API) to route around rate-limited upstreams.
• Centralized upstream credential management — One control plane for every provider’s auth story — API keys for OpenAI, Anthropic, Azure; AWS SDK default credential chain (IRSA, EKS Pod Identity, instance profiles).
• AI-native observability — OpenInference distributed tracing and OpenTelemetry GenAI semantic conventions across chat, embeddings, image generation, audio, MCP, and reasoning endpoints.
• Stable v1 API surface — AIGatewayRoute, AIServiceBackend, BackendSecurityPolicy, GatewayConfig, MCPRoute, and MCPRouteSecurityPolicy graduate to v1 — production-stable CRDs hardened by Tetrate, Bloomberg, and Nutanix across real workloads.

The Envoy AI Gateway roadmap includes features that community members participating in the project are working on for the next version:

• Dollar-based control, not just tokens — Cost governance that moves beyond token counts to actual spend limits.
• Deeper MCP authorization and identity — Continued investment in agentic infrastructure, including backend security policy for MCP, support for OIDC token exchange when connecting to MCP backends and finer-grained policy across tools, resources and prompts.
• Closer alignment with the agentic AI ecosystem — Strengthening collaboration with frontier model providers and the broader agentic AI community to ensure Envoy AI Gateway remains the open standard for AI traffic.

Get Started, Get Involved

The Envoy AI Gateway community invites broader participation as the project enters its next phase of growth. Organizations are encouraged to contribute extensions, governance policies and new integrations.

• Visit the project website and download Envoy AI Gateway.
• Get started with the Envoy AI Gateway installation guide.
• Visit Bloomberg’s open source and open source projects pages.
• Visit Nutanix’s Agent Gateway blog.

About Envoy AI Gateway

Envoy AI Gateway extends CNCF’s Envoy Gateway with native support for generative AI traffic, offering routing, failover, observability and compliance features. The project is community-driven and open source under the CNCF umbrella. Learn more at aigateway.envoyproxy.io.

About Tetrate

Tetrate is a primary upstream contributor to the Envoy project and a leader in its technical development. Tetrate enables safe, fast and profitable AI transformation by providing trusted connectivity in AI operations. As the driving force behind Envoy and Envoy AI Gateway, Tetrate builds on proven open source foundations to deliver solutions fit for regulated industries and government organizations. Tetrate’s Agent Router Enterprise, built on Envoy AI Gateway, gives enterprises the standards, guardrails and evaluation metrics needed to run Envoy AI Gateway with full governance, failover and optimization capabilities. This is why some of the largest financial institutions, governments and other enterprises rely on Tetrate to deliver modern application networking and security. Learn more at tetrate.io.

MEDIA CONTACT

Cristin Connelly
Cathey.co for Tetrate
cristin@cathey.co