# An AI Gateway Platform Built on Envoy's Proven Data Plane

> An AI gateway platform built on Envoy. Route, fail over, rate-limit, and trace LLM gateway traffic through one OpenAI-compatible endpoint. Change one base URL.

Source: <https://tetrate.io/ai-gateway>

Production-grade circuit breaking, weighted routing, and built-in OpenTelemetry tracing for your LLM gateway. Change one base URL to get started.

![AI Gateway configuration code example](https://tetrate.io/images/ai-gateway/hero/code-desktop.png)

Based on Envoy

## The infrastructure that runs production traffic at hyperscale, applied to LLM requests

Built on [the Agent Router project (formerly Envoy AI Gateway)](https://tetrate.io/envoy-ai-gateway) at the [Agentic AI Foundation (AAIF)](https://tetrate.io/blog/agent-router-joins-aaif). Tetrate co-created the project with Bloomberg. See [published performance benchmarks](https://tetrate.io/learn/ai/ai-gateway-benchmarks) for MCP overhead, control-plane scaling, and methodology.

- A C++ data plane that handles millions of requests per second without becoming a bottleneck at scale.
- Production-grade circuit breaking and outlier detection built for service-to-service traffic, applied to model providers.
- Run gateways fully managed by Tetrate, or self-host gateways in any cloud, region, on-premises site, or edge location. One management plane controls every gateway.
- Built-in OpenTelemetry emission at the proxy layer. Every request is a trace span with zero instrumentation in agent code.

## How the AI gateway evaluates, protects, and logs every LLM request

Every agent request passes through the same pipeline inside the AI gateway. Each stage runs in order and either returns a response or passes the request on. This pipeline is what teams mean by an [LLM gateway](https://tetrate.io/learn/ai/what-is-llm-gateway): a single control point between your agents and model providers. The independent Broadcom/VMware test of the Agent Router project (then named Envoy AI Gateway) measured [about 2 ms of data plane overhead per request](https://tetrate.io/learn/ai/ai-gateway-benchmarks).

### Identity Resolution

Look up the inbound API key to resolve owner, project, and team. All downstream stages use this context for quota checks, routing, and log attribution.

![Identity Resolution](https://tetrate.io/images/ai-gateway/differentiation/1-desktop.jpg)

### Policy + Rate Limit

Token-window rate limits run here. Buckets configured per tenant decrement from the token counts the gateway filter emits.

![Policy + Rate Limit](https://tetrate.io/images/ai-gateway/differentiation/2-desktop.jpg)

### Route Selection

The AI gateway extracts the model from the request, matches an AIGatewayRoute rule, and selects a backend: Anthropic, Bedrock, Vertex, OpenAI, Azure, your BYOK credentials, or self-hosted.

![Route Selection](https://tetrate.io/images/ai-gateway/differentiation/3-desktop.jpg)

### Forward and Failover

Send to the selected provider. On retry-eligible failures, such as configurable status codes (429, 500-503), connect failures, or timeouts, Envoy retries with exponential backoff, then shifts to the next-priority provider in the rule. The agent gets the backup provider's response.

![Forward and Failover](https://tetrate.io/images/ai-gateway/differentiation/4-desktop.jpg)

### Emit Log and Trace

Captures team, agent, model, provider, token counts, latency, retry count, and status. The log feeds spend rollups, dashboards, and your configured sink without touching agent code.

![Emit Log and Trace](https://tetrate.io/images/ai-gateway/differentiation/5-desktop.jpg)

## What each AI gateway stage gets you in practice

### Routing

Route through model aliases and routing rules. For ordered rules by tag or header, see [attribute-based routing](https://tetrate.io/smart-routing).

![AI gateway routing with model aliases and routing rules](https://tetrate.io/images/ai-gateway/benefits/routing-desktop.jpg)

### Failover

Automatic recovery from provider failures.

![AI gateway automatic failover across providers](https://tetrate.io/images/ai-gateway/benefits/failover-desktop.jpg)

### Logging

Every request produces a structured log record written to your configured sink.

![AI gateway structured request logging](https://tetrate.io/images/ai-gateway/benefits/logging-desktop.jpg)

### Token limits and cost budgets

Cap tokens inline and track spend against budgets.

![AI gateway token limit and cost budget controls](https://tetrate.io/images/ai-gateway/benefits/token-desktop.jpg)

### Traffic splitting

Distribute traffic across multiple models. Use smart routing to [canary a model](https://tetrate.io/smart-routing) on one cohort before moving everyone.

![AI gateway traffic splitting across models](https://tetrate.io/images/ai-gateway/benefits/traffic-desktop.jpg)

## Works with the AI stack you already run

Tetrate Agent Router Enterprise is provider-neutral, framework-agnostic, and OpenAI-compatible. It sits in front of your existing stack.

### Model providers

![Anthropic](https://tetrate.io/images/ar-product/stack/anthropic.svg)

![OpenAI](https://tetrate.io/images/ar-product/stack/openai.svg)

![Google Vertex AI](https://tetrate.io/images/ar-product/stack/vertex-ai.svg)

![AWS Bedrock AgentCore](https://tetrate.io/images/ar-product/stack/aws.svg)

![Azure OpenAI](https://tetrate.io/images/ar-product/stack/azure.svg)

![Mistral AI](https://tetrate.io/images/ar-product/stack/mistral.svg)

![Baseten](https://tetrate.io/images/ar-product/stack/baseten.svg)

Self-hosted
models

### Agent frameworks

![LangChain](https://tetrate.io/images/ar-product/stack/langchain.svg)

![LangGraph](https://tetrate.io/images/ar-product/stack/langgraph.svg)

![Claude Code](https://tetrate.io/images/ar-product/stack/claude-code.svg)

![Agentforce](https://tetrate.io/images/ar-product/stack/agentforce.svg)

![AutoGen](https://tetrate.io/images/ar-product/stack/autogen.svg)

![CrewAI](https://tetrate.io/images/ar-product/stack/crewai.svg)

![Pydantic AI](https://tetrate.io/images/ar-product/stack/pydantic-ai.svg)

![Claude Cowork](https://tetrate.io/images/ar-product/stack/claude-coworks.svg)

### Observability

![OpenTelemetry](https://tetrate.io/images/ar-product/stack/opentelemetry.svg)

Datadog

![Splunk](https://tetrate.io/images/ar-product/stack/splunk.svg)

![Grafana](https://tetrate.io/images/ar-product/stack/grafana.svg)

Any OTel-
compatible
stack

### Identity and access

![Okta](https://tetrate.io/images/ar-product/stack/okta.svg)

![Microsoft Entra ID](https://tetrate.io/images/ar-product/stack/entra.svg)

![Ping Identity](https://tetrate.io/images/ar-product/stack/ping-identity.svg)

Any OIDC/SAML
provider

Any SSO provider

## Compare alternatives

Evaluating AI gateways? Start with our evidence-based comparison hub, then read the head-to-head guides.

- [Best Enterprise AI Gateway 2026](https://tetrate.io/learn/ai/best-enterprise-ai-gateway)
- [Tetrate vs. LiteLLM](https://tetrate.io/litellm-alternative-agent-router-enterprise)
- [Tetrate vs. Portkey](https://tetrate.io/learn/ai/tetrate-vs-portkey)
- [Tetrate vs. Kong](https://tetrate.io/learn/ai/tetrate-vs-kong)
- [Tetrate vs. Bifrost](https://tetrate.io/learn/ai/tetrate-vs-bifrost)
- [Tetrate vs. Cloudflare](https://tetrate.io/learn/ai/tetrate-vs-cloudflare)
- [Tetrate vs. Azure API Management](https://tetrate.io/learn/ai/tetrate-vs-azure-api-management)
- [Tetrate vs. the Agent Router project](https://tetrate.io/learn/ai/tetrate-vs-envoy-ai-gateway)
- [Smart LLM routing for AI agents](https://tetrate.io/smart-routing)

## Frequently asked questions about AI gateways

### What does an AI gateway do?

An AI gateway sits between an organization's applications and its AI model providers, so every model request passes through one control point. At that point the gateway routes requests, fails over between providers, enforces token limits, and records a trace for each request. Agent Router Enterprise runs this gateway on Envoy and exposes the gateway as one OpenAI-compatible endpoint.

### How does an AI gateway handle a provider outage?

An AI gateway retries a failed request, then sends the request to the next provider configured for the same model. The calling agent gets its response from the backup provider. In Agent Router Enterprise, Envoy retries with exponential backoff on configurable status codes, connect failures, or timeouts before shifting the request to the next-priority provider.

### How is an AI gateway priced?

AI gateway vendors use different pricing models. Some vendors charge a share of the token spend they route, and others license the gateway itself. Pricing for Agent Router Enterprise depends on your deployment, and detailed quotes come through a [demo request](https://tetrate.io/contact/demo-request).

### Can an AI gateway run inside my own cloud?

Yes, when the gateway separates its data plane from its management plane. The data plane can then run in your own Kubernetes cluster, region, or on-premises site, so model traffic stays inside your network. Agent Router Enterprise manages all of its gateways from one Tetrate-hosted management plane, and you control which request data, such as prompts in logs, reaches that management plane.
