LLM Proxy: The Complete Guide to Proxies, Routers, and AI Gateways

Why your stack needs an LLM proxy server

An LLM proxy server gives every application one endpoint for routing, authentication, and traffic management. Applications stop calling each provider directly, so credentials and routing rules live in one place.

How an LLM API proxy works

An LLM API proxy authenticates each caller, picks a provider based on routing rules, forwards the request, and returns the response. Your application sends every request to the proxy endpoint, and the proxy calls OpenAI, Anthropic, or another provider on the application’s behalf. The application never needs to know which provider served the request.

Many proxies expose one endpoint in OpenAI-compatible format. An existing OpenAI SDK integration then moves to the proxy with a two-line change: the base URL and the API key.

Where a thin proxy breaks down

Once more than one team, model, or provider enters production, three problems show up that a thin proxy was never built to solve.

• The first failure point is availability: when a single frontier provider has an outage, agents across every team relying on it go down at once, because there is no failover or load balancing across providers. Anthropic has a bad hour. Every on-call gets paged.

• The second failure point is cost visibility: token spend is attributed to a single API key with no per-team, per-agent, or per-project breakdown. When a CFO asks which team spent a specific amount on tokens last month, no one can answer, because the proxy has no way to attribute spend at that granularity. AI cost visibility by team, user, and customer covers how that breakdown works at the gateway.

• The third failure point is onboarding: there is no standard on-ramp for a new developer to get model access. Teams set up their own keys, billing arrangements, and usage patterns, which leadership cannot see. A new developer’s first step is filing a ticket, because no self-serve, governed path exists. Identity-based developer onboarding is the pattern that removes that key sprawl.

LLM proxy vs. router vs. gateway: what’s the difference?

A proxy forwards requests, while a router selects which model or provider gets each request. A gateway adds policy enforcement, governance, and observability on top of routing, so the real difference is where policy lives. For the short definitions, see what is an LLM gateway and what is an AI gateway.

A proxy handles transport: provider abstraction, upstream credentials, connection management, and request forwarding. A router handles selection: choosing a model or provider based on cost, latency, availability, or capability. A gateway handles control: applying identity, access policies, budgets, rate limits, guardrails, and audit requirements.

Essential routing and authentication for LLM proxies

A basic proxy routes requests, authenticates callers, and manages API keys through a single endpoint that accepts requests in a standard format, usually OpenAI-compatible.

Provider key management means the proxy stores the actual OpenAI, Anthropic, or Google API keys, so application teams never handle provider credentials. Each team uses a proxy key. The proxy validates that key, checks which models the caller may access, and injects the matching provider key before forwarding the request.

Model aliases, routing rules, and traffic splitting let teams move between models without changing application code. A routing rule maps a model alias to one or more providers. Traffic splitting sends a share of requests to a different model or provider for testing or cost optimization. An application can ask for a model alias such as “default-chat”, and the routing rule decides which provider model serves the request.

AI gateways: standardizing model policy

A gateway enforces policy on live traffic. The mechanism is a policy layer that checks every request against cost, sovereignty, resilience, and risk rules. The layer enforces those rules before the request reaches the model.

• Cost policy means token limits enforced inline in the request path, with budgets tracking spend against a plan.

• Sovereignty policy means routing rules that keep requests from regulated workloads inside specific clouds, regions, or on-premises infrastructure.

• Resilience policy means automatic failover when a provider goes down.

• Risk policy means PII redaction, prompt controls, and other checks running in the data path.

A router can pick the cheapest model for a request. A gateway can also refuse that request when the calling team has reached its token limit.

How distributed AI gateways handle scale

Distributed AI gateways separate the management plane from the data plane. Data planes run close to traffic, while the management plane sets policy once for all of them. The management plane defines which teams can access which models, what budgets apply, which guardrails run, and which routing rules are active. Data planes enforce that policy on every call.

The service mesh pattern behind distributed gateways

Service meshes already separate the control plane from the data plane. In a service mesh such as Istio, Envoy proxies form the data plane and carry the traffic between services. The control plane configures those proxies.

Tetrate Agent Router Enterprise is built on the Agent Router project (formerly Envoy AI Gateway), an open source data plane at the Agentic AI Foundation (AAIF) that runs on the Envoy proxy. Agent Router Enterprise runs gateways in any cloud, region, on-premises site, or edge location, and manages all of the gateways from one management plane.

In Agent Router Enterprise, Envoy carries the traffic while AI-specific code runs in a separate process. If that process fails, the gateway blocks the request, so no request reaches a model without its policy checks.

Proxy deployment for data residency

A self-hosted data plane keeps prompts and data inside your network by running in your cloud, on-premises, or at the edge. The data plane runs in your Kubernetes clusters, such as Amazon EKS, Azure AKS, Google GKE, or OpenShift. The data plane opens one outbound HTTPS connection to the management plane. Nothing connects in, so you open no inbound firewall rules.

Tetrate hosts the management plane in every deployment, and can also run the data plane as a fully managed service. Agent Router Enterprise lets each customer store the full request, only counts and cost, or nothing. The customer can change that setting at any time.

Resilience when the management plane is unreachable

The management plane sits outside the data path. Each data plane keeps a local copy of the rules, so AI traffic keeps running when the management plane is unreachable.

During a management plane outage, rule changes pause. New models, keys, and policy changes wait until the management plane is back. Each data plane picks them up at its next successful check-in.

Five enterprise controls that separate proxies from gateways

Platform leaders can evaluate any proxy or gateway against five controls. Each control closes a gap that a thin proxy leaves open once more than one team, model, or provider enters production.

1. Managing multi-team AI permissions

Scoped access profiles, SSO login, and one API key per approved model set let access open widely without per-request review. Each team gets a profile that defines which models the team can call, what budgets apply, and which guardrails run.

In Agent Router Enterprise, the gateway checks every request against the caller’s profile before forwarding the request to the provider. A developer signs in with SSO, gets a scoped profile, and receives one API key that works for every approved model. The developer files no ticket and manages no per-provider keys. Applications can use short-lived tokens from your identity provider, such as Okta or Microsoft Entra ID, so there are no Agent Router Enterprise keys to manage for those applications.

2. Setting spend limits by group

In Agent Router Enterprise, token brokering enforces token limits per team or agent inline in the request path, so a looping agent stops as soon as it reaches its limit. Budgets update a few minutes behind live traffic. Use token limits when a team needs a hard cap, and use budgets to track spend against a plan.

Every token is attributed to a person, team, agent, or project. Finance can answer “who spent what” from showback and chargeback dashboards without waiting for provider billing reports.

3. Filtering PII from LLM requests

In Agent Router Enterprise, PII redaction runs inline in the data path. The gateway scans each request for PII and redacts a match before the request reaches the model. The same inspection can run on the response before it reaches the agent.

A guardrail rule defines what to detect, such as Social Security numbers, card numbers, or email addresses, and whether to redact or block a match. In monitor mode, the rule logs the match without changing the request, so teams can see what a rule would catch before it touches live traffic.

4. Tracking agent and model requests

Agent-aware tracing reconstructs what an agent did across multiple model calls and tool invocations. In Agent Router Enterprise, the gateway logs every request and attributes it to the calling agent. Tool calls get the same attribution and logging as model requests. The gateway exports traces over OpenTelemetry, so the traces reach the observability tools you already run. Each trace carries a correlation ID, which links the gateway log entry for a request to its trace. A team can reconstruct a multi-agent chain failure in minutes from one source, without opening separate provider dashboards.

5. Enforcing policy on agent tool calls

In Agent Router Enterprise, the MCP Gateway provides a curated, per-profile-authenticated tool catalog. Operators curate the catalog centrally, and developers assemble MCP profiles from it, so an agent can only reach the tools its profile includes. Tool calls go through the same identity, audit, and access controls as model traffic. Security can audit exactly what an agent reached and when. How to govern MCP server traffic with an MCP gateway covers the catalog, profiles, and audit pattern in more depth.

Step-up authorization, built with Ory, an open source identity and access management platform, lets routine agent actions proceed without friction. Higher-risk actions trigger human approval through your own identity system.

When a thin proxy is enough vs. when you need a full LLM proxy gateway

The threshold is the number of teams, models, and providers in production.

A thin proxy covers a single team running one model, within clear scope limits: no cross-team policy, no distributed deployment, and no agent tool governance. If one team’s agents call one provider, a proxy or router handles routing without the operational overhead of a distributed gateway.

Once the second team goes to production, the organization needs one enforcement point across teams, models, and providers. Consistent policy then replaces per-team arrangements, cost attribution happens by team, and agent governance covers tool calls as well as model calls.

Enterprise demand for this layer is growing. Gartner forecasts the market for securing AI will reach almost $4.8 billion in 2027, with AI gateway spending growing 70.9% that year, from $251 million to $429 million.

Migrating from home-grown proxies or LiteLLM

Migrating to Agent Router Enterprise requires no application rewrites. Teams move one app at a time while the existing proxy keeps running. For a full cutover checklist, see the LiteLLM migration guide.

Operational debt in custom AI routing

Home-grown gateways work. The cost is ongoing maintenance, with visibility limited to the traffic those gateways route. Keeping up with every provider change and running the gateway reliably across regions becomes a distributed systems problem for the platform team.

In one anonymized deployment at a large fintech company, consolidating home-grown gateways into one governed path widened model access beyond two frontier providers. The same move made AI ROI trackable for the first time, without application rewrites. Retiring those gateways freed the engineering time spent maintaining them.

LiteLLM bottlenecks across regions

LiteLLM keeps keys, teams, and budgets in one PostgreSQL database, while Redis shares the rate-limit counters across the instances in each region. In a multi-region setup, every region depends on that one database, or each region runs separate workers with their own keys and budgets. LiteLLM’s Global Control Plane, which needs a LiteLLM Enterprise license, is one admin screen that opens separate workers, so each change is still made worker by worker.

Agent Router Enterprise sets rules once in the management plane, and each data plane enforces them from its own local copy.

Migration with no application rewrites

Each app changes two settings: the base URL, keeping the /v1 path, and the API key. LiteLLM virtual keys cannot be read back out, so every app gets a new key: a short-lived token from your identity provider or an Agent Router Enterprise key. Any app can roll back by switching its base URL back.

A team that cannot change its apps yet can use a bridge. The team’s LiteLLM gateway points at Agent Router Enterprise and uses one key in place of the provider keys. Provider keys and team costs then move under central control before any app changes.

Conclusion

A proxy routes requests, while a gateway governs them. The difference is where policy lives, and whether governance covers agent tool calls or only model calls. The threshold is scale: a thin proxy works for one team on one model, while multiple teams need one enforcement point across every team, region, and provider. Evaluate any option against the five controls in this guide, and close the gaps before a second team reaches production.

Start a fast-track evaluation: first routed request under five minutes, nothing to install. Run the five controls from this guide against your own agent traffic in Agent Router Enterprise. For pricing details, request a quote.

FAQs

Is the management plane a single point of failure?

No. In Tetrate Agent Router Enterprise, the management plane sits outside the data path. Each data plane keeps a local copy of the rules, so AI traffic keeps running when the management plane is unreachable. During an outage, rule changes pause until the management plane is back.

Do I need to rewrite applications to use an LLM proxy gateway?

No. Agent Router Enterprise exposes an OpenAI-compatible endpoint, so each application changes two settings: the base URL and the API key.

What data does an LLM proxy capture from my requests?

It depends on the proxy. Agent Router Enterprise lets each customer store the full request, only counts and cost, or nothing. The customer can change that setting at any time. Many customers run the gateway inside their own cloud account, so prompts and responses never leave their network.

Can an LLM proxy work with self-hosted models?

Yes. Agent Router Enterprise routes to hosted provider APIs and to self-hosted or open-weight models you run yourself. Self-hosted models sit in the same model catalog as vendor models.

Key terms glossary

AI gateway: An AI gateway adds policy enforcement, governance, and observability on top of routing. For a team choosing a proxy or a gateway, the gateway is the option that enforces cost, access, and risk rules on every request.

Agent tracing: Agent tracing reconstructs what an agent did across multiple model calls and tool invocations by logging every request and attributing it to the calling agent. In Agent Router Enterprise, traces are exported over OpenTelemetry and carry a correlation ID that links each gateway log entry to its trace, so a team can reconstruct a multi-agent chain failure from one source without opening separate provider dashboards.

Control plane: In a service mesh, the control plane configures the proxies but does not carry traffic. Agent Router Enterprise calls its Tetrate-hosted configuration layer the management plane.

Data plane: The data plane is the proxy that sits in the request path and enforces policy on every call. Each data plane keeps a local copy of the rules, so traffic keeps running when the management plane is unreachable.

Frontier provider: A frontier provider is one of the small set of leading model providers, such as OpenAI, Anthropic, or Google. Failover across providers keeps agents running when a single frontier provider has an outage.

LLM proxy: An LLM proxy forwards requests from applications to model providers through one endpoint. A proxy suits a single team on a single model, where no cross-team policy is needed.

LLM router: An LLM router selects which model or provider gets each request, based on cost, latency, availability, or capability. A router applies model-level rules but has no policy layer for cost, access, or risk.

Management plane: The management plane is where administrators define policy: which teams can access which models, what budgets apply, and which guardrails run. Each data plane pulls that configuration at its next check-in.

Model Context Protocol (MCP): MCP is an open protocol that connects LLM applications to external data sources and tools. It gives applications a standard way to share context with language models and expose tools to AI systems.

Personally identifiable information (PII): PII is data that can identify a specific person, such as a Social Security number, credit card number, or email address. In an LLM gateway, a guardrail rule defines which PII types to detect and whether to redact or block a match.

Single sign-on (SSO): SSO lets a person sign in once through the company identity provider, such as Okta or Microsoft Entra ID, to reach many applications. In Agent Router Enterprise, SSO login gives a developer a scoped profile with one API key for every approved model, so the developer files no ticket.

Step-up authorization: Step-up authorization adds a stronger check before a higher-risk action goes ahead. It builds on step-up authentication, which asks for a stronger or more recent sign-in. In Agent Router Enterprise, step-up authorization (built with Ory) sends higher-risk agent actions to a human for approval through your own identity system.

Token brokering: Token brokering is how Agent Router Enterprise enforces token limits per team or agent inline in the request path. A looping agent stops as soon as it reaches its limit.