Announcing token brokering in Tetrate Agent Router Enterprise: enforce AI cost budgets in the request path
Cap AI agent costs at the AI gateway, without breaking developer workflows.
David Wang 
By David Wang, Head of Product, Tetrate
Read the announcement: Tetrate Adds Token Brokering Capability for AI Code Gen Cost Management.
A coding agent that used to make one model call now makes forty. The engineer who kicked it off sees one prompt and one answer. Finance sees the invoice at the end of the month. Cap the spend with a hard limit and the agent fails mid-task, so developers route around it or stop trusting the tool. Leave it uncapped and someone hand-approves models one request at a time, becoming the bottleneck between every developer and every endpoint.
Token brokering, new in Tetrate Agent Router Enterprise, removes that choice. A broker stands between parties whose interests pull apart, which is the situation exactly: the developer wants the strongest model available, and the company wants to stay inside the number it gave finance. Token brokering settles that trade on every request. It evaluates the request against the policy set once and routes to an approved model. When spend crosses a budget, it trips a circuit breaker and hands the request to an approved fallback model instead of returning an error.
How token brokering works on each request
Agent Router Enterprise is the AI gateway (also called an LLM gateway) sitting in the request path between the developers and agents sending requests and the models serving them. Every request triggers three policy decisions before a target is chosen.
| Policy decision | Question it answers | Example |
|---|---|---|
| Business | How much budget is left? | The team has $4,200 of a $5,000 monthly cap remaining |
| Technical | Which model and endpoint are healthy? | Serve the preferred model, switch to an alternative if it is unavailable |
| Governance | Where must this work stay, and which models are approved? | Keep this workload in the EU, approved in-region models only |
Agent Router Enterprise resolves those decisions against live state and enforces spend policy on every token in between.
One management plane holds the policy: the budget, the approved models, the regions, the fallback order. A distributed fleet of data planes enforces it, running wherever the traffic actually is, in a cloud VPC, on-premises, per region, at the edge. You set the number once, and every data plane enforces it on every request it sees.
Distributing enforcement is what makes a budget mean anything across a real fleet. A spend cap is only as good as the share of traffic it can see, and traffic that never passes through an enforcement point is traffic the budget does not govern.
Why an open-source proxy like LiteLLM stops scaling
Most teams manage early inference spend with an open-source proxy such as LiteLLM, then blame their own configuration when it falls short. The limit is architectural.
These proxies were designed for one team and a handful of models. The proxy runs as a single process with a fixed list of endpoints, and it can only meter the requests that pass through it. That holds until the models multiply. Once a company runs frontier, private, and edge models across several regions and providers, each proxy sees only its own traffic, so a budget set in one covers a fraction of the spend. Agents widen the gap, because a single agent request can fan out into dozens of calls, many of them to endpoints that proxy never sees.
Making the runtime faster solves the wrong problem. LiteLLM’s move to a Rust hot path lowers latency and memory, and the artifact it deploys is still a single binary holding one endpoint list, metering the traffic that passes through it. Ten regions running that binary means ten budgets that each see a tenth of the spend, and someone reconciling them by hand.
Distributed enforcement changes what an enforcement point is. Policy lives in one management plane and is pushed to a fleet of data planes, so adding a region adds an enforcement point rather than a budget to reconcile. Governing AI spend across regions and providers is a distributed systems problem, and it is answered by topology, in any language.
Two lines of setup, no code rewrite
Adding an AI gateway to your code is simple, and the benefits last. Point the SDK’s base URL at the proxy endpoint, the gateway’s OpenAI- and Anthropic-compatible base URL shown on the Console Dashboard, and pass an Agent Router key.
from openai import OpenAI
client = OpenAI(
- base_url="https://api.openai.com/v1",
+ base_url="https://AGENT_ROUTER_URL/v1", # agent router endpoint
- api_key="sk-...",
+ api_key="sk-...", # your agent router key
)
# unchanged
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello, world!"}],
)The rest of the code is untouched. An Anthropic application changes the base URL the same way, plus one line for auth, since the gateway takes the key as a bearer auth_token rather than the native x-api-key header.
from anthropic import Anthropic
client = Anthropic(
- base_url="https://api.anthropic.com",
+ base_url="https://AGENT_ROUTER_URL/v1", # agent router endpoint
- api_key="sk-ant-...",
+ auth_token="sk-...", # your agent router key
)
# unchanged
response = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=1024,
messages=[{"role": "user", "content": "Hello, world!"}],
)From there the gateway does the work you configured. It translates between providers, so a Chat Completions call can be served by Claude and returned in the format your code expects. And when a budget or a failed endpoint triggers a fallback, the request moves to the next approved model and comes back as a normal response, with no error path to handle. The model that actually served it shows up in Request Logs.
Setting AI budgets and cost policy
An administrator does the governance work once, in the Agent Router Enterprise Admin Dashboard or over its API. Provision the models and providers the platform offers, frontier and private, then create a budget: a monthly spend limit in US dollars that resets at the start of each calendar month in UTC.
A budget attaches to the identity that owns the API keys, so the owner decides what the ceiling covers. That is why the key-per-purpose convention matters. Give an agent its own key under its own service identity and the agent gets its own dollar cap.
| Budget scope | What the limit covers | When it fits |
|---|---|---|
| Whole team | One pooled limit across every member and every agent charged to the unit | A cost centre with a single ceiling agreed with finance |
| Each teammate | The same limit applied to every identity individually | A per-person allowance that holds regardless of what the rest of the team spends |
| One person | A limit on a single identity | An engineer, or an agent running under a dedicated service identity |
Then attach a fallback policy to the key. This is an ordered list of approved models, the primary at priority 0 and the alternatives beneath it. The gateway serves the primary under normal conditions and moves to the next model in line when a backend returns a recoverable error.
Token brokering extends that list to spend. Agent Router Enterprise meters tokens against the budget, and when a request would cross the cap, it trips a circuit breaker and moves the request to the next approved model. That model is usually a private one, which costs less and keeps inference inside the company’s own environment. The same ordered list that already handles a failed endpoint now handles a spent budget, so cost and availability run through one control.
Controlling runaway agent costs with rate limits
A budget alone would stop an agent stuck in a retry loop at two in the morning, but only by absorbing it. The cap trips after the money is spent, so a loop that burns a month of budget in an afternoon leaves the team on the fallback model for the next three weeks. The budget did its job and the team still pays for it.
Rate limits stop the burn instead of absorbing it. They apply to the agent’s key and act in real time, and Agent Router exposes three, each catching a different kind of runaway.
| Limit | What it caps | The agent failure it stops |
|---|---|---|
| Requests per minute | Call frequency, whatever the payload size | A retry storm, or a chatty agent looping on many small calls |
| Tokens per minute | Total tokens per minute, input plus output | An agent driving long contexts and verbose completions |
| Maximum parallel requests | Calls in flight at one moment | Fan-out, an agent opening many simultaneous connections |
Set all three on the agent’s key, sized at roughly two to three times its observed legitimate peak. The rate limit catches the fast runaway at the key in seconds, and the budget catches the slow accumulation in dollars over the month.
What a distributed AI gateway data plane requires
Budgets and rate limits are only as good as the layer that enforces them, and they fail quietly when it falls behind: a rate limit admits a burst because it is metering stale counts, a budget overshoots because one region reported late. Pushing policy to a hundred data planes is a distributed systems problem, with a long history of stale config, split-brain state, and proxies that slow down under the load they were bought for.
Whether you write that layer or adopt a new one, the risk is the same: your spend controls run on code nobody has attacked yet. A budget is a security boundary, and a bug in one is a leaked key that keeps spending. Three properties decide whether distributed enforcement holds under real traffic.
| Requirement | Why |
|---|---|
| One control plane, many enforcement points | One budget governs the whole fleet, and adding a region adds an enforcement point. Running ten copies of a single-process proxy gives you ten budgets to reconcile by hand |
| A decade of production traffic behind it | The failure modes were found by somebody else, under load heavier than yours, rather than by your budget in its first bad week |
| Maintainers from more than one company | The code is reviewed and attacked by engineers across several companies, and a security patch does not wait on one vendor’s roadmap |
Agent Router Enterprise delivers all three, built on Envoy AI Gateway, the open source AI gateway on CNCF’s Envoy Gateway project. Envoy has carried production traffic at Databricks, Google, Lyft, Netflix, and Spotify for a decade, which is where those properties were proven. Envoy AI Gateway reached 1.0 in June 2026, its first production-stable release, and runs in production at Bloomberg on the same code in the public repository, with maintainers from Tetrate, Nutanix, Tencent, and Bloomberg. No single vendor can hold a fix hostage, and no critical feature sits behind a license. Agent Router Enterprise runs that foundation across thousands of developers in hundreds of environments and regions.
Availability and how to get started
Token brokering is generally available now as part of Tetrate Agent Router Enterprise, at no additional charge to subscribers. It is not priced as a separate module.
Read the full announcement here, or get started with an enterprise trial and start managing your token cost today.
Frequently asked questions
What is token brokering?
Token brokering is a capability in Tetrate Agent Router Enterprise that evaluates every AI request against business, technical, and governance policy before choosing a model. It enforces token spend against a budget in the request path and routes each request to an approved model.
Does token brokering block requests when a budget runs out?
No. When spend would cross the budget, token brokering trips a circuit breaker and moves the request to the next model on the approved fallback list, typically a cheaper private model. The request is served rather than rejected, and the agent continues its task.
How do I cap costs for a single AI agent?
Give the agent its own API key. Rate limits on that key (requests per minute, tokens per minute, and maximum parallel requests) stop a retry storm or fan-out burst in real time. A budget on the identity that owns the key sets the dollar ceiling over the month.
Do developers have to change their code?
No. The gateway is OpenAI-compatible and Anthropic-compatible. A developer points an existing SDK at the gateway with a base URL change and an API key. Budget checks, fallback, and region rules run inside the gateway and stay invisible to the application.
How is this different from LiteLLM and similar proxies?
A single-process proxy is one enforcement point, so its budget covers only the traffic passing through that process. Ten regions means ten budgets to reconcile. Moving that process to a faster runtime such as Rust lowers latency and memory while the architecture stays the same. Agent Router Enterprise separates one central management plane from a distributed fleet of Envoy data planes, so a single budget and policy set is pushed out and enforced everywhere the traffic runs.
Can a budget fallback respect data residency?
Yes. A fallback list contains only the backends explicitly added to it, so one built from in-region providers cannot route out of region. A budget trip moves the request down that same list, so a cost fallback stays inside the residency boundary or fails closed.
David Wang 
