Token Optimization: How Engineering Leaders Control AI Token Costs
How engineering leaders track AI token spend by team, set budgets that hold, and decide who needs a Claude Code subscription, from one AI gateway.
David Wang 
By David Wang, Head of Product, Tetrate
Most engineering organizations know their monthly AI total and cannot decompose it. They cannot say which team, which model, or which person produced it, and they cannot say what a unit of finished work cost. When the number doubles, the only available response is to ask everyone to use less.
This guide covers the three decisions that move that number: understanding where spend goes, setting team budgets that hold, and deciding who needs a coding-agent subscription. Every screenshot comes from Tetrate’s own Agent Router Enterprise deployment, showing real traffic from an internal Tetrate engineering team. No customer data appears in this post. Names, emails, and team names are blurred, and the figures are unaltered.
Why AI cost measurement belongs at the gateway
An agent harness governs one thing well: itself. Claude Code, Codex, and Cursor each hold their own context and see only their own sessions. Controls at that level, including per-session spend caps, are scoped to one session on one machine.
Cost accrues across sessions, models, and people. The layer that spans all of them is the meta-provider, the path beneath the harness that unifies every model and tool an agent reaches and governs that traffic in one place (The AI Gateway Is the Meta-Provider for Your Agent Harness).
Every request crosses that boundary on the way to a provider, so it is the one place where token counts, resolved model, cost, and caller identity land in the same record, for every team, without asking any team to instrument anything.
Part 1: Understand your organization’s AI spend
Token usage vs. cost: why the two diverge
Usage → Analytics opens on a chart named Cost vs. Token Over Time. Cost is drawn as stacked bars against the left axis, total tokens as a line against the right. The two are plotted together because reading either alone produces a wrong answer.
The table beneath ranks API keys by cost, with the user and team each key belongs to. The top three keys show 140.7M tokens for $132.80, 100.3M tokens for $124.70, and 65M tokens for $85.90: effective cost from $0.94 to $1.32 per million, on the same day in the same organization.
Keys whose owner is not mapped to a team show as No team, which is its own audit: unmapped users are spend no cost center currently owns. The group-to-team mapping determines whether the team dimension can be trusted, since a user counted in the wrong team is spend charged to the wrong cost center.
Find which models drive your AI bill
Filtering to one team and grouping by model narrows the same view to a model mix.
| Model | Requests | Tokens | Cost | Effective $/M tokens |
|---|---|---|---|---|
| claude-sonnet-4-6 | 1.4K | 79.4M | $59.80 | $0.75 |
| claude-opus-4-8 | 512 | 45.3M | $53.20 | $1.17 |
| claude-sonnet-5 | 1.5K | 132.8M | $47.30 | $0.36 |
| claude-opus-4-7 | 120 | 3.7M | $7.40 | $2.00 |
| claude-opus-5 | 53 | 6M | $5.70 | $0.95 |
The heaviest model by token volume is the third heaviest by cost. A model carrying 1.4% of this team’s tokens accounts for 4% of its spend at $2.00 per million. Rank these five by tokens and by cost and the orders do not match, which is why a token quota is a poor instrument for controlling a bill.
Find which users and teams drive AI spend
Grouping by user answers who to talk to.
Each row carries the user’s team, their API key count, requests, and cost. The key count is the detail worth noticing. The top consumer holds 13 keys and spent $149.80; the second holds one key and spent $132.80. A user with one key is a user whose spend cannot be decomposed further, which is the argument for issuing one key per purpose instead of one key per person (Onboard developers and issue keys).
Measure cache hit rate and cached token share
The Tokens tab reports tokens in four classes rather than two, and the two cache classes price differently.
| Token class | What it is | How it prices against standard input |
|---|---|---|
| Input | Context read fresh, with no cache hit | Baseline |
| Cache creation | Context written into the provider’s cache on first use | A premium above baseline |
| Cache read | Context served from cache on a later call | A fraction of baseline |
| Output | Tokens the model generates | Several times baseline |
Cache read share: 89.5%. Nine of every ten tokens read come from cache. That is why claude-sonnet-5 ran at $0.36 per million in the model table above.
Cache reuse ratio: 17.1 to 1. Cache read divided by cache creation gives the number of times each cached token was read back. Because writing to cache carries a premium, a ratio near 1 means paying that premium and collecting no discount. At 17 reads per write, this organization is well past break-even.
Output share: 0.54%. The organization reads about 182 tokens for every one it produces, which is normal for coding agents. Shortening completions saves nothing worth measuring.
The per-user columns show how widely the ratio varies between people doing the same kind of work:
| Rank by tokens | Tokens | API keys | Fresh input share | Cache reuse ratio |
|---|---|---|---|---|
| 1 | 9.7B | 19 | 4.4% | 15.7 to 1 |
| 2 | 6.4B | 1 | 0.8% | 21.3 to 1 |
| 3 | 5.1B | 8 | 0.7% | 28.2 to 1 |
| 4 | 2.8B | 1 | 1.2% | 22.6 to 1 |
| 5 | 2.6B | 2 | 0.6% | 25.6 to 1 |
The largest consumer is the least cache-efficient, at just over half the third-ranked user’s ratio, and reads over six times as much fresh input as a share of total tokens. The adjacent column suggests why: 19 API keys against one or two for everyone else, so likely many workloads whose prompt prefixes never hold still. This view raises that question without settling it, which makes it one for the team owning those keys.
Turn the readings into cost decisions
| What you see | What it means | The move |
|---|---|---|
| A model low on tokens and high on cost | Premium capacity serving work a smaller model handles | Shift a weighted share with traffic splitting and read the quality data before committing |
| One person far above the rest | A heavier workload, or a less efficient one | Read their reuse ratio before assuming either |
| High fresh input, low reuse ratio | The prompt prefix moves too often to be cacheable | Put variable content after the stable system prompt and tool schemas, and batch related calls |
Keys showing as No team | Spend with no cost center | Fix the identity-provider group mapping |
Both figures are computed on these screens already, so a monthly review costs nothing beyond the review itself (Catch spend drift early).
Part 2: Set AI spend budgets and limits by team
Review every team budget in one list
Usage → Budgets holds every ceiling in one list, with a rollup across the portfolio.
The four cards are the review surface: spend against total budgeted, active budgets against dormant ones, and counts near and over limit. Two of the six team budgets here are inactive, which is the state a ceiling sits in during a renegotiation.
Choose a budget scope: team, per-member, or individual
| Scope | What the limit covers | When it fits |
|---|---|---|
| Whole team | One shared limit pooled across all members | A cost center with a single ceiling, where one member’s heavy month offsets another’s light one |
| Each teammate | The same limit applied to every member individually | A per-person allowance that holds regardless of what the rest of the team spends |
| One person | A limit on a single user | An individual bounded independently of any team |
Limits are entered in dollars, with presets at $100, $500, $1,000, $5,000, and $10,000, and reset at the start of each calendar month in UTC. Scopes coexist and the most restrictive applicable limit governs, so a team can carry a pooled ceiling while members carry tighter ones underneath it. Size the ceiling from the ranked user view in Part 1 instead of from guesswork: comfortably above normal consumption, well below the level that would be a problem (Keep each team’s spend inside an agreed ceiling).
Read budget burn before the ceiling hits
A pooled budget leads with a single meter, $260.75 of $5,000.00 · monthly, plus Used, Remaining, and Days left, with Top contributors ranking members by share. Read Used and Days left together: 80% used with 20 days left is a conversation to have this week.
A per-member budget leads with membership health instead. 7 of 7 members within limits breaks into over, near, and healthy. Total allotted is the per-member cap multiplied by the member count. The member list sorts by spend, so exceptions surface at the top, and the near rows are the ones worth a message before they cross (See budget exhaustion coming).
Budgets alert, rate limits block
| Budget | Rate limit | |
|---|---|---|
| Controls | Cumulative spend over a month | Throughput over a short window |
| Scope | Team, each teammate, or one person | Model, API key, user, or group |
| Enforcement | Alerts only. Requests continue to be served | Inline at the gateway. Requests over the ceiling get a 429 |
| Catches | A slow overrun across weeks | A retry storm or fan-out burst happening now |
Budget spend is evaluated on the management plane against aggregated request logs, so a budget is a forecasting and accountability control, and the alert has to reach someone who will act. Rate limits come in three forms, capping tokens per minute, requests per minute, and requests in flight at once. The working pattern is budgets on production keys, where a false-positive block is worse than an overrun, and rate limits on research, evaluation, and CI keys, where a runaway is worse than a rejection (Stop runaway workloads, Choose the right cost control).
Budgets are mutable in place, so a ceiling can be raised mid-period when growth is agreed and funded, or paused during a renegotiation, with each change recorded in the audit trail. Any view exports as CSV for showback and chargeback, grouped by user, team, or model (Bill AI spend back to the teams that incur it).
Part 3: Decide who needs a Claude Code subscription
How passthrough mode works
A developer on a Claude Max or Pro plan can point Claude Code at the gateway and keep paying the flat subscription fee. Anthropic still authenticates them with Claude Code’s stored OAuth token, and the gateway routes and observes the traffic without putting those tokens on Agent Router billing. Agent Router calls this passthrough mode, and it sits alongside two other modes.
| Mode | Who pays the provider | Counted toward budget |
|---|---|---|
| Passthrough | The developer, through a flat subscription | No, shown alongside for scale |
| Managed | The organization, through provider credentials on the gateway | Yes |
| BYOK | The team, through its own provider account | No, attributed to that account directly |
The gateway meters passthrough traffic and prices it at standard API rates anyway, which sets up the comparison below.
Comparing subscription cost against à la carte usage
Under the meter sits passthrough usage (not counted toward budget) ≈ $13,535.17, against $64.33 of billed spend. Same seven people, same period, a ratio of roughly 210 to 1. The per-member rows carry both figures side by side:
| Rank by billed | Billed | Passthrough at API rates | Rank by passthrough |
|---|---|---|---|
| 1 | $26.41 | ≈ $6,218.57 | 1 |
| 2 | $17.35 | ≈ $1,998.73 | 3 |
| 3 | $9.70 | ≈ $3,470.55 | 2 |
| 4 | $4.63 | ≈ $267.08 | 6 |
| 5 | $4.53 | ≈ $0.49 | 7 |
| 6 | $1.17 | ≈ $790.89 | 4 |
| 7 | $0.53 | ≈ $788.86 | 5 |
Only the top member holds position across the two rankings. The person ranked fifth on billed spend is last on actual consumption, at 49 cents of passthrough, because their work runs through metered API calls. The two ranked sixth and seventh climb to fourth and fifth, because theirs runs almost entirely on a subscription.
Decide which subscription seats to keep
The passthrough column is a per-person estimate of what a seat is worth. Compare it against the current list price of the subscription and the decision falls out.
| Passthrough per month | Read | Action |
|---|---|---|
| Far above the seat price | The subscription is saving money every month | Keep the seat, and confirm the volume is necessary before celebrating it |
| Near the seat price | Roughly break-even | Leave it in place and re-read next cycle |
| Far below the seat price | Not enough usage to justify a flat fee | Move to metered access and reclaim the seat |
| Near zero, with billed spend | Already working à la carte | No seat needed |
On the team above, one seat is carrying more than $6,000 of monthly work and another is carrying 49 cents. Both pay the same flat fee. Reallocating the second seat is a decision no provider console can support, because no single provider sees both the subscription traffic and the metered traffic for the same person.
Two cautions. The passthrough figure is an estimate at list prices, so treat it as a scale comparison and not an invoice (Read the shadow bill). And a low figure can mean low usage or a person who has not been pointed at the gateway yet, so confirm the routing before reclaiming anything.
Get started with Agent Router Enterprise
Tetrate Agent Router Enterprise is now available as a fully managed offering on a monthly subscription, built for teams that need to understand and control what their AI costs. Tetrate runs both the management plane and the data plane, so there is nothing to install. Trial licenses are issued within 24 hours.
FAQ
What is token optimization?
Making AI spend decomposable by team, model, and person, then acting on the figures that move it: effective cost per million tokens, cache hit rate, and what a unit of finished work costs. It is a measurement problem before it is a tuning problem.
Why do AI token usage and cost not move together?
Cached input, model choice, and prompt length change cost per token independently of volume. In one 24-hour window, the heaviest model by token count ranked third by spend, and effective cost ranged from $0.36 to $2.00 per million tokens across five models on the same team.
What is a cache reuse ratio?
Cache read tokens divided by cache creation tokens, which gives the average number of times each cached token was read back. Because writing to cache carries a price premium, a ratio near 1 means paying that premium without collecting the discount.
Do AI spend budgets block requests?
In Tetrate Agent Router Enterprise, budgets alert rather than block. Spend is evaluated on the management plane against aggregated request logs, so requests continue to be served past the ceiling. A rate limit is the control that rejects requests inline at the gateway with a 429.
What is passthrough mode?
Passthrough mode lets a developer route Claude Code through the gateway while still paying a flat Claude Max or Pro subscription fee. The gateway observes and prices that traffic at standard API rates for comparison, and it never draws down a budget.
How do I decide who needs a Claude Code subscription?
Compare each person’s passthrough figure against the current seat price. Passthrough far above the seat price means the subscription is saving money. Far below means that person is cheaper on metered API access and the seat can be reclaimed.
David Wang 
