What Is LLM Routing?
LLM routing is choosing a model for each AI request at run time, based on the task, cost, latency, or policy. People also call it model routing. The software that makes the choice is called an LLM router or a model router.
What is LLM routing?
LLM routing picks a model for each request when the request arrives. The choice can depend on the type of task, the price of each model, the delay you can accept, or a policy rule. Without routing, a team picks one model before deployment and sends every request to that model, whether the request is hard or easy.
The two decisions answer different questions. Model selection decides which models a team may use at all, and the team usually makes that decision once, before deployment. LLM routing then picks one model from that set for each request. Most production systems need both decisions.
Why does LLM routing matter for AI agents?
An AI agent makes many model calls to finish one task. The agent plans, calls tools, reads the results, and checks its own work. Most of those calls are routine steps that a small model can handle. If every step goes to a frontier model, you pay frontier prices for routine work on every run.
Gartner predicted on August 17, 2026 that AI inference costs per agentic workflow will increase more than fivefold through 2028, even as token prices fall (Gartner press release). Gartner points to inference tiering and routing as the response. Inference tiering sends simpler subtasks to smaller, cheaper models and keeps larger models for the hard steps.
To see whether tiering works for your agents, measure cost per successful task. That metric counts every call an agent makes to finish a task, including retries and the routing calls themselves.
What are the main LLM routing methods?
Routing methods differ in the signal the router reads and in what the decision itself costs. Some methods read a header or a fixed rule and cost almost nothing. Other methods call a classifier or a second model, which adds a model call to every request. The table compares ten common methods.
| Method | How the router decides | Cost of the decision | Best for |
|---|---|---|---|
| Rule-based or attribute-based routing | Matches fixed rules on request metadata, such as a header the application sends | Close to zero | Known workload types, such as batch or interactive work |
| Cost-based routing | Picks the cheapest model that meets a stated requirement | Close to zero | High-volume work with a clear quality floor |
| Latency-aware routing | Picks the model or endpoint with the lowest recent response time | Low, but needs live latency data | User-facing chat and voice apps |
| Semantic routing | Compares an embedding of the prompt to embeddings of known request types | One embedding call per request | Apps with a stable set of request types |
| Intent or classifier routing | A small model or an LLM judge reads the prompt and picks a tier | One extra model call per request | Mixed traffic where difficulty varies a lot |
| Stage routing | Reads signals from the agent’s recent steps, such as tool results | Low | Long agent runs with planning and execution steps |
| Escalation routing | Starts on a small model and moves up after errors or failed checks | Extra calls only when a step fails | Agents that can detect their own failures |
| Model cascades | Tries a small model and passes the request to a larger one when confidence is low | A second call when confidence is low | Tasks that produce a usable confidence score |
| Traffic splitting | Sends a set percentage of requests to each model, chosen by chance | Close to zero | Testing a new model on a share of traffic |
| Fallback | Moves to the next model when the first model fails | Extra calls only on failure | Staying available during a provider outage |
Two limits. The cost column covers the routing decision only, and the model call that comes after can cost far more. And methods that read the prompt, such as semantic and classifier routing, send the prompt to one more component, which matters when the prompt contains sensitive data.
What is the difference between LLM routing and model cascading?
LLM routing picks one model before the call, and the request goes only to that model. A model cascade calls a small model first, checks the answer, and sends the request to a larger model when the check fails or confidence is low. Routing pays for the decision up front. A cascade pays for a second call on hard requests.
Hybrid designs use both. A router can pick a starting tier, and a cascade can move the request up one tier if the first answer fails a check. Escalation routing in the table above is a cascade that starts on errors.
What is the difference between an LLM router and an LLM gateway?
An LLM router selects which model or provider gets each request, based on cost, latency, availability, or capability. A gateway carries the traffic and applies credentials, budgets, access rules, guardrails, and logs. The LLM proxy guide compares a proxy, a router, and a gateway in full, and What is an LLM gateway? covers the gateway layer.
What other names do vendors use for LLM routing?
Cloud providers and model marketplaces sell LLM routing under their own product names. The methods differ. Some products route inside one model family. Others classify the prompt or match on attributes the application sends. The table lists five products and how each vendor describes routing, as of October 2026.
| Vendor | Product name | How the product describes routing | Link |
|---|---|---|---|
| Amazon Bedrock | Intelligent Prompt Routing | Routes each prompt between two models in the same model family. AWS said, as of October 2026, the feature can reduce cost by up to 30% without compromising on accuracy. | AWS |
| Microsoft Foundry | Model router | A trained model routes each prompt to an eligible model, with Balanced, Quality, and Cost modes. Routes only to eligible models and honors data zone boundaries. | Microsoft Learn |
| OpenRouter | Auto Router | A classifier assigns the prompt one of about 30 task types, then the router picks from models ranked by share of spend for that task type. | OpenRouter docs |
| NVIDIA | NeMo Switchyard | Open source routing library with random, LLM classifier, stage, and escalation strategies. | GitHub |
| Tetrate | Smart routing in Tetrate Agent Router Enterprise | Model aliases and attribute-based routing (preview), bounded by each project’s allowed models. | Tetrate docs |
What does LLM routing trade off?
LLM routing trades cost against quality and latency, and the routing decision has a cost of its own. A smaller model costs less per call but may fail more often. A classifier or LLM judge adds a model call and some delay to every request before the real work starts.
| Factor | What routing can gain | What routing can lose |
|---|---|---|
| Cost | Lower spend on routine calls | More retries when a small model fails |
| Quality | Frontier models on the hard steps | A weak answer when the router misjudges a request |
| Latency | Faster answers from small models | Extra delay from the routing decision |
| Decision cost | A better choice from a classifier or judge | One more model call on every request |
Tetrate measured the cost and delay of the routing decision itself in Jev-directed model routing.
Two limits. A method that looks cheap per call can raise the total cost per task if the cheaper model fails more often. And the trade-off changes with your workload, so test each routing method against your own single-model baseline before you trust a vendor’s number.
Why does an LLM router need an allowed model set?
A router that can pick any model in a catalog can pick a model your team isn’t allowed to use, or a model hosted in the wrong region. The allowed model set should come from access policy. The gateway should check each routing choice against that set before the request runs, and refuse any choice outside the set.
A routing rule steers traffic. A routing rule can’t restrict access, because the application often supplies the signal the rule reads. Access control belongs in the model set each team or project gets. Control which AI models each team can use explains how to set that boundary. Data residency routing applies the same idea to regions.
Two limits. An allowed model set stops a router from reaching the wrong model, but the set can’t make the router choose well inside the set. And each model you add to the set widens every routing choice, so review the set when it changes.
How does Tetrate Agent Router Enterprise route requests today?
Tetrate Agent Router Enterprise calls its routing mechanisms smart routing. Model aliases are generally available, and attribute-based routing has been in preview since v0.7.0. Routing chooses only among the models a project is allowed to use, so routing can’t reach a model the project doesn’t have. Fallback, traffic splitting, budget-based fallback, and rate limiting are also available.
The smart routing guide shows how to set up each mechanism.
Agent Router Enterprise
Tetrate Agent Router Enterprise routes AI agent traffic across providers and your own models, with policy, cost controls, and audit on every request — in cloud, on-prem, or edge.
Frequently asked questions
Is LLM routing the same as model routing? Yes. LLM routing and model routing name the same practice: choosing a model for each request at run time. Vendors also use names such as prompt routing, model router, and auto router for products that do the same job.
Does LLM routing reduce cost? LLM routing can reduce cost when routine requests go to smaller, cheaper models and those models still pass your quality checks. Routing can raise cost when the cheaper model fails often and requests must be retried. Compare cost per successful task against a single-model baseline to see which case applies to you.
Does LLM routing add latency? Some methods do. Rule-based and attribute-based routing add almost no delay, because the router reads a header or a fixed rule. Classifier and LLM judge routing add a model call before the real request runs. Smaller models often answer faster, which can offset the delay of the routing decision.
Can LLM routing keep data in one region? Yes, if the router can choose only among models hosted in the allowed region and every fallback stays in that region. When no in-region model can serve the request, the gateway should return an error so the request stays in the region.
Can an LLM router choose a model that a team is not allowed to use? A router can, if nothing checks the router’s choice. The fix is an allowed model set for each team or project, defined by access policy, with the gateway checking each routing choice against that set. The router can then pick only from models the team already has.
Related reading
- LLM proxy: the complete guide to proxies, routers, and AI gateways
- Cost per successful task: how to measure AI agent cost
- Data residency routing for LLMs
- Cost per token
- LLM provider failover when a model provider goes down
- How to switch LLM models safely
Learn more about Tetrate Agent Router Enterprise — enterprise AI agent routing with policy, cost controls, and audit across every gateway.
