LLM Provider Failover: What Happens When a Model Provider Goes Down
Last updated: October 2026
LLM provider failover means that when a model provider goes down, the gateway sends each failed request to the next provider in an ordered list. Different tools call the same pattern LLM failover, LLM fallback, or multi-provider routing. An application that calls the provider directly fails with the provider. An application that calls through an AI gateway with a fallback policy keeps working, and the application does not see the switch.
TL;DR: LLM failover reroutes a failed request to the next backend in an ordered fallback chain, so a provider outage degrades one route instead of taking down the application. The gateway falls back on timeouts, rate limits, and provider server errors. The trade-offs are the latency of the failed attempt and a possible price difference on the backup. Test the chain before an outage forces it into use.
Why LLM provider failover matters: every provider has outages, and direct integrations do not handle them
Provider outages are common. Rate limits are hit during peak hours, regions slow down or fail, and new model launches use up capacity. An application with one provider has three choices during an outage. The application can retry the same provider, return an error to the user, or fail without telling the user. Most teams write their own retry logic, and none of the retries help when the provider is down for an hour. An outage causes more than one failed request. Requests queue behind the dead provider, agents stall mid-task, and every team that shares the provider pages its own on-call for the same incident.
Hard downtime is only one failure mode. Providers also return elevated error rates, hit capacity throttling with 429 responses, change model behavior after a silent update, and degrade in one region while another region stays healthy. Retrying the same provider does not cover those cases. Failover at the gateway does. Retries inside each application also leave every team with a different recovery path during the same outage. A multi-provider routing strategy moves outage handling from each application into the gateway, where one policy covers every team.
Tetrate’s post on the AI gateway as a meta-provider describes the gateway as the layer that decides the provider, so the application only builds the request.
Retries, circuit breakers, and failover solve different failures
Most teams reach for retries first. A retry resends the failed request to the same provider after a short wait, which covers a dropped connection and does nothing during an hour-long outage. A circuit breaker stops calling a backend that keeps failing, which protects latency budgets and still leaves those requests unanswered. Failover is the only mechanism of the three that produces a response during a provider outage, because it moves the request to a backend that works.
| Mechanism | The failure it covers | What it cannot do |
|---|---|---|
| Retry | A dropped connection or a brief blip on one provider | Produce a response while the provider is down |
| Circuit breaker | Error-rate spikes that would waste further requests | Answer the requests it blocks |
| Failover | Outages, rate limits, and regional degradation | Fix a badly formed request, which every backend rejects |
The three mechanisms stack, and they belong at different layers. Retries can stay in the client. The fallback chain belongs at the gateway, where one policy covers every team and nobody maintains recovery code per application.
Provider status pages and SLAs do not keep requests flowing
An outage raises two separate questions, and a provider’s service level agreement (SLA) answers only the first. The first is what the provider owes you, and the answer is usually a service credit calculated after the incident closes. A credit refunds part of the bill, and the requests that failed during the outage stay failed. The second question is what your application does in the meantime, and only infrastructure on your side of the provider boundary can answer it.
Status pages have the same limit. A provider’s status page confirms an incident after the provider declares one, and your error rate moves before the declaration. The gateway watches every response instead: the status endpoint reports whether each provider answered over the last five minutes, and the fallback chain acts on each failed request without waiting for anyone’s announcement.
Fallback is an ordered list of backends per model
In Tetrate Agent Router Enterprise, a fallback policy adds an ordered list of fallback models to each primary model and provider pair. A request for Claude Sonnet on Anthropic might list Claude Sonnet on AWS Bedrock second and GPT on Azure OpenAI third. If the primary fails with an error on the fallback list, the gateway sends the same request to the next backend in the list. The application receives the first successful response, or the last backend’s error if every backend fails.
The gateway picks the policy by scope: API key first, then project, then organization. The narrowest policy applies in full. There is no merging.
Which errors trigger LLM provider failover
The gateway behavior reference lists what moves a request down the chain:
| Failure | Triggers fallback? | Why |
|---|---|---|
| Provider timeout or connection failure | Yes | The provider did not answer |
| Provider 429 (rate limit) | Yes | Another provider has capacity |
| Provider 500, 502, 503, 504, 529 | Yes | The provider is slow or failing |
| Provider 401 (credential rejected) | Yes, per configuration | The next backend carries a different credential |
| Client 400 (badly formed request) | No | The next provider would reject the same request |
| Client 401, 403, 404 | No | The problem is with the caller |
| Content refusal from the model | No | A refusal is a valid response |
| Budget-exhausted 429 from the gateway | No | The gateway blocked the request on purpose |
The exact trigger set can be set per API key. Two rules apply to every trigger. First, fallback decides on response headers. Once the first byte of a streamed response arrives, the stream finishes on the same provider. Second, every request starts at the primary. The gateway keeps no record that the primary failed. Each new request tries the primary again.
How to know which provider served a request
No response header signals that a fallback occurred. The model field in the response names the model that served the request. The Request Logs in the console show the same resolved model per request. Log the model field if your application depends on a specific model’s behavior.
Set the client timeout for the whole chain. The wait adds up across the chain: a request that fails on three backends waits out three failed attempts before the client sees an error. The x-should-retry header tells the client whether a retry is worth attempting: true on 5xx, false on client errors and a used-up budget.
Multi-provider failover means one model name maps to many contracts
To fall back from Anthropic to Bedrock, both must exist as provider entries. The guide to connecting provider subscriptions across clouds creates one entry per subscription and region, then maps a model alias to all of them. Fallback then tries each entry in order. A self-hosted model can be the last entry in the chain, as the backend with no vendor outage.
If a policy limits a model to in-region providers, the residency guide says the request should fail before leaving the region. Fallback stays inside the allowed set. Budgets still apply on whichever provider serves the request, and the caller identity stays on the request so cost attribution survives the switch.
Model version pinning keeps a model change from surprising your app
Providers sometimes change a model behind a stable name. Pin production traffic to a specific model version through the gateway so that change waits for your review. The smart routing guide shows how a model alias maps to the backends you choose. Move the alias to a new version after you evaluate the new version.
Version pinning matters most during a failover. A chain that lands on a newer version mid-outage turns the outage into an unplanned model upgrade. Pin the version on each backend in the chain, so the fallback serves the version you tested.
LLM failover trades latency and price for availability
Failover keeps the application answering, and each hop adds a measurable cost. The latency cost is the time spent on the failed attempt. The gateway sets no request timeout of its own, so a failed attempt lasts until the provider times out, drops the connection, or returns an error. The second backend receives the request only after the first attempt fails, so the full wait for that failure is added to the response time. Every extra backend adds one more possible failed attempt, so keep the chain as short as the outcome needs.
The price cost depends on the failover pattern. The per-token price of the same model can differ between providers. Currently, regional endpoints for Claude 4.5 and later models on Bedrock and Google Cloud cost 10% more than global endpoints, so a chain that fails over to a regional backend pays more per token for the same model. Cross-model fallback changes both the price and the answer. A self-hosted model as the last entry removes the vendor outage risk, and it adds capacity your team has to operate.
| Fallback pattern | What changes on failover | What to check before it enters the chain |
|---|---|---|
| Same model, different provider (Claude Sonnet on Anthropic, then on Bedrock) | Behavior stays close. The per-token price can differ between providers | The secondary provider’s price for the same model |
| Different model (Claude Sonnet, then GPT on Azure OpenAI) | Both the answer and the price change | Output quality on your own evals |
| Self-hosted model as the last entry | No vendor outage risk. Capacity becomes your team’s responsibility | Whether the pool absorbs your peak traffic |
A day spent on fallback shows up in Usage Analytics as the same team’s spend on a different provider.
Set up LLM provider failover in five steps
Five steps take a team from one provider to a tested fallback chain.
- Create a provider entry for every subscription you hold, including cloud marketplace channels such as Bedrock, Vertex AI, and Azure OpenAI.
- Map one model alias to every backend that can serve it, and add each fallback model to the project’s model catalog.
- Order the chain on purpose: same-model backends first for stable behavior and a self-hosted model last if you run one.
- Set the client timeout to cover every backend in the chain, because the gateway sets no request timeout of its own.
- Force a failure on a test key, confirm the resolved model in Request Logs, and read the cost on the secondary path in Usage Analytics.
Coding agents and AI tools fail over through the gateway
An outage often reaches coding agents first, because one provider serves every developer at once. An agent that calls the provider directly stops when the provider stops. The same agent pointed at the gateway inherits the fallback chain. The gateway endpoint speaks both the OpenAI and Anthropic APIs, so the agent connects through a base URL change.
Claude Code is the documented case. A managed Claude Code key uses the same fallback chain as any other key, and the chain can include Anthropic, Vertex AI, OpenAI, Gemini, and self-hosted models. A passthrough key on a Claude subscription falls back only inside Anthropic, because the subscription pays only for Anthropic.
Fallback, traffic splitting, and regional load balancing are three different layers
The routing overview separates the three:
| Method | What decides where a request goes | When the method acts |
|---|---|---|
| Traffic splitting | A weight, such as 70/30 | On every request, at random |
| Fallback | A failure from the chosen backend | Only when a request fails |
| Regional load balancing | Endpoint health, queue depth, and cache state inside a pool | Before both of the above, for copies of the same model |
The regional load balancing guide uses the Kubernetes Gateway API inference extension. The extension picks an endpoint per request inside a pool of copies of the same model. Unhealthy endpoints are used last inside the pool. Fallback starts only when the whole pool fails.
To move traffic to a new model on purpose, with aliases and canary splits, see how to switch LLM models safely.
Use the failover chain to migrate providers without downtime
A provider migration is a planned failover. Add the new provider as a backend, send it a small share of traffic with a canary split, and keep the old provider in the chain as the fallback while the new one proves itself. The application never changes, because the alias stays the same while the backends behind it change.
A migration through the fallback chain has no cutover moment. The old provider keeps serving whatever the new one fails, until you remove the old provider from the chain.
The gateway itself must stay available
A gateway that is a single point of failure does not improve availability. The HA and DR guide covers multiple gateway replicas and data planes in several regions that all serve traffic. The guide also covers DNS-level failover between gateway sets with health checks you run. The operations and reliability page adds that a data plane keeps routing with cached configuration when the management plane is unreachable.
To tell a provider outage from a gateway outage, the status endpoints help. A GET / with no API key returns serving or not_serving. GET /v1/status with an API key returns whether each provider answered over a five-minute window. Both endpoints are answered by the data plane and work during a management-plane outage.
Test the chain before the outage
A fallback chain that has never been tested may not work. The Service tier fallback guide recommends forcing a failure on the primary, such as a wrong key on a test key. Then confirm that the resolved model in Request Logs shows the fallback. After the test, check that token counts, cost, and logs still match policy on the secondary path, using Usage Analytics. Repeat the test after every change in the order, and after provider or credential changes.
Six setup tips make LLM provider failover reliable
- Fallback switches providers before the first byte of a streamed response. A stream that has started finishes on the same provider.
- Use fallback inside each data plane, and DNS to fail over between data planes.
- Add every fallback model to the project’s model catalog, because fallback routes to the models the project has.
- Send cross-provider traffic through the Chat Completions API or the Anthropic Messages API. The Responses API routes to OpenAI and Azure OpenAI.
- Use managed Claude Code keys for fallback across every provider family. Passthrough traffic falls back inside Anthropic, because the Claude subscription pays only for Anthropic.
- Compare the per-token price of each backend when you order the chain, because the order sets what an outage costs.
Agent Router Enterprise
Tetrate Agent Router Enterprise routes AI agent traffic across providers and your own models, with policy, cost controls, and audit on every request — in cloud, on-prem, or edge.
Frequently asked questions
What is LLM provider failover? Failover is the automatic rerouting of a failed AI request to a second provider or model. A gateway performs the reroute so the application never sees the failure.
Which errors cause a gateway to fall back? A gateway falls back on timeouts, connection failures, provider rate limits (429), and provider server errors (5xx, 529). Client-side errors such as a malformed request do not, because the next provider would reject the same request.
Is LLM failover the same as load balancing? No. Load balancing spreads requests across healthy backends on every request, and failover acts only after a request fails. The two work together: regional load balancing picks an endpoint inside a pool, and fallback leaves the pool when every endpoint fails.
How much latency does LLM failover add? A failover adds the time the failed attempt took. The gateway sets no request timeout of its own, so the second backend starts only after the first provider times out, drops the connection, or returns an error. Set the client timeout to cover the whole chain, and keep the chain short.
Do provider SLAs cover LLM outages? An SLA usually pays a service credit after the incident closes, and the requests that failed during the outage stay failed. Failover is the control that keeps requests flowing while the outage runs, and it has to live on your side of the provider boundary.
Can failover help me migrate to a new LLM provider? Yes. Add the new provider as a backend, send it a canary share of traffic, and keep the old provider as the fallback until the new one proves itself. The alias stays the same, so applications do not change.
Does fallback work for streaming responses? Yes, before the first byte arrives. A stream that has started finishes on the same provider, and the client receives any error that happens mid-stream.
How do I know which provider served my request?
Read the model field in the response, or check the resolved model in the gateway’s Request Logs. No response header announces a fallback.
Does fallback work for Claude Code? Yes. A managed Claude Code key uses the same fallback chain as any other key. The chain can include Anthropic, Vertex AI, OpenAI, Gemini, and self-hosted models. A passthrough key on a Claude subscription falls back only inside Anthropic.
Can the fallback go to a model I run myself? Yes. A self-hosted model registered as a provider can be in any position in the chain. The last position makes the self-hosted model the backend that does not depend on an outside vendor.
Does multi-provider failover require maintaining multiple provider contracts? You need credentials for each failover channel. Bring-your-own-key support means existing contracts carry over, and cloud marketplace channels such as Bedrock, Vertex, and Azure often fall under agreements you already have.
Will responses differ when I fail over? Same-model failover across providers keeps behavior close. Cross-model fallback can change the answer, so approve fallback targets first and log which model served the request.
How do I test an LLM fallback chain? Force a failure on the primary, for example with a wrong key on a test key. Then confirm that the resolved model in the gateway’s Request Logs shows the fallback model. Repeat after every change to the chain.
How do I tell a provider outage from a gateway outage?
Call GET / on the gateway with no API key. A serving reply means the gateway is up. Then call GET /v1/status with an API key to see whether each provider answered in the last five minutes.
What if the gateway goes down? Run multiple replicas, deploy data planes in more than one region, and set up DNS failover between gateway sets. The data plane keeps routing when the management plane is unreachable.
Related reading
- AI Gateway product page
- LiteLLM vs Tetrate Agent Router Enterprise: The 2026 Comparison
- Apply advanced routing rules in the docs
- Route requests across providers in the docs
- Risk Mitigation Strategies in the Learning Center
Learn more about Tetrate Agent Router Enterprise — enterprise AI agent routing with policy, cost controls, and audit across every gateway.
