LLM Provider Failover: What Happens When a Model Provider Goes Down
Last updated: October 2026
LLM provider failover means that when a model provider goes down, the gateway sends each failed request to the next provider in an ordered list. An application that calls the provider directly fails with the provider. An application that calls through an AI gateway with a fallback policy keeps working, and the application does not see the switch.
Why LLM provider failover matters: every provider has outages, and direct integrations do not handle them
Provider outages are common. Rate limits are hit during peak hours, regions slow down or fail, and new model launches use up capacity. An application with one provider has three choices during an outage. The application can retry the same provider, return an error to the user, or fail without telling the user. Most teams write their own retry logic, and none of the retries help when the provider is down for an hour.
Hard downtime is only one failure mode. Providers also return elevated error rates, hit capacity throttling with 429 responses, change model behavior after a silent update, and degrade in one region while another region stays healthy. Retrying the same provider does not cover those cases. Failover at the gateway does. Retries inside each application also leave every team with a different recovery path during the same outage.
Tetrate’s post on the AI gateway as a meta-provider describes the gateway as the layer that decides the provider, so the application only builds the request.
Fallback is an ordered list of backends per model
In Tetrate Agent Router Enterprise, a fallback policy adds an ordered list of fallback models to each primary model and provider pair. A request for Claude Sonnet on Anthropic might list Claude Sonnet on AWS Bedrock second and GPT on Azure OpenAI third. If the primary fails with an error on the fallback list, the gateway sends the same request to the next backend in the list. The application receives the first successful response, or the last backend’s error if every backend fails.
The gateway picks the policy by scope: API key first, then project, then organization. The narrowest policy applies in full. There is no merging.
Which errors trigger LLM provider failover
The gateway behavior reference lists what moves a request down the chain:
| Failure | Triggers fallback? | Why |
|---|---|---|
| Provider timeout or connection failure | Yes | The provider did not answer |
| Provider 429 (rate limit) | Yes | Another provider has capacity |
| Provider 500, 502, 503, 504, 529 | Yes | The provider is slow or failing |
| Provider 401 (credential rejected) | Yes, per configuration | The next backend carries a different credential |
| Client 400 (badly formed request) | No | The next provider would reject the same request |
| Client 401, 403, 404 | No | The problem is with the caller |
| Content refusal from the model | No | A refusal is a valid response |
| Budget-exhausted 429 from the gateway | No | The gateway blocked the request on purpose |
The exact trigger set can be set per API key. Two rules apply to every trigger. First, fallback decides on response headers. Once the first byte of a streamed response arrives, the stream finishes on the same provider. Second, every request starts at the primary. The gateway keeps no record that the primary failed. Each new request tries the primary again.
How to know which provider served a request
No response header signals that a fallback occurred. The model field in the response names the model that served the request. The Request Logs in the console show the same resolved model per request. Log the model field if your application depends on a specific model’s behavior.
Set the client timeout for the whole chain. A chain of three backends with a 30-second timeout each can take 90 seconds before the request fails. The x-should-retry header tells the client whether a retry is worth attempting: true on 5xx, false on client errors and a used-up budget.
Fallback across providers means one model name maps to many contracts
To fall back from Anthropic to Bedrock, both must exist as provider entries. The guide to connecting provider subscriptions across clouds creates one entry per subscription and region, then maps a model alias to all of them. Fallback then tries each entry in order. A self-hosted model can be the last entry in the chain, as the backend with no vendor outage.
If a policy limits a model to in-region providers, the residency guide says the request should fail before leaving the region. Fallback stays inside the allowed set. Budgets still apply on whichever provider serves the request, and the caller identity stays on the request so cost attribution survives the switch.
Model version pinning keeps a model change from surprising your app
Providers sometimes change a model behind a stable name. Pin production traffic to a specific model version through the gateway so that change waits for your review. The smart routing guide shows how a model alias maps to the backends you choose. Move the alias to a new version after you evaluate the new version.
Fallback, traffic splitting, and regional load balancing are three different layers
The routing overview separates the three:
| Method | What decides where a request goes | When the method acts |
|---|---|---|
| Traffic splitting | A weight, such as 70/30 | On every request, at random |
| Fallback | A failure from the chosen backend | Only when a request fails |
| Regional load balancing | Endpoint health, queue depth, and cache state inside a pool | Before both of the above, for copies of the same model |
The regional load balancing guide uses the Kubernetes Gateway API inference extension. The extension picks an endpoint per request inside a pool of copies of the same model. Unhealthy endpoints are used last inside the pool. Fallback starts only when the whole pool fails.
To move traffic to a new model on purpose, with aliases and canary splits, see how to switch LLM models safely.
The gateway itself must stay available
A gateway that is a single point of failure does not improve availability. The HA and DR guide covers multiple gateway replicas and data planes in several regions that all serve traffic. The guide also covers DNS-level failover between gateway sets with health checks you run. The operations and reliability page adds that a data plane keeps routing with cached configuration when the management plane is unreachable.
To tell a provider outage from a gateway outage, the status endpoints help. A GET / with no API key returns serving or not_serving. GET /v1/status with an API key returns whether each provider answered over a five-minute window. Both endpoints are answered by the data plane and work during a management-plane outage.
Test the chain before the outage
A fallback chain that has never been tested may not work. The Service tier fallback guide recommends forcing a failure on the primary, such as a wrong key on a test key. Then confirm that the resolved model in Request Logs shows the fallback. After the test, check that token counts, cost, and logs still match policy on the secondary path, using Usage Analytics. Repeat the test after every change in the order, and after provider or credential changes.
Six setup tips make LLM provider failover reliable
- Fallback switches providers before the first byte of a streamed response. A stream that has started finishes on the same provider.
- Use fallback inside each data plane, and DNS to fail over between data planes.
- Add every fallback model to the project’s model catalog, because fallback routes to the models the project has.
- Send cross-provider traffic through the Chat Completions API or the Anthropic Messages API. The Responses API routes to OpenAI and Azure OpenAI.
- Set a short timeout on the primary, so each request moves to the next backend quickly during a long outage.
- Use managed Claude Code keys for fallback across every provider family. Passthrough traffic falls back inside Anthropic, because the Claude subscription pays only for Anthropic.
Agent Router Enterprise
Tetrate Agent Router Enterprise routes AI agent traffic across providers and your own models, with policy, cost controls, and audit on every request — in cloud, on-prem, or edge.
Frequently asked questions
What is LLM provider failover? Failover is the automatic rerouting of a failed AI request to a second provider or model. A gateway performs the reroute so the application never sees the failure.
Which errors cause a gateway to fall back? A gateway falls back on timeouts, connection failures, provider rate limits (429), and provider server errors (5xx, 529). Client-side errors such as a malformed request do not, because the next provider would reject the same request.
Does fallback work for streaming responses? Yes, before the first byte arrives. A stream that has started finishes on the same provider, and the client receives any error that happens mid-stream.
How do I know which provider served my request?
Read the model field in the response, or check the resolved model in the gateway’s Request Logs. No response header announces a fallback.
Does fallback work for Claude Code? Yes. A managed Claude Code key uses the same fallback chain as any other key. The chain can include Anthropic, Vertex AI, OpenAI, Gemini, and self-hosted models. A passthrough key on a Claude subscription falls back only inside Anthropic.
Can the fallback go to a model I run myself? Yes. A self-hosted model registered as a provider can be in any position in the chain. The last position makes the self-hosted model the backend that does not depend on an outside vendor.
Does multi-provider failover require maintaining multiple provider contracts? You need credentials for each failover channel. Bring-your-own-key support means existing contracts carry over, and cloud marketplace channels such as Bedrock, Vertex, and Azure often fall under agreements you already have.
Will responses differ when I fail over? Same-model failover across providers keeps behavior close. Cross-model fallback can change the answer, so approve fallback targets first and log which model served the request.
How do I test an LLM fallback chain? Force a failure on the primary, for example with a wrong key on a test key. Then confirm that the resolved model in the gateway’s Request Logs shows the fallback model. Repeat after every change to the chain.
How do I tell a provider outage from a gateway outage?
Call GET / on the gateway with no API key. A serving reply means the gateway is up. Then call GET /v1/status with an API key to see whether each provider answered in the last five minutes.
What if the gateway goes down? Run multiple replicas, deploy data planes in more than one region, and set up DNS failover between gateway sets. The data plane keeps routing when the management plane is unreachable.
Related reading
- AI Gateway product page
- LiteLLM vs Tetrate Agent Router Enterprise: The 2026 Comparison
- Apply advanced routing rules in the docs
- Route requests across providers in the docs
- Risk Mitigation Strategies in the Learning Center
Learn more about Tetrate Agent Router Enterprise — enterprise AI agent routing with policy, cost controls, and audit across every gateway.
