LLM Provider Failover: What Happens When a Model Provider Goes Down
Last updated: September 2026
LLM provider failover means that when a model provider goes down, the gateway sends each failed request to the next provider in an ordered list. An application that calls the provider directly fails with the provider. An application that calls through an AI gateway with a fallback policy keeps working, and the application does not see the switch.
Why LLM provider failover matters: every provider has outages, and direct integrations do not handle them
Provider outages are common. Rate limits are hit during peak hours, regions slow down or fail, and new model launches use up capacity. An application with one provider has three choices during an outage. The application can retry the same provider, return an error to the user, or fail without telling the user. Most teams write their own retry logic, and none of the retries help when the provider is down for an hour.
Tetrate’s post on the AI gateway as a meta-provider describes the gateway as the layer that decides the provider, so the application only builds the request.
Fallback is an ordered list of backends per model
In Tetrate Agent Router Enterprise, a fallback policy adds an ordered list of fallback models to each primary model and provider pair. A request for Claude Sonnet on Anthropic might list Claude Sonnet on AWS Bedrock second and GPT on Azure OpenAI third. If the primary fails with an error on the fallback list, the gateway sends the same request to the next backend in the list. The application receives the first successful response, or the last backend’s error if every backend fails.
The gateway picks the policy by scope: API key first, then project, then organization. The narrowest policy applies in full. There is no merging.
Which errors trigger LLM provider failover
The gateway behavior reference lists what moves a request down the chain:
| Failure | Triggers fallback? | Why |
|---|---|---|
| Provider timeout or connection failure | Yes | The provider did not answer |
| Provider 429 (rate limit) | Yes | Another provider has capacity |
| Provider 500, 502, 503, 504, 529 | Yes | The provider is slow or failing |
| Provider 401 (credential rejected) | Yes, per configuration | The next backend carries a different credential |
| Client 400 (badly formed request) | No | The next provider would reject the same request |
| Client 401, 403, 404 | No | The problem is with the caller |
| Content refusal from the model | No | A refusal is a valid response |
| Budget-exhausted 429 from the gateway | No | The gateway blocked the request on purpose |
The exact trigger set can be set per API key. Two rules apply to every trigger. First, fallback decides on response headers. Once the first byte of a streamed response arrives, the gateway cannot switch providers mid-stream. Second, every request starts at the primary. The gateway keeps no record that the primary failed. Each new request tries the primary again.
How to know which provider served a request
No response header signals that a fallback occurred. The model field in the response names the model that served the request. The Request Logs in the console show the same resolved model per request. Log the model field if your application depends on a specific model’s behavior.
Set the client timeout for the whole chain. A chain of three backends with a 30-second timeout each can take 90 seconds before the request fails. The x-should-retry header tells the client whether a retry is worth attempting: true on 5xx, false on client errors and a used-up budget.
Fallback across providers means one model name maps to many contracts
To fall back from Anthropic to Bedrock, both must exist as provider entries. The guide to connecting provider subscriptions across clouds creates one entry per subscription and region, then maps a model alias to all of them. Fallback then tries each entry in order. A self-hosted model can be the last entry in the chain, as the backend with no vendor outage.
If a policy limits a model to in-region providers, the residency guide says the request should fail before leaving the region. Fallback stays inside the allowed set.
Fallback, traffic splitting, and regional load balancing are three different layers
The routing overview separates the three:
| Method | What decides where a request goes | When the method acts |
|---|---|---|
| Traffic splitting | A weight, such as 70/30 | On every request, at random |
| Fallback | A failure from the chosen backend | Only when a request fails |
| Regional load balancing | Endpoint health, queue depth, and cache state inside a pool | Before both of the above, for copies of the same model |
The regional load balancing guide uses the Kubernetes Gateway API inference extension. The extension picks an endpoint per request inside a pool of copies of the same model. Unhealthy endpoints are used last inside the pool. Fallback starts only when the whole pool fails.
The gateway itself must stay available
A gateway that is a single point of failure does not improve availability. The HA and DR guide covers multiple gateway replicas and data planes in several regions that all serve traffic. The guide also covers DNS-level failover between gateway sets with health checks you run. The operations and reliability page adds that a data plane keeps routing with cached configuration when the management plane is unreachable.
To tell a provider outage from a gateway outage, the status endpoints help. A GET / with no API key returns serving or not_serving. GET /v1/status with an API key returns whether each provider answered over a five-minute window. Both endpoints are answered by the data plane and work during a management-plane outage.
Test the chain before the outage
A fallback chain that has never been tested may not work. The Service tier fallback guide recommends forcing a failure on the primary, such as a wrong key on a test key. Then confirm that the resolved model in Request Logs shows the fallback. Repeat the test after every change in the order.
LLM provider failover at the gateway has six limits
- Fallback cannot switch providers after the first response byte of a stream.
- Fallback cannot cross data planes. Failover across data planes uses DNS.
- Fallback cannot route outside the project’s model catalog.
- The Responses API translates only to OpenAI and Azure OpenAI, so cross-provider fallback is not available for those requests.
- No provider health memory exists. During a long outage, every request waits for the primary’s timeout before moving on, so set a short timeout on the primary.
- Claude Code passthrough traffic falls back only inside Anthropic, because the Claude subscription pays only for Anthropic. Managed Claude Code keys fall back across every provider family.
Now Available
MCP Catalog with verified first-party servers, profile-based configuration, and OpenInference observability are now generally available in Tetrate Agent Router Service. Start building production AI agents today with $5 free credit.
Frequently asked questions
What is LLM provider failover? Failover is the automatic rerouting of a failed AI request to a second provider or model. A gateway performs the reroute so the application never sees the failure.
Which errors cause a gateway to fall back? A gateway falls back on timeouts, connection failures, provider rate limits (429), and provider server errors (5xx, 529). Client-side errors such as a malformed request do not, because the next provider would reject the same request.
Does fallback work for streaming responses? Fallback works for streaming responses only before the first byte arrives. Once a stream has started, the gateway cannot switch providers. A mid-stream failure is forwarded to the client.
How do I know which provider served my request?
Read the model field in the response, or check the resolved model in the gateway’s Request Logs. No response header announces a fallback.
Does fallback work for Claude Code? Yes. A managed Claude Code key uses the same fallback chain as any other key. The chain can include Anthropic, Vertex AI, OpenAI, Gemini, and self-hosted models. A passthrough key on a Claude subscription falls back only inside Anthropic.
Can the fallback go to a model I run myself? Yes. A self-hosted model registered as a provider can be in any position in the chain. The last position makes the self-hosted model the backend that does not depend on an outside vendor.
How do I test an LLM fallback chain? Force a failure on the primary, for example with a wrong key on a test key. Then confirm that the resolved model in the gateway’s Request Logs shows the fallback model. Repeat after every change to the chain.
How do I tell a provider outage from a gateway outage?
Call GET / on the gateway with no API key. A serving reply means the gateway is up. Then call GET /v1/status with an API key to see whether each provider answered in the last five minutes.
What if the gateway goes down? Run multiple replicas, deploy data planes in more than one region, and set up DNS failover between gateway sets. The data plane keeps routing when the management plane is unreachable.
Related reading
- AI Gateway product page
- LiteLLM vs Tetrate Agent Router Enterprise: The 2026 Comparison
- Apply advanced routing rules in the docs
- Route requests across providers in the docs
- Risk Mitigation Strategies in the Learning Center
MCP Catalog with verified first-party servers, profile-based configuration, and OpenInference observability are now generally available in Tetrate Agent Router Service . Start building production AI agents today.
