Data Residency Routing for LLMs
Data residency routing sends each LLM request only to model endpoints inside an allowed region. The rule also covers failover and the model a router picks. The design goal is to fail closed: when an in-region model fails, the gateway returns an error, and the request stays in the region.
What is data residency routing?
Data residency routing is a gateway rule that limits where an LLM request can go. Each model maps only to provider endpoints in an allowed region. The limit applies to the first attempt, to every fallback, and to every choice a router makes. Other names for the same design are region-locked routing and jurisdiction-aware routing.
Residency routing controls where the prompt goes for inference. Where the gateway itself runs is a separate decision, covered in AI gateway deployment models.
What is the difference between data residency and data sovereignty?
Data residency is about location: where data is processed and stored. Data sovereignty is about authority: which government’s laws apply to the data, and who controls the infrastructure and the models. A system can meet a residency rule and still fall under a foreign law, for example when a foreign company operates the in-region endpoint.
| Data residency | Data sovereignty | |
|---|---|---|
| Question it answers | Where is the data processed and stored? | Whose laws apply, and who controls the systems? |
| Typical rule | Process prompts only in the EU | Use models and infrastructure under local control |
| What routing can enforce | Which regional endpoints receive each request | Which providers and self-hosted models are in the allowed set |
What is sovereign AI? covers sovereignty across data, models, operations, and compute.
How do LLM routers and failover break data residency?
Three common designs send requests out of region without anyone choosing to. A global endpoint pool spreads requests across regions. A fallback rule moves traffic to another region during an outage or a capacity shortage. And a router that may pick any model in a catalog can pick a model hosted somewhere else.
| Cause | What happens | What to change |
|---|---|---|
| Global endpoint pools | The provider balances requests across regions to use spare capacity | Use regional endpoints only |
| Silent cross-region fallback | An outage or capacity shortage sends traffic to the nearest healthy region, and nothing warns you | Put only in-region models on the fallback list |
| A router that may pick any catalog model | The router picks a cheaper or faster model hosted in another region | Limit the router to an allowed model set for each region |
What does fail closed mean for LLM traffic?
Fail closed means the gateway returns an error when no in-region model can serve a request. The request doesn’t leave the region to find a healthy model. The opposite design, fail open, keeps serving traffic by sending the request wherever a model is available, and the residency rule breaks for as long as the outage lasts.
The trade-off is plain. Under fail closed, some requests fail during an outage, and no request leaves the region. Under fail open, every request gets an answer, and some answers come from outside the region. Teams with a residency requirement choose fail closed and plan for the errors with retries, a second in-region provider, or a clear message to the user.
Two limits. Fail closed protects the residency rule only for traffic that goes through the gateway. And fail closed turns a provider outage into an outage for your users, unless a second in-region model is ready.
How do you keep failover inside a region?
Put only in-region models on the fallback list. When the first model fails, the gateway moves to the next permitted model in the same region, even when a cheaper or faster model is available somewhere else. If the list runs out, the gateway fails closed and returns an error.
Build the list before the outage. A useful in-region list has at least two providers, or one provider plus self-hosted capacity, so one provider’s outage doesn’t stop all traffic. LLM provider failover covers fallback order in more depth.
Two limits. In-region backup capacity is often smaller, so a regional outage can push the backup provider into its rate limits. And some models aren’t offered in every region, so the in-region fallback may be a different model from the primary.
How should an LLM router choose models under residency rules?
The access policy defines an allowed model set for each region. The router chooses only inside that set, using whatever method it uses for cost or quality. The gateway then checks the router’s choice against the set before the request runs, so the gateway refuses a wrong choice before any data moves.
The check matters because a router aims for cost or quality, and region isn’t part of that goal unless you add region to the rules. Apply region as a hard limit on the candidate list before the router scores any model. LLM routing explains the routing methods and why every router needs an allowed model set.
How do open-weight and self-hosted models support sovereign AI?
Open-weight models can run on GPUs in your own country, in your own data center or with a local cloud provider. Self-hosted capacity gives a router more in-region choices. Routing can send traffic to that capacity first and use commercial APIs only for overflow, as long as the overflow endpoints are also in the region.
Tetrate’s post on running Kimi K3 for sovereign AI describes one setup, with vLLM on a GPU cluster. AI gateway deployment models explains how a self-hosted gateway enforces residency as a routing rule.
What evidence should a residency routing system record?
For each request, record the requested model, the served model, the region that served the request, the policy that applied, and any policy that gave way. Those five fields show where each prompt went and why. Without those records, you can’t show an auditor that failover stayed in the region.
| Field | What the field shows |
|---|---|
| Requested model | What the application asked for |
| Served model | What answered the request, after routing and fallback |
| Region | Where the served model processed the prompt |
| Policy that applied | The residency or routing rule that made the choice |
| Policy that gave way | Any rule that lost to a higher-priority rule, such as a cost rule that lost to the residency rule |
Legal teams may use these records as input to a transfer impact assessment. The records describe what the system did. Whether a transfer is lawful is a decision for your legal team.
How does Tetrate Agent Router Enterprise enforce residency routing today?
In Tetrate Agent Router Enterprise, a routing policy can limit a model to provider entries in an allowed region. Fallbacks stay in that region. If the in-region provider fails, the request fails and stays in the region. Each request records which model served the request, which policy applied, and which policy gave way.
Agent Router Enterprise can also send traffic to private GPU capacity first and send overflow to commercial APIs only when private capacity is full. The product page describes these controls.
Agent Router Enterprise
Tetrate Agent Router Enterprise routes AI agent traffic across providers and your own models, with policy, cost controls, and audit on every request — in cloud, on-prem, or edge.
Frequently asked questions
Does data residency routing make an AI system GDPR compliant? No single control makes an AI system compliant with GDPR. Data residency routing is one technical measure that limits where prompts are processed. Compliance also depends on contracts, data handling, retention, and legal review, so ask your legal team.
Can an LLM router keep data in the EU? Yes, if every model the router may choose and every fallback is hosted in the EU, and the gateway fails closed when no EU model is available. The router then has no path out of the EU for traffic that goes through the gateway.
What happens when the only in-region model is down? The gateway returns an error, and the request stays in the region. That behavior is called fail closed. To reduce these errors, add a second in-region provider or self-hosted capacity to the fallback list.
Is data residency the same as sovereign AI? No. Data residency is about where data is processed and stored. Sovereign AI is broader: sovereign AI also covers which laws apply and who controls the models, the infrastructure, and the operations. Residency routing supports sovereign AI, but residency routing covers only location.
Related reading
- What is sovereign AI?
- Self-hosted AI gateway vs hybrid vs fully managed: deployment models and data residency
- Unpacking the cost of sovereign AI: how a distributed AI gateway pays for the program
- What is LLM routing?
- Cost per successful task
Learn more about Tetrate Agent Router Enterprise — enterprise AI agent routing with policy, cost controls, and audit across every gateway.
