Self-Hosted AI Gateway vs Hybrid vs Fully Managed: Deployment Models and Data Residency

Three deployment models differ in who runs the data plane

Tetrate Agent Router splits into two planes. The management plane stores configuration, policies, analytics, and the consoles. The data plane is the Envoy-based gateway that carries every AI request. The deployment models page names three options:

In all three models, Tetrate hosts the management plane, including the Admin Console. In the self-hosted model, prompts and responses never pass through Tetrate systems.

The management plane never carries live traffic

The planes and components page sets the rule: the management plane stores configuration and telemetry and never sees request content. The data flows page separates three flows:

1. Configuration: the data plane polls the management plane over one outbound HTTPS connection. No inbound connection from Tetrate is needed. If the connection fails, the data plane uses cached configuration and keeps routing.
2. Requests: client to gateway to provider. In the self-hosted model, content leaves your environment only to reach the model provider you chose.
3. Observability: metrics and traces go to your own Grafana, Datadog, or New Relic. No prompt content by default.

Request logs in Full mode include prompt and response bodies, and the management plane stores those logs. If bodies must stay in your environment, set request logs to Metadata only or Off. Usage and billing views still work in both modes.

A self-hosted AI gateway installs into any Kubernetes cluster by CLI, Helm, or GitOps

The self-hosted data plane is the Controller plus the AI Gateway in your cluster. The installation overview documents three paths:

Self-managed clusters such as k3s, kubeadm, and OpenShift use Helm or GitOps. Supported Kubernetes versions are listed on the release compatibility page. Cloud-specific guides exist for AWS, Azure, and GCP.

A self-hosted AI gateway enforces data residency as a routing rule

Where the data plane runs decides where the gateway processes the request. Where the model runs decides where the prompt is sent next. The data residency guide enforces residency at the routing layer. A routing policy maps a model only to provider entries in the allowed region. The region restriction applies to the primary path, every fallback, and every load-balanced path. If the in-region provider fails, the request fails. The gateway does not fall back to an out-of-region provider.

The compliance reference lists US and EU residency options. Because residency is a policy, one deployment can give different projects different guarantees.

Provider retention is separate from residency. Whether OpenAI, Anthropic, or a cloud provider keeps prompts depends on that provider’s own zero-retention setting. Some settings are available only on some contract levels. The docs state that Tetrate does not guarantee any provider offers no-retention terms, so confirm with the provider and with legal.

Self-hosted models are registered as another provider

Running your own model keeps prompts inside your own network. Any OpenAI-compatible endpoint that the data plane can reach registers as a provider. The custom and self-hosted models guide uses an Ollama endpoint as the example, and vLLM works the same way. Once registered, the self-hosted model is listed in the same catalog as GPT, Claude, and Gemini, with the same budgets and guardrails. The self-hosted model can be the fallback for a vendor model, or the primary with a vendor model as fallback.

Tetrate’s post on running Kimi K3 for sovereign AI describes each part of the setup. The setup is vLLM on a GPU cluster, a load balancer, and Agent Router as the self-hosted data plane. The post adds private-link addresses and a second self-hosted region as a fallback that sends nothing outside the country.

Private connectivity keeps provider traffic off the public internet

The self-hosted data plane can reach providers over AWS PrivateLink, Azure Private Link, or GCP Private Service Connect. The network and security page explains that you set the provider base URL to the private host name in the Admin Console. No inbound firewall rules are required. Outbound traffic is one HTTPS connection to the management plane plus the provider endpoints you configure. The fallback chain also acts as a network allowlist, because the gateway can only route to backends that appear in the chain.

Private connectivity also works inbound. Azure AI Foundry agents that send traffic to the gateway need the gateway reachable inside the virtual network the Agent Service uses. A self-hosted data plane in that network keeps the traffic on your infrastructure while Tetrate hosts the management plane.

The hybrid model saves more than the gateway costs in sovereign AI programs

Tetrate’s post on the cost of sovereign AI argues that a sovereign program should start with the gateway. The gateway records who spent what and where the traffic ran. The post’s example starts from $2 million in annual spend and saves $680,000 per year. The savings come from cheaper models for routine work, removal of spend that no one owns, and a share of traffic moved to servers inside the country.

What a self-hosted AI gateway does not cover

1. There is no option that runs without an internet connection, because the management plane is always Tetrate-hosted.
2. Request logs in Full mode send prompt bodies to the management plane until you change the mode.
3. Provider retention is the provider’s setting. Residency covers routing only.
4. Compliance reports (SOC 2 Type II, ISO/IEC 27001) are available after signing an NDA. The docs do not present a current certificate.

Frequently asked questions

What is a self-hosted AI gateway? A self-hosted AI gateway runs the data plane, which carries the AI traffic, in your own Kubernetes cluster. The vendor hosts the management and analytics plane. Prompts and responses stay in your environment.

What is the difference between hybrid and fully managed? In fully managed, the vendor runs the data plane and prompts pass through vendor infrastructure. In hybrid, you run the data plane and only configuration and telemetry reach the vendor.

How does an AI gateway enforce data residency? An AI gateway enforces data residency through routing policy. A model is mapped only to provider endpoints in the allowed region, and that mapping applies to the primary route and every fallback. Out-of-region providers are excluded, so a request fails before leaving the region.

Can I run my own models behind the gateway? Yes. Any OpenAI-compatible endpoint, such as vLLM or Ollama, registers as a provider and gets the same budgets, guardrails, and fallback as vendor models.

Does the gateway keep working if the vendor’s management plane is down? Yes. The data plane caches configuration and keeps routing. Configuration changes queue until the connection returns.

Does self-hosting mean no data reaches the vendor? Configuration, telemetry, and usage counts reach the management plane. Prompt and response bodies reach the management plane only if request logs are in Full mode.

How do you install a self-hosted AI gateway? Install the data plane into a Kubernetes cluster in one of three ways: the tare CLI on EKS, AKS, or GKE; Helm from a private registry; or GitOps manifests for Argo CD or Flux. Self-managed clusters such as k3s, kubeadm, and OpenShift use Helm or GitOps.

Can an AI gateway run without an internet connection? No. The data plane needs one outbound HTTPS connection to the vendor-hosted management plane to receive configuration. The data plane keeps routing on cached configuration if that connection fails.

Related reading

• Tetrate Agent Router product page
• Architecture overview in the docs
• Self-hosted data plane in the docs
• Route outbound traffic through a corporate proxy in the docs
• AI Compliance Framework in the Learning Center