Run Your Own Kimi K3 for Sovereign AI with Tetrate Agent Router Enterprise
Kimi K3 is the largest open-weight model ever shipped. Here is what it takes to serve it in your own perimeter, and how to route it with Agent Router Enterprise.
David Wang 
A frontier-class language model that runs entirely inside your own network did not exist until this summer. Kimi K3, released July 16, 2026 by Moonshot AI, is a 2.8-trillion-parameter open-weight model, the largest shipped to date, and it closes enough of the gap with the top proprietary systems that self-hosting is now a real option for regulated work. You download the weights and run them behind your own firewall.
Sovereign AI comes down to three questions: whose model, on whose hardware, in whose jurisdiction. Open weights answer all three, with no data egress, no vendor rate limits, and no jurisdictional exposure. The catch is operational. K3 is straightforward to download and hard to run at team scale, a multi-GPU model that several teams will want to share safely.
Tetrate Agent Router Enterprise treats your self-hosted K3 cluster as one more backend behind a single OpenAI-compatible endpoint, with access control, failover, cost attribution, and guardrails. It is the same gateway behind our token brokering release: that governed who spends, this governs where the model runs.
What Kimi K3 is
Kimi K3 is a frontier-class large language model from Moonshot AI. It shipped on July 16, 2026, and Moonshot has scheduled the full weights for July 27.
| Attribute | Detail |
|---|---|
| Parameters | 2.8 trillion total (Mixture-of-Experts) |
| Active experts | 16 of 896 per token, roughly 1.8% of the pool |
| Context window | 1 million tokens |
| Modality | Text plus native vision |
| Reasoning | Always-on “thinking mode” |
| API compatibility | OpenAI SDK compatible |
| License | Open weight, downloadable and available for self-hosting |
| Independent ranking | Top-3 on major intelligence indices; #1 on Frontend Code Arena per Artificial Analysis and Vals reporting |
Moonshot positions K3 as competitive with the strongest proprietary systems and ahead of several of them on coding and agentic benchmarks. Treat vendor benchmark claims with the usual caution. The point for a platform team is narrower: the quality gap that used to justify shipping regulated data to an external API has closed far enough that in-perimeter inference is now a real option.
The cost picture points the same way. On published API pricing, K3 lands well under the top US frontier model, and self-hosting removes the per-token price altogether.
Why Kimi K3 and open models matter for sovereign AI
Sovereign AI means running models, data, and inference under your own control and inside your own jurisdiction. Open weights are what make it achievable, because they remove three dependencies at once.
| Dependency removed | What it means in practice |
|---|---|
| Data egress | Prompts and completions never leave your network boundary. No third party sees regulated content. |
| Vendor availability | No rate limits, deprecations, or region cutoffs imposed by an external provider. |
| Jurisdictional exposure | Inference runs where you say it runs, which matters for data-residency and export regimes. |
Kimi K3 is the frontier example used through this post, but the argument is model-agnostic. The same self-hosting path applies to the wider open-weight field, including the DeepSeek, GLM, Qwen, and Llama families. Pick the model your workload needs, and the sovereignty case does not change.
Open weights give you a model. Turning it into a service your teams can use safely is the work.
What it takes to serve Kimi K3
A 2.8-trillion-parameter model is not a laptop workload, even as a sparse MoE. The Mixture-of-Experts design activates only about 1.8% of parameters per token, which cuts compute per request. It does not cut memory. The full weight set stays resident across your accelerators, so serving K3 means a multi-GPU cluster.
The serving stack itself is well-trodden. K3 exposes an OpenAI-compatible API, so a standard inference server handles it.
| Layer | Role |
|---|---|
| Inference server (e.g. vLLM) | Loads the weights, serves the OpenAI-compatible endpoint on GPU nodes |
| GPU fleet | NVIDIA GPU nodes with enough aggregate memory to hold the full weight set |
| Endpoint selection | Load-balances across replicas so one busy pod is not a bottleneck or a single point of failure |
| Gateway | Fronts all of it with one endpoint, access control, failover, attribution, and guardrails |
That last row is where self-hosting efforts stall. You can stand up vLLM and get a working endpoint in an afternoon. Then five teams want access, each hardcodes a different base URL, nobody can see who is spending what, there is no PII redaction, and when a GPU node dies every agent pointed at it goes down with it.
Self-hosting solves sovereignty. It does not solve operations. That is a routing problem, and it is the same one you already solve for hosted providers.
Why the routing layer is the real requirement
A self-hosted model is a single endpoint. To make it usable across an organization you need the control plane you would put in front of any provider:
- One OpenAI-compatible endpoint so no team rewrites agent code to point at your K3 cluster.
- Failover so a degraded GPU node or a saturated cluster does not take agents down.
- Access control so only approved teams and projects reach the model.
- Cost and usage attribution so you know which team, agent, and project consumed which capacity.
- Guardrails enforced centrally, not reimplemented in every agent.
The common objection is that a gateway in the request path costs latency. It does not, in any amount that matters. An independent VMware/Broadcom benchmark measured Envoy AI Gateway adding about 2 ms under enterprise LLM load, against model responses measured in seconds. This is the same argument we made in why your AI bill is a gateway problem: control belongs at the gateway, where it applies once to every request, rather than in every agent.
How to run Kimi K3 with Agent Router Enterprise
Agent Router Enterprise is an Envoy-based AI gateway that sits in front of your model backends and exposes a single OpenAI-compatible endpoint. It is provider-neutral. The request pipeline extracts the model from each request, matches a route rule, and selects a backend, which can be Anthropic, Bedrock, Vertex, OpenAI, Azure, your BYOK credentials, or a self-hosted cluster. Kimi K3 on your own GPUs is another backend. It is also the meta-provider beneath your agent harnesses, the single layer where model access, tools, and guardrails live.
Deploy the data plane inside your own VPC, on-prem, or at the edge, managed by one control plane. Because the gateway runs in your perimeter, K3 traffic never leaves it. That is what makes the sovereignty story hold end to end rather than only at the model layer.
Tetrate Agent Router Enterprise provides four capabilities for a self-hosted model, each configurable from the admin console or the API.
| Capability | What it does for a self-hosted K3 |
|---|---|
| Model catalog | Register your K3 endpoint as a model. It becomes available at the same OpenAI-compatible endpoint your teams already call, so no agent code changes. |
| Access profiles | Scope which teams and projects can use K3, tied to SSO identity. |
| AI Guardrails | Enforce PII redaction, prompt filtering, and policy checks in the request path, applied once at the gateway. |
| Cost and token visibility | Attribute K3 usage per team, agent, and project for showback and chargeback. |
An Agent Router Enterprise admin sets this up in four steps.
1. Serve K3 and deploy the data plane. Run vLLM on your GPU cluster with an OpenAI-compatible endpoint, and front the replicas with a load balancer so one busy node is not a bottleneck. Deploy Agent Router as a self-hosted data plane so prompts, responses, and credentials stay on your infrastructure.
2. Add K3 to the catalog. In the Admin Dashboard, add your K3 endpoint under Providers Management, then enable the model in Models Management, following Custom and self-hosted models. Set the endpoint to a private-link address (AWS PrivateLink, Azure Private Link, GCP Private Service Connect) to keep traffic off the public internet. The same provisioning is available as configuration as code through the management API. Developers then call kimi-k3 through the OpenAI-compatible API using the SDK, with no change to the base URL they already use.
3. Scope access and turn on guardrails. Issue scoped keys by team through Onboard developers and issue keys and SSO, then enable PII redaction and prompt filtering under Safety and data protection. Attribution is automatic once traffic flows, visible in usage analytics and audit logs.
4. Set budgets and fallback. Keep traffic on in-perimeter K3 by default. Set per-key and per-team budgets, and configure a routing fallback so that when your cluster is saturated or down, requests spill to an approved cloud provider (Bedrock, Vertex, Azure) pinned to an approved region. Pin the region through data-residency controls and regional load balancing, so the spillover still lands inside policy. For workloads that must never leave your boundary, set the fallback to a second self-hosted region instead. Either way it is a policy setting, not a code change.
Run Kimi K3 in your own perimeter
Kimi K3 removes the quality excuse for shipping regulated data to an external API. What remains is an operations problem: serving a very large model, and making it safe for many teams to use. Agent Router Enterprise handles the second half so your team can spend its time on the first.
Token brokering governs who spends and how much. Sovereign serving governs where the model runs and who can see the data. Agent Router Enterprise enforces both at the same gateway. Book a demo to see it against your own stack.
FAQ
What is Kimi K3? Kimi K3 is a 2.8-trillion-parameter open-weight Mixture-of-Experts model from Moonshot AI, released July 16, 2026, with full weights due July 27. It has a 1 million token context window, native vision, always-on reasoning, and an OpenAI-compatible API.
Can you self-host Kimi K3? Yes. Moonshot released K3 under an open-weight license, so you can download the weights and serve them on your own GPU cluster with an OpenAI-compatible server such as vLLM. Expect a multi-GPU deployment, because the full weight set stays resident even though only about 1.8% of parameters activate per token.
What is sovereign AI? Sovereign AI means running models, data, and inference under your own control and inside your own jurisdiction. Open-weight models like Kimi K3 make it practical by removing dependence on an external API provider.
Why do you need an AI gateway to serve Kimi K3? A self-hosted model is a single endpoint. An AI gateway like Agent Router Enterprise gives many teams one OpenAI-compatible endpoint with access control, failover, cost attribution, and guardrails, without each team rewriting agent code.
How much latency does the gateway add? An independent VMware/Broadcom benchmark measured Envoy AI Gateway adding about 2 ms under enterprise LLM load, against model responses measured in seconds.
How does sovereign serving relate to token brokering? Token brokering governs who can spend and how much. Sovereign serving governs where the model runs. Agent Router Enterprise enforces both policies in the same request path.
David Wang 
