AI Gateway for Kubernetes and EKS: How the Gateway Fits Your Cluster and Your Coding Tools
Last updated: September 2026
An AI gateway for Kubernetes installs into your existing cluster as a standard Helm chart or GitOps manifest. The gateway uses your cloud IAM and sends metrics and traces to your monitoring tools. Coding tools such as Claude Code, Cursor, and GitHub Copilot connect to the gateway by changing a base URL and an API key. Azure AI Foundry agents and the Copilot CLI can send their model calls through the gateway today.
An AI gateway for Kubernetes installs as a Helm chart in two namespaces
The self-hosted data plane of Tetrate Agent Router Enterprise is a set of ordinary Kubernetes resources. The architecture overview describes an Envoy-based gateway plus a Controller that polls a Tetrate-hosted management plane over one outbound HTTPS connection. No inbound connection is required.
| Resource | Value | Purpose |
|---|---|---|
Namespace tars-system | Controller | Polls configuration, applies policy |
Namespace tars-dataplane | Gateway | Carries AI traffic |
Service egress, port 10080 | Gateway entry point | Serves /v1/* and /mcp/* |
GatewayClass tars-egress | Envoy Gateway v1.7.x | Standard Kubernetes Gateway API |
| Kubernetes version | 1.32 to 1.35 for current releases | See the compatibility page |
Images are available for linux/amd64 and linux/arm64. The installation overview lists three install paths. The tare CLI bundles the chart and runs checks before install on EKS, AKS, and GKE. Plain Helm pulls the chart as an OCI artifact from your private registry. For GitOps, tare install renders the manifests so you commit them to the Argo CD or Flux repository instead of applying them by hand.
On EKS, the install uses your load balancer and your IAM roles
The AWS installation guide asks for a cluster of at least three nodes with OIDC enabled and the EBS CSI driver. The smallest setup is three m5.large nodes, and the largest documented setup is three m5.2xlarge nodes. Images are copied to ECR, and one tare install command deploys the data plane.
Making the gateway reachable from outside uses the tare gateway install command, which creates a Network Load Balancer through the EKS in-tree cloud provider. No AWS Load Balancer Controller or IAM policy work is needed for the default path. Then you add a CNAME to the load balancer hostname.
For Bedrock, the data plane can sign requests with the pod’s IAM identity through IRSA or EKS Pod Identity. No static AWS keys are stored in the cluster. One command binds the role: tare dataplane identity bind --type irsa --create-iam --cluster <name> --region <region>.
EKS may default to a Kubernetes version newer than the supported range, so fix the version at cluster creation. An existing cluster-wide Envoy Gateway install clashes with the gateway’s GatewayClass, so use a dedicated cluster.
Metrics and traces go to the monitoring tools you already run
The gateway exports traces over OTLP on gRPC port 4317 or HTTP port 4318 to Grafana, Datadog, New Relic, Honeycomb, or Jaeger. Each component also exposes a Prometheus scrape endpoint. Spans carry the model name, prompt and completion token counts, and the request ID, using OpenInference attribute names. A contentMode: metadata-only setting drops prompt content from traces. The setting is under Settings → OpenTelemetry Export in the Admin Console, or in Helm values.
Configuration can also be stored in Git. The configuration-as-code guide explains that policies, routing, budgets, the model catalog, and API keys are the same objects in the console, the API, and Helm values. Changes apply in under a minute without a restart. Set each value in one place only, because a GitOps tool undoes changes made by hand.
How Claude Code, Cursor, and other coding tools connect to the AI gateway
Every tool that accepts an OpenAI-compatible or Anthropic-compatible base URL can point at the gateway. The coding agents overview documents the current list:
| Tool | How the tool connects | Notes |
|---|---|---|
| Claude Code | tare integrate claude-code writes ANTHROPIC_BASE_URL and ANTHROPIC_API_KEY to ~/.claude/settings.json | Managed mode reaches every provider family through one profile. Passthrough mode keeps a Claude Max subscription as the billing path and still logs every request |
| Claude Desktop (Cowork and Code tabs) | tare integrate claude-cowork, or Developer Mode → Configure Third-Party Inference | Gateway mode only, no subscription passthrough. See the Claude Desktop post |
| Visual Studio Code | The Tetrate Agent Router Model Provider extension registers the gateway as a language model provider | One key serves the chat view, agent mode, and any extension that uses the Language Model API. Requires VS Code 1.106 or later |
| Cursor | Base URL override in the settings UI, plus a .cursorrules file so generated code targets the gateway | The docs cover generated code in detail and the settings override only in general terms |
| GitHub Copilot in VS Code, Copilot CLI | See the Microsoft Copilot table below | |
| Codex CLI, Aider, Cline, Continue, Coder | tare integrate <tool> writes the tool’s config file | See the tare CLI guide |
The tare integrate command finds installed tools and creates or reuses a gateway key. The command writes the config file with 0600 permissions, backs up the previous file, and makes one test call. A --dry-run flag shows the change without applying it.
Claude Code reaches every provider family through one gateway profile
Claude Code sends requests in the Anthropic Messages format. The gateway accepts that format on /v1/messages and translates the request for other providers. Since Agent Router release 0.3.0, one Claude Code profile reaches Anthropic, Claude on Vertex AI, OpenAI, Gemini, and self-hosted OpenAI-compatible models. No flag changes per model are needed. Your own tools and MCP servers work on every family, because the gateway converts the tool definitions for each backend.
| Detail | What happens | What to do |
|---|---|---|
| Large MCP setups on OpenAI or self-hosted models | Claude Code sends more tool definitions than OpenAI accepts | Set ENABLE_TOOL_SEARCH=true so Claude Code searches its tools instead of sending all of them |
A model is enabled but missing from the /model picker | Claude Code filters the list by its own naming rules | Start Claude Code with --model <name>. The model is reachable |
| Anthropic server-side tools | These tools run inside Anthropic only | Keep that work on Anthropic models |
| Switching provider family mid-session | Not tested with customers | Start a new session when you change family |
Passthrough mode covers Anthropic models only, because the subscription pays only for Anthropic. Passthrough fallback stays inside Anthropic, and the console shows this limit.
Which Microsoft Copilot products can route through an AI gateway
Copilot is the name of several Microsoft products, and each one treats an outside gateway differently. The Tetrate VS Code extension covers the developer tools. Azure AI Foundry covers the agent platforms.
| Microsoft product | Routes through the gateway? | How | Status |
|---|---|---|---|
| Azure AI Foundry Agent Service (prompt agents) | Yes | Foundry’s bring your own model connection sends agent traffic to your gateway endpoint first | Generally available since April 2026 |
| GitHub Copilot in VS Code (chat and agent mode) | Yes | The Tetrate VS Code extension registers the gateway as a language model provider. Microsoft’s custom endpoint setting covers the chat window only | Available today |
| GitHub Copilot CLI | Yes | Three environment variables, per the Copilot CLI custom models guide | Available today |
| Copilot Studio agents (MCP tools) | Yes | Add the Agent Router MCP profile URL as an MCP server in Copilot Studio | Available. Copilot Studio calls the URL from Microsoft’s cloud, so the gateway endpoint must be reachable from there. Test the auth setup in a short pilot |
| Copilot Studio agents (prompts and tools) | Through Azure AI Foundry | Copilot Studio calls a Foundry model, and the Foundry connection calls the gateway | Prove the Foundry step in a short pilot |
| Copilot Studio main model | No | Copilot Studio has no field for a gateway URL as the main model | Not possible today |
| Custom engine agents for Microsoft 365 | Yes | You bring your own orchestration and models, and the gateway routes the models behind the agent | Available. Requires developer work |
| Visual Studio 2026 (Copilot Agent) | Preview | Add an OpenAI provider with a custom URL in Agent (Preview), version 18.10 and later | Preview. Use VS Code for production |
| Microsoft 365 Copilot assistant in Word, Excel, Teams, and Outlook | No | Microsoft picks that model | Use a custom engine agent or a published Foundry agent instead |
Three settings decide whether the Foundry connection works. Pick the connection type Other source (called ModelGateway in the CLI and Bicep). Choose API Management only if you already handle model requests with API Management. Enter the base path only in the Base URL field, because Foundry adds chat/completions. Name the model as <connection-name>/<model-name> in the agent. Foundry supports the Other source connection for prompt agents in the Agent SDK only, and Microsoft does not support the Responses API on that connection.
For GitHub Copilot in VS Code, the extension stores the key in VS Code secret storage backed by the operating system keychain. The key never lands in settings.json. The approved model list is stored in the repository in .vscode/settings.json, so a team commits one list and every developer sees the same models. For Copilot Business and Enterprise, the organization admin turns bring-your-own-key on or off in Copilot policy settings, so confirm the policy before a rollout date.
Applications connect the same way as tools
An application built on the OpenAI SDK, the Anthropic SDK, LangChain, Vercel AI SDK, Pydantic AI, or CrewAI swaps the base URL and key. The integrations reference lists the full set. The one exception is a tool that hardcodes a provider SDK with no base URL override, because such a tool cannot use any gateway.
What the AI gateway does not support today
- GitHub Copilot inline code completions. GitHub runs inline completions.
- The Microsoft 365 Copilot assistant in Word, Excel, Teams, and Outlook on your own models.
- The gateway as the Copilot Studio main model.
- Visual Studio 2026 in production. The custom provider is in preview, and admin controls for the custom provider are not shipped.
- A cluster that already runs a cluster-wide Envoy Gateway.
Now Available
MCP Catalog with verified first-party servers, profile-based configuration, and OpenInference observability are now generally available in Tetrate Agent Router Service. Start building production AI agents today with $5 free credit.
Frequently asked questions
Can I install an AI gateway into my existing EKS cluster? Yes, if the cluster has at least three nodes, OIDC enabled, and a supported Kubernetes version. The install is a Helm chart or a set of rendered manifests. A cluster that already runs a cluster-wide Envoy Gateway needs a separate cluster.
Does the gateway need inbound access from the vendor? No. The data plane makes one outbound HTTPS connection to the management plane. No inbound firewall rules are required.
How do I connect Claude Code to the gateway?
Run tare integrate claude-code. The command writes the base URL and API key to the Claude Code settings file and makes a test call. A passthrough flag keeps a Claude Max subscription as the billing path.
Can Claude Code use OpenAI or Gemini models through the gateway? Yes. Claude Code sends Anthropic Messages requests, and the gateway translates them for OpenAI, Gemini, Claude on Vertex AI, and self-hosted OpenAI-compatible models. One profile covers every family. Anthropic server-side tools stay on Anthropic models.
Can GitHub Copilot use the gateway? GitHub Copilot can use the gateway for chat and agent mode in VS Code, through the Tetrate extension or Microsoft’s custom endpoint setting. The Copilot CLI can use the gateway through three environment variables. Inline code completions stay on GitHub’s servers.
Can Azure AI Foundry agents route through an AI gateway? Yes. Foundry’s bring-your-own-model connection, generally available since April 2026, sends prompt agent traffic to the gateway endpoint first. Pick the Other source connection type, enter the base path only, and name the model as connection-name/model-name.
Can Microsoft 365 Copilot run on my own models? No. The assistant in Word, Excel, Teams, and Outlook runs on models Microsoft picks. To use your own models inside Microsoft 365, build a custom engine agent and route the models behind that agent through the gateway.
Does the gateway work with Datadog or Grafana? Yes. Traces export over OTLP and metrics expose Prometheus scrape endpoints. A metadata-only setting keeps prompt content out of traces.
Can I manage gateway configuration in Git? Yes. Policies, routing, budgets, the model catalog, and keys can be set through Helm values and reconciled by Argo CD or Flux. Changes apply in under a minute.
Which Kubernetes versions does the AI gateway support? Current Agent Router data plane releases support Kubernetes 1.32 to 1.35. The release compatibility page lists the range for each release.
How does the gateway call Amazon Bedrock without static AWS keys?
The data plane signs Bedrock requests with the pod’s IAM role through IRSA or EKS Pod Identity. One tare dataplane identity bind command creates and binds the role, and no AWS keys are stored in the cluster.
Related reading
- Developer On-Ramp product page
- Adding Tetrate Agent Router as a model provider in Visual Studio Code
- Adding Tetrate Agent Router as a third-party inference gateway in Claude Desktop
- Custom engine agents for Microsoft 365 at Microsoft Learn
- Sizing and scale in the docs
- tare CLI reference in the docs
MCP Catalog with verified first-party servers, profile-based configuration, and OpenInference observability are now generally available in Tetrate Agent Router Service . Start building production AI agents today.
