What Is an AI Ingress Gateway?
TL;DR: Agent Router for Ingress puts Tetrate Agent Router Enterprise controls at the front door of your agents and self-hosted models, so unknown callers cannot spend your GPU time.
An AI ingress gateway is the entry point for requests coming into your AI agents and self-hosted models. The gateway checks who is calling and how many tokens the caller can spend before the request reaches your agent or your GPUs.
A self-hosted model with a direct Service endpoint accepts work from any caller that can reach the endpoint. One caller can send long prompts all day. Your GPU queue fills, other callers wait, and nothing warns you which caller caused the wait.
Tetrate Agent Router Enterprise and Agent Router for Ingress put a gateway in front of your agents and models. The gateway names each caller, applies token and rate limits per caller, and records each request. AI ingress vs. AI egress covers both directions in one model.
AI Ingress Differs From Web Ingress in Four Ways
| Difference | Web ingress | AI ingress |
|---|---|---|
| Token cost per request | Each request costs about the same to serve | One request can cost a few tokens or a hundred thousand tokens |
| Streamed responses | Most responses end in one short transfer | Model answers stream token by token and can run for minutes |
| Prompt payloads | Requests carry form fields and small JSON bodies | Prompts carry files, code, and personal data that need checks |
| GPU-backed backends | Backends scale with CPU pods | Backends run on scarce GPUs with queues and batching |
Web ingress routes by host and path. Web ingress counts requests. AI ingress must also price each request in tokens, read streamed answers, inspect prompt payloads, and protect GPU capacity. AI gateway vs. API gateway gives the longer comparison.
A request counter treats a ten-token answer and a hundred-thousand-token answer as equal work. Your GPU bill treats the two answers as very different work. Token-aware limits at ingress match the limit to the cost.
Two limits. An AI ingress gateway can limit and route requests, and the gateway cannot add GPU capacity. A caller surge still needs autoscaling, queue policy, or more hardware. The gateway also sees the caller key on each request. Callers that share one key share one limit and one audit identity.
Controls an AI Ingress Gateway Applies
The gateway authenticates callers with API keys or tokens from your identity provider. The gateway rejects unknown callers before the request reaches an agent or a model.
The gateway applies token limits and rate limits per caller. A caller that reaches the limit gets a clear error. Other callers keep their capacity.
The gateway routes by model name. Callers ask for a model, and the gateway sends the request to a healthy backend that serves the model. The AI gateway performance benchmarks publish measured gateway overhead.
The gateway applies prompt and response checks. Guardrails can block or redact personal data in prompts. Audit records name the caller, the model, and the token count for each request.
The gateway reports usage per caller. Your platform team can see which caller spends GPU time and can set chargeback or showback from gateway records. LLM cost chargeback covers the finance side.
Ingress for Self-Hosted Inference Uses the Gateway API Inference Extension
Self-hosted models need routing that understands inference work. Requests differ in prompt length and cost. Backends differ in load and in the model each backend serves.
The Gateway API Inference Extension adds inference-aware routing to Gateway API. An inference gateway can route to model backends with load signals from the backends. What is an inference gateway? explains the inference gateway pattern and the relation to AI gateways.
Your team still needs caller identity, token limits, and audit at the same entry point. A routing rule that picks a backend does not name the caller. The AI ingress gateway adds caller controls at the entry point and keeps inference routing behind the entry point.
Moving AI Workloads Off Ingress NGINX
Kubernetes announced the retirement of Ingress NGINX. Best-effort maintenance ends in March 2026, and the project receives no new releases after the retirement date. Teams that run Ingress NGINX need a new ingress path.
AI workloads give your team a reason to pick an AI-aware ingress path during the migration. You can move web routes to a Gateway API ingress and put agents and self-hosted models behind an AI ingress gateway. The Ingress NGINX to Envoy Gateway migration guide covers the Gateway API migration steps.
Agent Router for Ingress is the Tetrate answer for AI ingress. The data plane runs in your cluster, close to your agents and your GPUs. Tetrate manages the gateway lifecycle, and your team sets caller policy, model routing, and limits. Outbound calls from your agents need the other direction as well, and what is an AI egress gateway? covers the outbound controls. The AI gateway glossary defines both terms.
Agent Router Enterprise
Tetrate Agent Router Enterprise routes AI agent traffic across providers and your own models, with policy, cost controls, and audit on every request — in cloud, on-prem, or edge.
Tetrate co-created the open source data plane project, and who created Envoy AI Gateway covers the project history. Teams in an active evaluation can compare options in best enterprise AI gateway.
Availability and How to Get Started
Agent Router for Ingress is available for AI workloads on Kubernetes, with the data plane in your cluster and lifecycle management from Tetrate. Tetrate Agent Router Enterprise covers outbound model and MCP traffic from the same platform. Read the Agent Router Enterprise docs on gateway request authentication and book a demo to test ingress controls on your own models.
Frequently Asked Questions
What Is an AI Ingress Gateway?
An AI ingress gateway is the entry point for requests coming into your AI agents and self-hosted models. The gateway checks who is calling and how many tokens the caller can spend before the request reaches your agent or your GPUs.
How Does AI Ingress Differ From Web Ingress?
AI requests carry token costs that vary by request. Model answers stream over time. Prompts can carry personal data. Backends run on scarce GPUs. Web ingress routes by host and path and counts requests, and web ingress has no token model.
What Controls Does an AI Ingress Gateway Apply?
An AI ingress gateway applies caller authentication, token limits, rate limits, model routing, prompt checks, and audit records. Each control applies per caller or per key.
What Is the Gateway API Inference Extension?
The Gateway API Inference Extension adds inference-aware routing to Gateway API. The extension helps a gateway pick a model backend with signals such as backend load and the model each backend serves.
When Does Ingress NGINX Retire?
Kubernetes announced that best-effort Ingress NGINX maintenance ends in March 2026. After March 2026, the project gets no new releases, no bug fixes, and no security updates.
Can One Product Cover AI Ingress and AI Egress?
Yes. Agent Router for Ingress covers requests coming into agents and self-hosted models. Tetrate Agent Router Enterprise covers calls leaving your cluster for model providers and MCP servers.
Sources
- Ingress NGINX retirement, Kubernetes blog
- Authenticate gateway requests with an external IdP, Tetrate docs
- Gateway API Inference Extension
Related: What is an inference gateway?, on routing and policy for self-hosted model endpoints.
