AI Gateway Glossary: Key Terms for Enterprise AI Infrastructure

Last updated: October 2026

Clear, neutral definitions of the core concepts in enterprise AI infrastructure — from the gateway layer to the protocols, controls, and governance that surround it. Each entry links to a full explanation.

Gateways & routing

What is an AI gateway? — The infrastructure layer between applications or agents and AI model providers, centralizing routing, security, cost, and governance for AI traffic.
What is an LLM gateway? — A routing and management layer between applications and large language model providers; closely related to, and often used interchangeably with, an AI gateway.
What is an inference gateway? — A layer that routes and manages traffic to model inference endpoints, often self-hosted or in-cluster, with model-aware load balancing.
AI gateway vs. API gateway — How an AI gateway differs from a traditional API gateway, and when you need each.
What is an AI ingress gateway?: The entry point for requests coming into AI agents and self-hosted models, with caller identity and token limits at the entry point.
What is an AI egress gateway?: The control point for calls leaving a cluster for model providers and MCP servers, with destination rules, provider key handling, and token counts per team.
What is LLM routing?: Choosing a model for each AI request at run time, based on the task, cost, latency, or policy.
What is smart routing?: Routing in Tetrate Agent Router Enterprise that picks a model for each request from an approved catalog, using ordered tag rules, region rules, or a decision model, with policy applied first and a decision record kept per request.
What is data residency routing?: Routing that keeps LLM requests, fallbacks, and router choices inside an allowed region.

Cost & measurement

What is cost per successful task?: Total spend for a piece of AI agent work divided by the number of tasks that pass a defined check.

Governance & sovereignty

What is sovereign AI? — Control over where models run, where data flows, and which jurisdiction’s rules apply — across data, model, operations, compute, and compliance.

Agent Router Enterprise

Tetrate Agent Router Enterprise routes AI agent traffic across providers and your own models, with smart routing, policy, cost controls, and audit on every request, in cloud, on-prem, or edge.

Learn more

Smart routing — model choice, attribute-based routing, and Tetrate-auto in Agent Router Enterprise.
Best Enterprise AI Gateways 2026 — a comparison of the leading AI gateways.
Your AI bill is an AI gateway problem — routing, caching, and showback/chargeback at the gateway.
Who created Envoy AI Gateway? — the provenance of the open-source AI gateway Tetrate co-created.
AI Gateway benchmarks — published performance methodology and results.

This glossary is maintained by Tetrate, an AI infrastructure company and co-creator of Envoy AI Gateway. Explore Tetrate Agent Router Enterprise.

Learn more about Tetrate Agent Router Enterprise — enterprise AI agent routing with smart routing, policy, cost controls, and audit across every gateway.