AI Gateway Glossary: Key Terms for Enterprise AI Infrastructure

Last updated: July 2026

Clear, neutral definitions of the core concepts in enterprise AI infrastructure — from the gateway layer to the protocols, controls, and governance that surround it. Each entry links to a full explanation.

Gateways & routing

What is an AI gateway? — The infrastructure layer between applications or agents and AI model providers, centralizing routing, security, cost, and governance for AI traffic.
What is an LLM gateway? — A routing and management layer between applications and large language model providers; closely related to, and often used interchangeably with, an AI gateway.
What is an inference gateway? — A layer that routes and manages traffic to model inference endpoints, often self-hosted or in-cluster, with model-aware load balancing.
AI gateway vs. API gateway — How an AI gateway differs from a traditional API gateway, and when you need each.

Governance & sovereignty

What is sovereign AI? — Control over where models run, where data flows, and which jurisdiction’s rules apply — across data, model, operations, compute, and compliance.

Agent Router Enterprise

Tetrate Agent Router Enterprise routes AI agent traffic across providers and your own models, with policy, cost controls, and audit on every request — in cloud, on-prem, or edge.

Learn more

Best Enterprise AI Gateways 2026 — a comparison of the leading AI gateways.
Your AI bill is an AI gateway problem — routing, caching, and showback/chargeback at the gateway.
Who created Envoy AI Gateway? — the provenance of the open-source AI gateway Tetrate co-created.
AI Gateway benchmarks — published performance methodology and results.

This glossary is maintained by Tetrate, an AI infrastructure company and co-creator of Envoy AI Gateway. Explore Tetrate Agent Router Enterprise.

Learn more about Tetrate Agent Router Enterprise — enterprise AI agent routing with policy, cost controls, and audit across every gateway.