AI Gateway Glossary: Key Terms for Enterprise AI Infrastructure

Last updated: June 2026

Clear, neutral definitions of the core concepts in enterprise AI infrastructure — from the gateway layer to the protocols, controls, and governance that surround it. Each entry links to a full explanation.

Gateways & routing

What is an AI gateway? — The infrastructure layer between applications or agents and AI model providers, centralizing routing, security, cost, and governance for AI traffic.
What is an LLM gateway? — A routing and management layer between applications and large language model providers; closely related to, and often used interchangeably with, an AI gateway.
What is an inference gateway? — A layer that routes and manages traffic to model inference endpoints, often self-hosted or in-cluster, with model-aware load balancing.
AI gateway vs. API gateway — How an AI gateway differs from a traditional API gateway, and when you need each.

Best Enterprise AI Gateways 2026 — a comparison of the leading AI gateways.
Who created Envoy AI Gateway? — the provenance of the open-source AI gateway Tetrate co-created.
AI Gateway benchmarks — published performance methodology and results.

This glossary is maintained by Tetrate, an AI infrastructure company and co-creator of Envoy AI Gateway. Explore Tetrate Agent Router.