Envoy AI Gateway Latency: Benchmarking Tetrate Agent Router Enterprise's Data Plane
An independent VMware/Broadcom benchmark measured the Envoy AI Gateway adding just ~2 ms of latency under enterprise LLM load.
Tetrate Team 
TL;DR: An independent benchmark published by Broadcom (VMware Cloud Foundation) measured the Envoy AI Gateway adding roughly 2 milliseconds of latency, about 0.01% overhead, under realistic enterprise LLM traffic. The Envoy AI Gateway is the open-source data plane behind Tetrate Agent Router Enterprise, so the result is a high-fidelity indicator of the latency enterprises can expect in production. The gateway was never the bottleneck; the throughput ceiling was set by GPU compute, not by the proxy.
Why AI gateway latency matters
An AI gateway sits between applications and large language models, enforcing routing, security, rate limiting, and cost controls on every request. Because LLM traffic uses long-lived, chunked HTTP/2 streams, a poorly engineered gateway can impose a “streaming tax”: latency overhead added while the proxy buffers continuous token streams. For interactive agentic and human workloads, that overhead directly degrades the user experience.
The question every platform team asks is simple: how much latency does the gateway itself add? An independent benchmark now answers it.
The benchmark: who ran it and how
Broadcom’s VMware Cloud Foundation team engineered a science-grounded validation of the Envoy AI Gateway and published it in June 2026. The setup mirrors a real-world Tetrate Agent Router Enterprise deployment: the Envoy AI Gateway ran on a VMware Kubernetes Service cluster and routed traffic between an external frontier model and an on-premises vLLM cluster of four NVIDIA H100 GPUs serving a 120-billion-parameter open-source model.
Rather than using simplistic synthetic load, the team modeled traffic on established queuing theory:
| Traffic type | Distribution | What it simulates |
|---|---|---|
| Agentic bursts | Markov-Modulated Poisson Process | Autonomous agents firing cascaded tool calls |
| CI/CD and terminal siege | Gamma (CV=1.5) | Large, irregular text dumps into context |
| Human pacing | Exponential | Knowledge workers reading, pausing, interacting |
To keep the load generator honest, conversations were unrolled offline into pre-serialized payloads and replayed as raw bytes, so the test measured the gateway rather than the test harness. A high-cardinality dataset of more than 20,000 unique sessions prevented artificially inflated cache hit rates from masking true compute cost.
The headline result: ~2 ms of gateway overhead
By isolating upstream latency (gateway to model and back) from total client-facing round-trip time, the benchmark measured the proxy’s own contribution.
| Metric | Value |
|---|---|
| Absolute gateway overhead | ~0.002 s (2 ms) |
| Gateway overhead percentage | ~0.01% |
| Upstream latency (model) | 14.291 s |
| Downstream latency (client-facing) | 14.293 s |
The multi-second figures are the model generating a full response. The gateway adds only 2 ms of that total, and it stays flat even under peak saturation.
Interactive latency stayed fast under sustained load
During a three-hour endurance run at 190 concurrent users, latency stayed well inside interactive SLOs:
| Metric | Average |
|---|---|
| Time-to-first-token (TTFT) | 0.103 s |
| Time-per-output-token (TPOT) | 0.035 s |
| Inter-token latency (ITL) | 0.035 s |
| Queue time | < 0.001 s |
| Prefill time | 0.074 s |
A blended TTFT of 0.103 seconds means the experience stays instantaneous for users, even while the gateway mediates aggressive agentic bursts in the background.
Where the ceiling actually was
The system reached its saturation point at 224 concurrent users, where time-to-first-token climbed sharply. The important detail: this ceiling was a property of the GPU cluster, not the gateway. KV cache preemption stayed at zero, proving the limit was GPU compute throughput, not gateway capacity or memory. The gateway’s overhead remained flat at ~2 ms throughout. In production, that ceiling moves with the size of the GPU fleet behind the gateway.
The operational lesson: pin proxy concurrency
The benchmark surfaced a deployment trap worth its own line item. Early runs showed an alarming 3.5 to 6.2 seconds of apparent latency per request. The cause was not the proxy design. It was a Linux Completely Fair Scheduler throttling trap: Envoy spawned worker threads to match the node’s 64-plus CPU cores while its Kubernetes container quota was far smaller, starving the proxy of CPU cycles.
The fix was to pin the proxy to its container CPU limits with the --concurrency flag, which eliminated the overhead entirely. Any team running an Envoy-based AI gateway on Kubernetes should align the --concurrency flag with allocated CPU limits to avoid this trap.
Does gateway latency actually affect user experience?
Some AI gateways market single-digit microseconds against a mock backend under steady load. Those can be strong engineering results, but a mock-backend test and a production-like validation are not the same measurement. The VMware run used real GPU inference, bursty agentic and human traffic, and a high-cardinality dataset — the useful question is whether the overhead gap is one a user can perceive.
In this benchmark, full responses averaged around 14 seconds and TTFT was 0.103 seconds. Gateway overhead is a rounding error against that budget:
| Component of a request | Typical magnitude |
|---|---|
| Model generation (full response) | seconds |
| Time-to-first-token | ~100 milliseconds |
| Gateway overhead (Envoy AI Gateway, this benchmark) | ~2 milliseconds |
| Gateway overhead (microsecond-class gateways) | fractions of a millisecond |
A user cannot feel 2 ms versus 0.05 ms when the model takes 14,000 ms. Microsecond efficiency matters mainly for extremely high-throughput pipelines or non-LLM proxy hops. For enterprise LLM and agent workloads, once overhead is below the perception threshold — and ~2 ms is — what shapes experience is flat latency under load, clean failover, and security and cost controls on the same data path. The benchmark’s most useful finding is not that overhead is small; it is that it stays flat and predictable to the compute ceiling.
What this means for Tetrate Agent Router Enterprise
Tetrate Agent Router Enterprise is an enterprise AI gateway and agent routing platform built on the Envoy AI Gateway data plane. Because this benchmark validated that exact data plane under production-like conditions, enterprises evaluating Tetrate Agent Router Enterprise can rely on independent, third-party evidence that the foundation adds negligible latency.
Tetrate Agent Router Enterprise builds on that foundation with capabilities the benchmark did not test, including multi-provider routing and failover, token-aware budgets and FinOps chargeback, zero-trust credential injection, and agent-aware routing for multi-agent systems.
Take the next step
If you are evaluating an AI gateway for enterprise LLM and agent workloads, Tetrate Agent Router Enterprise pairs this validated low-latency data plane with the routing, security, and cost controls production teams need. Contact the Tetrate team to discuss a proof of concept.
Frequently asked questions
How much latency does the Envoy AI Gateway add? An independent Broadcom benchmark measured approximately 2 milliseconds of overhead, roughly 0.01%, under realistic enterprise LLM traffic, and the overhead stayed flat at peak saturation.
What is the “streaming tax” in an AI gateway? The streaming tax is latency overhead a proxy adds while buffering the long, chunked token streams that LLM responses produce. The benchmark showed the Envoy AI Gateway effectively eliminates it once concurrency is pinned correctly.
What is the data plane behind Tetrate Agent Router Enterprise? Tetrate Agent Router Enterprise is built on the open-source Envoy AI Gateway, the same data plane validated in this benchmark.
Was the gateway the performance bottleneck? No. The benchmark’s throughput ceiling at 224 concurrent users was caused by GPU compute limits, not the gateway. KV cache preemption stayed at zero, confirming a compute-bound rather than gateway-bound limit.
What was the benchmark hardware? A VMware Kubernetes Service cluster on VMware Cloud Foundation, with a backend of four NVIDIA H100 GPUs serving a 120-billion-parameter open-source model through vLLM.
How do I avoid latency spikes with Envoy on Kubernetes?
Pin the proxy’s --concurrency flag to the container’s allocated CPU limits. The benchmark traced multi-second latency spikes to a Linux scheduler throttling trap that this fix resolves.
Does a few microseconds of gateway overhead matter for user experience? Rarely, for LLM workloads. Model generation takes seconds and time-to-first-token is around 100 milliseconds, so the difference between a 2-millisecond gateway and a microsecond-class one falls far below human perception. What affects experience more is whether overhead stays flat under load, fails over cleanly, and enforces security and cost controls without a second data path. Raw microsecond overhead matters mainly in extremely high-throughput or non-LLM proxy scenarios.
Tetrate Team 
