Doubling Envoy Throughput on NVIDIA BlueField
We modified Envoy to offload TLS cryptography to NVIDIA BlueField-3 hardware accelerators. On a single core with AES-256 and a 5 MB payload, throughput went from 6.6 Gbps to 13.6 Gbps — 2x, without touching L7 policy.
David Wang and Liam Byrne 
Encryption is no longer optional. Internal traffic is encrypted, service-to-service traffic is encrypted, and the payloads keep getting larger as AI workloads push model weights, embeddings, and long responses through the same proxies that used to carry small JSON. Most teams add nodes to keep up. But that work is landing in the wrong place.
Symmetric cryptography runs on general-purpose Arm cores that should be running routing and policy, and it scales with every encrypted byte, so it crowds out that policy work and forces more nodes to keep up. NVIDIA BlueField already offloads networking, storage, and security to free host resources for AI workloads, and its hardware can take the symmetric-crypto work too. The question is whether an L7 proxy can use that offload without giving up the routing, policy, and observability it was deployed for.
We modified Envoy to offload TLS cryptography to the NVIDIA BlueField-3 hardware accelerators instead of running it in software on the DPU’s Arm cores. On a single core with AES-256 and a 5 MB payload, throughput went from 6.6 Gbps to 13.6 Gbps. That is 2x, and it came without touching any of Envoy’s L7 policy or changing anything an operator configures.
Where the crypto tax lands
An L7 proxy terminating TLS pays for the symmetric crypto on every byte of every connection, for the life of that connection. Running that crypto in software, it competes for cycles with the routing, rate limiting, JWT validation, and WAF logic that the proxy exists to run. The larger the payload, the more of the CPU budget goes to crypto rather than policy.
NVIDIA BlueField can offload this symmetric-crypto work to hardware, but a stock Envoy doesn’t use it and runs the crypto in software anyway. We wanted to move that work off the Arm cores and measure what they got back.
What we built
Envoy runs on the BlueField-3 Arm cores. Upstream Envoy ships with BoringSSL and runs all TLS on those cores. Our build is based on the envoy-openssl fork and is further optimized for the BlueField DPU: it enables kTLS so the AES symmetric crypto can be handed to the DPU’s hardware offload, statically links OpenSSL, and adds a few smaller performance tweaks. All L7 policy continues to run on the Arm cores. The offload handles the per-byte cipher work; Envoy handles routing, policy, and observability, exactly as before.
The change is transparent to the control plane. Operators configure routes and filters the way they always have. Nothing above the data plane is aware the crypto moved. This matters for adoption: an existing Envoy deployment gets the offload without being re-architected, and the people running it don’t learn anything new.
How we measured it
Envoy ran on the BlueField-3 Arm cores (B3240: 16 Cortex-A78 cores, 32 GB RAM, 400 GbE), on a DPU imaged to Ubuntu 24.04 with NVIDIA DOCA 3.2.1. We compared two server configurations:
- Baseline: upstream Envoy with BoringSSL, all TLS on the Arm cores.
- Accelerated: our BlueField-optimized envoy-openssl build with NIC hardware kTLS offload.
Both ran the same routes, the same policy, and the same traffic. The only difference was whether the AES crypto ran in software on the Arm cores or was offloaded to the NIC hardware.
Load came from Graviton clients (scaling up to a c8gn.8xlarge, ~77 Gbps measured) running Fortio against Envoy, with concurrency tuned for maximum throughput. We recorded Fortio’s calls, average latency, and QPS, derived Gbps from those, and captured DPU CPU with mpstat.
TLS conditions were fixed across runs: TLS 1.2 with a 4096-bit RSA server key, ciphers ECDHE-RSA-AES128-GCM-SHA256 and ECDHE-RSA-AES256-GCM-SHA384. We used fixed direct-response payloads from 1 KB to 10 MB to isolate TLS cost from any origin behavior. Results below report AES-256, at 1 core and 16 cores, across five L7 configurations: no extra filters, local rate limit, JWT validation, rate limit plus JWT, and WAF. The testbed topped out at roughly 77 Gbps of network bandwidth, which matters for the 16-core results below.
Bigger payloads, bigger gains
Crypto cost scales with payload size, so the offload’s value does too. At a single core with AES-256:
| Payload | Baseline: Upstream Envoy | Accelerated: Envoy + NVIDIA BlueField with Hardware kTLS | Gain |
|---|---|---|---|
| 2 MB | 5.8 Gbps | 11.3 Gbps | +93% |
| 5 MB | 6.6 Gbps | 13.6 Gbps | +106% |
Below about 500 KB the gain is small: there isn’t enough crypto work to be worth offloading. The heavier the payload, the more the Arm cores were spending on crypto, and the more there is to reclaim.
L7 policy keeps working
Offloading crypto raises an obvious question: does it cost you the policy features that justify running an L7 proxy at all? It doesn’t. With a 1 MB payload on a single core, every L7 configuration kept its advantage.
| L7 Configuration | Throughput Improvement |
|---|---|
| No extra filters | +46.7% |
| Local rate limit | +58.5% |
| JWT validation | +44.5% |
| Rate limit + JWT | +34.9% |
| WAF | +11.2% |
WAF is the low number. It inspects request bodies in software on the Arm cores, and that work was never crypto-bound, so offloading the crypto can’t speed it up. An 11% gain on a workload dominated by body inspection is what the architecture predicts.
AES-256 costs the same as AES-128
With kTLS offload, AES-256 runs at the same throughput as AES-128. In software on the Arm cores, the stronger cipher costs throughput; offloaded to hardware, it doesn’t.
For anyone under FIPS, FedRAMP, or financial-services rules that mandate stronger ciphers, that removes a trade-off: AES-256 no longer costs throughput.
CPU utilization: a third fewer cores
Throughput is the visible number; CPU utilization is the one that drives node count. At equivalent request rates with AES-256 on a single core, the accelerated path used far less of the Arm core:
| Payload | CPU Utilization Reduction |
|---|---|
| 100 KB | -37% |
| 200 KB | -36% |
| 500 KB | -28% |
Latency improved alongside it, with p99 gains of +14.7%, +23.6%, and +25.5% across those payloads. The tail moved more than the median, which is where latency-sensitive traffic feels it.
That reclaimed headroom translates directly: a third fewer nodes for the same workload, more room to absorb a traffic spike before SLAs break, or budget to run more L7 policy — WAF, more filters, richer observability — on the same hardware.
At 16 cores, the story changes
At 16 cores the throughput gains compress. The testbed hit a network ceiling near 77 Gbps, and with many cores to spread crypto across, host parallelism masks per-core savings. The non-WAF L7 configurations still held 19–22% gains; WAF held +10.4%.
At this core count the value is CPU headroom rather than raw throughput: you’re no longer crypto-bound, you’re reclaiming cores. The doubled-throughput result holds at low core counts; the efficiency gains hold at every core count we tested.
What this buys you
The result is the same offload thesis BlueField is built on, now extended to L7. Crypto moves to the DPU hardware, the Arm cores go back to routing and policy, and the AI factory runs the same encrypted traffic on fewer nodes, with stronger ciphers, at no performance cost.
What’s next
Next, we go deeper into the platform. We want to rerun the 16-core test without the network bottleneck to see the real throughput gain. We’re extending the offload by adding direct hardware acceleration via the NVIDIA XLIO library. More results to come.
David Wang and Liam Byrne 
