Google's HTTP/2 codec slows Envoy

A routine Envoy upgrade to version 1.34 moved the default HTTP/2 codec from the nghttp2 library to Google's oghttp2 implementation. After the upgrade, CPU usage on a customer's dedicated proxy rose by about one fifth with no configuration or traffic changes. Investigators bisected versions and traced the regression to the codec switch, noting that users had already reported 15-25% latency increases since v1.34. The default was flipped back to nghttp2 in v1.37.0, with a source comment indicating another attempt would be made once performance matched nghttp2.

To quantify the difference, the team benchmarked both codecs on four microarchitectures: Intel Sapphire Rapids, AMD Zen 4, AWS Graviton4, and Google Axion. The test used same-host loopback traffic, either h2load driving Envoy to a Go h2c backend or Envoy generating direct responses, with each process pinned to disjoint physical cores and Envoy run at --concurrency 1. Under these conditions, nghttp2 consistently outperformed oghttp2 by 15-25% in requests per second per core for header-heavy proxied traffic, and by 7-19% under heavy connection churn. The results were reproducible across all tested hosts, confirming a portable performance gap.

Flamegraphs for both codecs showed that the dominant workload is header decompression, which occurs twice per hop as the proxy decodes incoming HPACK-compressed headers and re-encodes them for outgoing traffic. HPACK combines a static index table of 61 common header entries with a dynamic table that tracks recently seen fields, allowing repeated headers to be sent as one or two bytes. Header values that cannot be found in either table are transmitted as literal strings encoded with Huffman coding, a method that assigns shorter bit patterns to more frequent symbols. Because edge proxies see many fresh connections and many unique values such as request IDs or rotating tokens, the Huffman path becomes a hot spot in the codec's execution.

The investigation also highlighted that build configuration can dramatically affect measured performance; a plain Bazel build without the -c opt flag produces a debug binary that runs 15-32× slower per core than an optimized build, potentially masking or exaggerating codec differences. Engineers should therefore verify that Envoy is compiled with optimization flags when evaluating codec performance. The authors plan a follow-up effort to improve oghttp2's speed, aiming to close the gap with nghttp2.