This is an ongoing series of Istio and Envoy security updates from Tetrate. Subscribe to our newsletter to get notified as we release each bulletin.
On October 10th, 2023, the Envoy Security Team disclosed CVE 2023-44487 Envoy HTTP/2 “Rapid Reset” DDos attack. This Envoy HTTP/2 vulnerability is a high-severity and exploitable vulnerability that impacts all Istio and Envoy users that have internet-facing gateways. Tetrate recommends that all Istio and Envoy users upgrade to the next patch release as soon as a fix is available. Tetrate Istio Distro (TID) users should register to receive instructions on how to access backported fixes for Istio 1.15 and 1.16.
If you are a Tetrate Istio Subscription (TIS) customer, Tetrate has been working on remediations. Updated Istio 1.17, 1.18, and 1.19 are already available in your repo, we are also finalizing backport remediations for Istio 1.15 and 1.16. Expect updates from Tetrate to your CVE contacts by email in the coming days.
If you are also a Tetrate Service Bridge (TSB) or Tetrate Service Express (TSE) customer, you can also expect updates from Tetrate to your CVE contacts by email in the coming days as we finalize their releases.
In this security bulletin, we’ll share an overview of the Envoy HTTP/2 vulnerability, our remediation recommendations, and how to obtain the Istio patches.
What Is the Envoy HTTP/2 Vulnerability?
This vulnerability affects multiple products since it originates in a common library, nghttp2. Envoy & Istio are using the designation CVE 2023-44487. We also know that this vulnerability is being actively exploited by malicious actors.
The most basic form of attack sees malicious users submit a high number of specially crafted requests, starving legitimate connections of CPU and causing either elevated latencies or request timeouts and subsequently a denial of service.
In more complicated production scenarios, the attack may cause request timeouts to the sidecar services, and the abusive requests may reach backend services. This happens when Envoy is configured to fail open when a sidecar service times out, with the intention of preventing a sidecar service outage from impacting user traffic. This may lead to malicious users gaining access to systems that would otherwise be protected by sidecars such as Web Application Firewalls and external authorization.
The attack can be detected by observing elevated downstream_rq_http2_total or downstream_rq_http3_total counters without a substantial increase in the downstream_cx_active counter. In some known forms of the attack, the value of downstream_rq_rx_reset will be elevated, while during other known forms of attack the downstream_rq_5xx or downstream_rq_4xx will be elevated.
What Should Affected Users Do?
Given the severity and exploitability of this vulnerability, we recommend all affected users plan an upgrade to fixed versions as soon as possible for all internet-facing systems.
If you are running Istio 1.17 or newer, an updated patch release of Tetrate Istio Distro is available already. You can register for installation instructions and sign up for future security bulletins. Since Istio 1.17 or newer is still under community support, you can also get updates directly from the community.
If you are running Istio 1.15 or 1.16, Istio community support has expired. You should upgrade to the next minor version, or backport the CVE fix yourself if you are not able to perform a minor upgrade at this time. Note that community support for Istio 1.17 is set to expire October 27th, 2023.
How to Remediate Envoy HTTP/2 Vulnerability for Istio 1.15 and 1.16
Tetrate Istio Subscription offers extended CVE support for users to stay on the same Istio minor version longer. Extended CVE support allows users to avoid backporting CVE fixes, provide more stability for their end customers, and avoid an unplanned minor version upgrade. TIS customers will receive backported Istio CVE fixes for 14 months (instead of 8 months from the Istio community).
If you are already a Tetrate Istio Subscription customer, patched Istio 1.17-1.19 is already available in your repo. If you are running 1.15 or 1.16, Tetrate is finalizing the backport releases, and your CVE contact will be notified by email as soon as a release is available.
What’s Next
Envoy HTTP/2 is a serious vulnerability and you should plan your next upgrade accordingly.
For free Tetrate Istio Distro users, please register immediately to receive future security bulletins and installation instructions for Envoy HTTP/2 remediations.
If you are running Istio 1.15 or 1.16 and require a backported Envoy HTTP/2 fix, please contact us to learn more about Tetrate Istio Subscription.
If you are running Istio 1.17 and are concerned about the impending end of community support on October 27, you can learn more about TIS extended support and contact us to subscribe to TIS.
###
If you’re new to service mesh, Tetrate has a bunch of free online courses available at Tetrate Academy that will quickly get you up to speed with Istio and Envoy.
Are you using Kubernetes? Tetrate Enterprise Gateway for Envoy (TEG) is the easiest way to get started with Envoy Gateway for production use cases. Get the power of Envoy Proxy in an easy-to-consume package managed by the Kubernetes Gateway API. Learn more ›
Getting started with Istio? If you’re looking for the surest way to get to production with Istio, check out Tetrate Istio Subscription. Tetrate Istio Subscription has everything you need to run Istio and Envoy in highly regulated and mission-critical production environments. It includes Tetrate Istio Distro, a 100% upstream distribution of Istio and Envoy that is FIPS-verified and FedRAMP ready. For teams requiring open source Istio and Envoy without proprietary vendor dependencies, Tetrate offers the ONLY 100% upstream Istio enterprise support offering.
Need global visibility for Istio? TIS+ is a hosted Day 2 operations solution for Istio designed to simplify and enhance the workflows of platform and support teams. Key features include: a global service dashboard, multi-cluster visibility, service topology visualization, and workspace-based access control.
Get a Demo