Production systems aren’t the only critical environments for companies, platform and development teams also rely on stable, secure infrastructure. In this blog post, you’ll learn how a multinational payment card services company leveraged Tetrate’s support expertise to resolve an unusual TLS error in an Istio ingress gateway—just in time for a high-stakes demo.
This company uses Istio and Envoy at its edge ingress to enforce TLS and mTLS encryption, enhancing security and ensuring compliance with U.S. federal regulations. However, just before a critical delivery deadline, the engineering team encountered an issue they couldn’t resolve, putting their launch timeline at risk. Tetrate Support stepped in to diagnose and fix the problem.
Diagnosing the Issue
Under tight deadlines, the customer exposed the mTLS host mks-demo-service-engineer-tms-mks.ingress.cluster.partner.com. Soon after, they opened a high-severity support ticket, reporting a cryptic error when attempting to connect via curl:
$ curl -v --cacert ca.crt --cert rrqa.crt --key rrqa.key 'https://mks-demo-service-engineer-tms-mks.ingress.cluster.partner.com:8443' Host mks-demo-service-engineer-tms-mks.ingress.cluster.partner.com:8443 was resolved. IPv6: (none) IPv4: xx.xxx.xxx.xx Trying xx.xxx.xxx.xx:8443... Connected to mks-demo-service-engineer-tms-mks.ingress.cluster.partner.com (xx.xxx.xxx.xx) port 8443 ALPN: curl offers h2,http/1.1 (304) (OUT), TLS handshake, Client hello (1): CAfile: ca.crt CApath: none Recv failure: Connection reset by peer LibreSSL/3.3.6: error:02FFF036:system library:func(4095):Connection reset by peer Closing connection curl: (35) Recv failure: Connection reset by peer
Developers were blocked from testing critical functionality, requiring immediate expert assistance. The error message, “Recv failure: Connection reset by peer“, was ambiguous and could have stemmed from various causes, including kernel bugs or firewall issues.
Solution
Step 1: Isolating the Problem
First, we analyzed the scope of the issue by checking other services in the environment. We discovered that all *.ingress.cluster.partner.com hosts were down, while others, like *.stage.dolphin.partner.com, were operational.
Next, we inspected the gateway’s Envoy configuration, looking for transport_socket errors at the listener filter level, such as incorrect cipher suites or TLS version mismatches. Cross-referencing with the working server names helped rule out misconfigurations in user settings.
Step 2: Checking the Client Side
Since server-side issues weren’t immediately apparent, we examined whether the curl client was contributing to the error. We tested a request using an explicitly allowed cipher suite:
$ curl -v --ciphers ECDHE-ECDSA-AES256-GCM-SHA384 --cacert ca.crt --cert rrqa.crt --key rrqa.key 'https://mks-demo-service-engineer-tms-mks.ingress.cluster.partner.com:8443'
The error persisted, albeit in a slightly different form:
OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to mks-demo-service-engineer-tms-mks.ingress.cluster.partner.com:8443
This ruled out client-side issues.
Step 3: Modifying Gateway Configuration
To further isolate the issue, we temporarily allowed plain HTTP traffic by modifying the Istio Gateway resource:
Before (HTTPS enabled):
- hosts:
- '*.ingress.cluster.partner.com'
port:
name: https
number: 8443
protocol: HTTPS
tls:
credentialName: envoy-ingress-cert
minProtocolVersion: TLSV1_2
After (HTTP enabled for debugging):
- hosts:
- '*.ingress.cluster.partner.com'
port:
name: http
number: 8443
protocol: HTTP
This change allowed traffic, confirming that the issue was related to TLS.
Step 4: Investigating Certificates and Secrets
We validated the certificate using OpenSSL:
$ openssl verify cert-decoded.crt cert-decoded.crt: OK
The certificate itself was correct, and the corresponding Kubernetes secret was in the right namespace. However, running istioctl proxy-config secrets <envoy pod> revealed a critical clue:
RESOURCE NAME TYPE STATUS kubernetes://envoy-system-cert WARMING false kubernetes://envoy-ingress-cert WARMING false kubernetes://envoy-sky-cert-cacert WARMING false
Three secrets were in a WARMING state instead of ACTIVE, indicating an issue with their configuration.
The Root Cause: A Simple Typo
A sharp-eyed Tetrate engineer discovered the culprit—an incorrect key name in the Kubernetes secret. The certificate data was stored in an opaque secret type, where the expected keys were cacert, key, and cert. However, instead of cacert, the secret used ca.cert.
For TLS secrets, Istio expects .crt and .key file extensions. This minor but critical discrepancy caused Istio to reject the secret, preventing TLS termination.
Resolution and Outcome
Once the incorrect key name was corrected, the issue was immediately resolved. The demo environment was restored within 24 hours, despite working across multiple time zones.
This case highlights the value of expert support in diagnosing complex service mesh issues. Tetrate Istio Subscription provides technical solutions and strategic guidance for building resilient microservice architectures. In this instance, Tetrate’s rapid response allowed the customer to successfully launched their demo on time. Additionally, tools included in TIS, like Tetrate Config Analyzer (TCA), could have been used in this scenario to proactively validate and optimize Istio configurations before deployment.
###
If you’re new to service mesh, Tetrate has a bunch of free online courses available at Tetrate Academy that will quickly get you up to speed with Istio and Envoy.
Are you using Kubernetes? Tetrate Enterprise Gateway for Envoy (TEG) is the easiest way to get started with Envoy Gateway for production use cases. Get the power of Envoy Proxy in an easy-to-consume package managed by the Kubernetes Gateway API. Learn more ›
Getting started with Istio? If you’re looking for the surest way to get to production with Istio, check out Tetrate Istio Subscription. Tetrate Istio Subscription has everything you need to run Istio and Envoy in highly regulated and mission-critical production environments. It includes Tetrate Istio Distro, a 100% upstream distribution of Istio and Envoy that is FIPS-verified and FedRAMP ready. For teams requiring open source Istio and Envoy without proprietary vendor dependencies, Tetrate offers the ONLY 100% upstream Istio enterprise support offering.
Need global visibility for Istio? TIS+ is a hosted Day 2 operations solution for Istio designed to simplify and enhance the workflows of platform and support teams. Key features include: a global service dashboard, multi-cluster visibility, service topology visualization, and workspace-based access control.
Get a Demo