Tetrate Solves a Critical TLS Issue for a Payment Services Company

Diagnosing the Issue

Under tight deadlines, the customer exposed the mTLS host mks-demo-service-engineer-tms-mks.ingress.cluster.partner.com. Soon after, they opened a high-severity support ticket, reporting a cryptic error when attempting to connect via curl:

$ curl -v --cacert ca.crt --cert rrqa.crt --key rrqa.key 'https://mks-demo-service-engineer-tms-mks.ingress.cluster.partner.com:8443'
Host mks-demo-service-engineer-tms-mks.ingress.cluster.partner.com:8443 was resolved.
IPv6: (none)
IPv4: xx.xxx.xxx.xx
Trying xx.xxx.xxx.xx:8443...
Connected to mks-demo-service-engineer-tms-mks.ingress.cluster.partner.com (xx.xxx.xxx.xx) port 8443
ALPN: curl offers h2,http/1.1
(304) (OUT), TLS handshake, Client hello (1):
CAfile: ca.crt
CApath: none
Recv failure: Connection reset by peer
LibreSSL/3.3.6: error:02FFF036:system library:func(4095):Connection reset by peer
Closing connection
curl: (35) Recv failure: Connection reset by peer

Developers were blocked from testing critical functionality, requiring immediate expert assistance. The error message, “Recv failure: Connection reset by peer“, was ambiguous and could have stemmed from various causes, including kernel bugs or firewall issues.

Solution

Step 1: Isolating the Problem

First, we analyzed the scope of the issue by checking other services in the environment. We discovered that all *.ingress.cluster.partner.com hosts were down, while others, like *.stage.dolphin.partner.com, were operational.

Next, we inspected the gateway’s Envoy configuration, looking for transport_socket errors at the listener filter level, such as incorrect cipher suites or TLS version mismatches. Cross-referencing with the working server names helped rule out misconfigurations in user settings.

Step 2: Checking the Client Side

Since server-side issues weren’t immediately apparent, we examined whether the curl client was contributing to the error. We tested a request using an explicitly allowed cipher suite:

$ curl -v --ciphers ECDHE-ECDSA-AES256-GCM-SHA384 --cacert ca.crt --cert rrqa.crt --key rrqa.key 'https://mks-demo-service-engineer-tms-mks.ingress.cluster.partner.com:8443'

The error persisted, albeit in a slightly different form:

OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to mks-demo-service-engineer-tms-mks.ingress.cluster.partner.com:8443

This ruled out client-side issues.

Step 3: Modifying Gateway Configuration

To further isolate the issue, we temporarily allowed plain HTTP traffic by modifying the Istio Gateway resource:

Before (HTTPS enabled):

- hosts:
    - '\*.ingress.cluster.partner.com'
  port:
    name: https
    number: 8443
    protocol: HTTPS
  tls:
    credentialName: envoy-ingress-cert
    minProtocolVersion: TLSV1_2

After (HTTP enabled for debugging):

- hosts:
    - '\*.ingress.cluster.partner.com'
  port:
    name: http
    number: 8443
    protocol: HTTP

This change allowed traffic, confirming that the issue was related to TLS.

Step 4: Investigating Certificates and Secrets

We validated the certificate using OpenSSL:

$ openssl verify cert-decoded.crt
cert-decoded.crt: OK

RESOURCE NAME                          TYPE     STATUS
kubernetes://envoy-system-cert         WARMING  false
kubernetes://envoy-ingress-cert        WARMING  false
kubernetes://envoy-sky-cert-cacert     WARMING  false

Three secrets were in a WARMING state instead of ACTIVE, indicating an issue with their configuration.

The Root Cause: A Simple Typo

A sharp-eyed Tetrate engineer discovered the culprit—an incorrect key name in the Kubernetes secret. The certificate data was stored in an opaque secret type, where the expected keys were cacert, key, and cert. However, instead of cacert, the secret used ca.cert.

For TLS secrets, Istio expects .crt and .key file extensions. This minor but critical discrepancy caused Istio to reject the secret, preventing TLS termination.

Resolution and Outcome

Once the incorrect key name was corrected, the issue was immediately resolved. The demo environment was restored within 24 hours, despite working across multiple time zones.

This case highlights the value of expert support in diagnosing complex service mesh issues. Tetrate Istio Subscription provides technical solutions and strategic guidance for building resilient microservice architectures. In this instance, Tetrate’s rapid response allowed the customer to successfully launched their demo on time. Additionally, tools included in TIS, like Tetrate Config Analyzer (TCA), could have been used in this scenario to proactively validate and optimize Istio configurations before deployment.