Automate Istio CA rotation in production at scale

One of Istio’s core capabilities is to facilitate a zero trust network architecture by managing identity for services in the mesh. To retrieve valid certificates for mTLS communication in the mesh, individual workloads issue a certificate signing request (CSR) to istiod. Istiod, in turn, validates the request and uses a certificate authority (CA) to sign the CSR to generate the certificate. By default, Istio uses its own self-signed CA for this purpose, but best practice is to integrate Istio into your existing PKI by creating an intermediate CA for each Istio deployment.

If you’re managing multiple clusters, that means issuing multiple intermediate CAs, each of which should be set to expire within a few months or sooner. Managing the lifecycle of these CAs is critical, since they must be rotated before they expire or bad things happen. This article will show you how to make this CA management easier to reduce risk and increase the overall stability of your system.

A critical step when rotating the CA is to ensure that the new one is actually used. By default Istio only loads its CA on startup. But, Istio can be configured to watch for changes and automatically  reload its CA when it is updated. This tutorial, drawn from production playbooks we’ve developed working with our enterprise customers who are managing large numbers of Istio deployments, will show how to configure Istio to automatically reload its CA. We will also cover how to configure cert-manager to automatically rotate Istio’s intermediate CA on a regular schedule ahead of expiry to increase the operational efficiency of managing CAs on multiple clusters.

Prerequisites

You’ll need at least the following for the tutorial:

• A running Kubernetes cluster. A simplified Kubernetes install like minikube or similar is suitable for demo purposes;
• Istioctl v1.14.2 or higher;
• Cert-manager v1.7.2 or higher.

Task A: Install and configure cert-manager to automatically rotate Istio’s CA

Step A1: Install cert-manager

The following command will install cert-manager in your cluster. To install a newer version of cert-manager, change the github URL.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/downlo

Step A2: Configure the CA

For demo purposes, we will set up a self-signed CA, but DO NOT USE A SELF-SIGNED CA IN PRODUCTION. For production purposes, you should configure cert-manager to use your existing PKI.

cat << EOF | kubectl apply -f -
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned
  namespace: cert-manager
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: selfsigned-ca
  namespace: cert-manager
spec:
  isCA: true
  duration: 21600h # 900d
  secretName: selfsigned-ca
  commonName: certmanager-ca
  subject:
    organizations:
      - cert-manager
  issuerRef:
    name: selfsigned
    kind: Issuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-ca
spec:
  ca:
    secretName: selfsigned-ca
EOF

Step A3: Configure the intermediate CA for Istio

Setup the intermediate CA Istio will use to sign workload certificates, with rotation set to occur every 60 days (1440h) and renew before 15 days (360h) of expiry:

kubectl create namespace istio-system
cat << EOF | kubectl apply -f -
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: cacerts
  namespace: istio-system
spec:
  secretName: cacerts
  duration: 1440h
  renewBefore: 360h
  commonName: istiod.istio-system.svc
  isCA: true
  usages:
    - digital signature
    - key encipherment
    - cert sign
  dnsNames:
    - istiod.istio-system.svc
  issuerRef:
    name: selfsigned-ca
    kind: ClusterIssuer
    group: cert-manager.io
EOF

Note: Cert-manager exposes certificates and keys as kubernetes.io/tls Secrets. Istio can digest kubernetes.io/tls type Secrets starting with version 1.14.2.

Task B: Install and configure Istio to automatically update its CA

Use istioctl to install Istio. The following IstioOperator config  sets the environment variable AUTO_RELOAD_PLUGIN_CERTS=trueto make Istio automatically reload its CA on update:

cat << EOF | istioctl apply --skip-confirmation -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: demo-istio-install
  namespace: istio-system
spec:
  profile: demo
  components:
    pilot:
      k8s:
        env:
        - name: AUTO_RELOAD_PLUGIN_CERTS
          value: "true"
EOF

Task C: Configure and verify Istio’s intermediate CA rotation

Step C1: Configure rotation intermediate CA

Let’s say requirements have changed and we need to tighten the CA rotation period from 60 days (1440h) to 30 days (720h):

cat << EOF | kubectl apply -f -
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: cacerts
  namespace: istio-system
spec:
  secretName: cacerts
  duration: 720h
  renewBefore: 360h
  commonName: istiod.istio-system.svc
  isCA: true
  usages:
    - digital signature
    - key encipherment
    - cert sign
  dnsNames:
    - istiod.istio-system.svc
  issuerRef:
    name: selfsigned-ca
    kind: ClusterIssuer
    group: cert-manager.io
EOF

Step C2: Verify the new intermediate CA is reloaded

Watching the logs should reveal the CA change:

kubectl logs -l app=istiod -n istio-system -f

You should see something like this in your log ouput:

2022-08-11T20:18:41.493247Z	info	Update Istiod cacerts
2022-08-11T20:18:41.493483Z	info	Using kubernetes.io/tls secret type for signing ca files
2022-08-11T20:18:41.716843Z	info	Istiod has detected the newly added intermediate CA and updated its key and certs accordingly
2022-08-11T20:18:41.717170Z	info	x509 cert - Issuer: "CN=istiod.istio-system.svc", Subject: "", SN: 1c43c1686425ee2e63f2db90bd3cf17f, NotBefore: "2022-08-11T20:16:41Z", NotAfter: "2032-08-08T20:18:41Z"
2022-08-11T20:18:41.717220Z	info	x509 cert - Issuer: "CN=certmanager-ca,O=cert-manager", Subject: "CN=istiod.istio-system.svc", SN: c172b51eeb4a2891fe66f30babb42bb0, NotBefore: "2022-08-11T20:17:25Z", NotAfter: "2022-08-13T20:17:25Z"
2022-08-11T20:18:41.717254Z	info	x509 cert - Issuer: "CN=certmanager-ca,O=cert-manager", Subject: "CN=certmanager-ca,O=cert-manager", SN: ea1760f2dcf9806a8c997c4bc4b2fb30, NotBefore: "2022-08-11T20:13:33Z", NotAfter: "2025-01-27T20:13:33Z"
2022-08-11T20:18:41.717256Z	info	Istiod certificates are reloaded

Summary

As we’ve seen, using cert-manager to automate Istio CA rotation makes it easy to efficiently handle a critical operations function. Configuring Istio to automatically reload its CA removes the need to manually restart Istio, removing a potential source of human error.

Service mesh is a powerful tool for implementing zero trust security practices and increasing business agility and continuity at scale. Building effective operational practices for service mesh is critical to harnessing that power. As founders and core contributors to Istio and Envoy, we help our customers do that every day.