One of Istio’s core capabilities is to facilitate a zero trust network architecture by managing identity for services in the mesh. To retrieve valid certificates for mTLS communication in the mesh, individual workloads issue a certificate signing request (CSR) to istiod. Istiod, in turn, validates the request and uses a certificate authority (CA) to sign the CSR to generate the certificate. By default, Istio uses its own self-signed CA for this purpose, but best practice is to integrate Istio into your existing PKI by creating an intermediate CA for each Istio deployment.
If you’re managing multiple clusters, that means issuing multiple intermediate CAs, each of which should be set to expire within a few months or sooner. Managing the lifecycle of these CAs is critical, since they must be rotated before they expire or bad things happen. This article will show you how to make this CA management easier to reduce risk and increase the overall stability of your system.
A critical step when rotating the CA is to ensure that the new one is actually used. By default Istio only loads its CA on startup. But, Istio can be configured to watch for changes and automatically reload its CA when it is updated. This tutorial, drawn from production playbooks we’ve developed working with our enterprise customers who are managing large numbers of Istio deployments, will show how to configure Istio to automatically reload its CA. We will also cover how to configure cert-manager to automatically rotate Istio’s intermediate CA on a regular schedule ahead of expiry to increase the operational efficiency of managing CAs on multiple clusters.
Prerequisites
You’ll need at least the following for the tutorial:
- A running Kubernetes cluster. A simplified Kubernetes install like minikube or similar is suitable for demo purposes;
- Istioctl v1.14.2 or higher;
- Cert-manager v1.7.2 or higher.
Task A: Install and configure cert-manager to automatically rotate Istio’s CA
Step A1: Install cert-manager
The following command will install cert-manager in your cluster. To install a newer version of cert-manager, change the github URL.
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/downloStep A2: Configure the CA
For demo purposes, we will set up a self-signed CA, but DO NOT USE A SELF-SIGNED CA IN PRODUCTION. For production purposes, you should configure cert-manager to use your existing PKI.
cat << EOF | kubectl apply -f -
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: selfsigned
namespace: cert-manager
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: selfsigned-ca
namespace: cert-manager
spec:
isCA: true
duration: 21600h # 900d
secretName: selfsigned-ca
commonName: certmanager-ca
subject:
organizations:
- cert-manager
issuerRef:
name: selfsigned
kind: Issuer
group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned-ca
spec:
ca:
secretName: selfsigned-ca
EOF
Step A3: Configure the intermediate CA for Istio
Setup the intermediate CA Istio will use to sign workload certificates, with rotation set to occur every 60 days (1440h) and renew before 15 days (360h) of expiry:
kubectl create namespace istio-system
cat << EOF | kubectl apply -f -
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: cacerts
namespace: istio-system
spec:
secretName: cacerts
duration: 1440h
renewBefore: 360h
commonName: istiod.istio-system.svc
isCA: true
usages:
- digital signature
- key encipherment
- cert sign
dnsNames:
- istiod.istio-system.svc
issuerRef:
name: selfsigned-ca
kind: ClusterIssuer
group: cert-manager.io
EOF
Note: Cert-manager exposes certificates and keys as kubernetes.io/tls Secrets. Istio can digest kubernetes.io/tls type Secrets starting with version 1.14.2.
Task B: Install and configure Istio to automatically update its CA
Use istioctl to install Istio. The following IstioOperator config sets the environment variable AUTO_RELOAD_PLUGIN_CERTS=trueto make Istio automatically reload its CA on update:
cat << EOF | istioctl apply --skip-confirmation -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: demo-istio-install
namespace: istio-system
spec:
profile: demo
components:
pilot:
k8s:
env:
- name: AUTO_RELOAD_PLUGIN_CERTS
value: "true"
EOF
Task C: Configure and verify Istio’s intermediate CA rotation
Step C1: Configure rotation intermediate CA
Let’s say requirements have changed and we need to tighten the CA rotation period from 60 days (1440h) to 30 days (720h):
cat << EOF | kubectl apply -f -
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: cacerts
namespace: istio-system
spec:
secretName: cacerts
duration: 720h
renewBefore: 360h
commonName: istiod.istio-system.svc
isCA: true
usages:
- digital signature
- key encipherment
- cert sign
dnsNames:
- istiod.istio-system.svc
issuerRef:
name: selfsigned-ca
kind: ClusterIssuer
group: cert-manager.io
EOF
Step C2: Verify the new intermediate CA is reloaded
Watching the logs should reveal the CA change:
kubectl logs -l app=istiod -n istio-system -fYou should see something like this in your log ouput:
2022-08-11T20:18:41.493247Z info Update Istiod cacerts
2022-08-11T20:18:41.493483Z info Using kubernetes.io/tls secret type for signing ca files
2022-08-11T20:18:41.716843Z info Istiod has detected the newly added intermediate CA and updated its key and certs accordingly
2022-08-11T20:18:41.717170Z info x509 cert - Issuer: "CN=istiod.istio-system.svc", Subject: "", SN: 1c43c1686425ee2e63f2db90bd3cf17f, NotBefore: "2022-08-11T20:16:41Z", NotAfter: "2032-08-08T20:18:41Z"
2022-08-11T20:18:41.717220Z info x509 cert - Issuer: "CN=certmanager-ca,O=cert-manager", Subject: "CN=istiod.istio-system.svc", SN: c172b51eeb4a2891fe66f30babb42bb0, NotBefore: "2022-08-11T20:17:25Z", NotAfter: "2022-08-13T20:17:25Z"
2022-08-11T20:18:41.717254Z info x509 cert - Issuer: "CN=certmanager-ca,O=cert-manager", Subject: "CN=certmanager-ca,O=cert-manager", SN: ea1760f2dcf9806a8c997c4bc4b2fb30, NotBefore: "2022-08-11T20:13:33Z", NotAfter: "2025-01-27T20:13:33Z"
2022-08-11T20:18:41.717256Z info Istiod certificates are reloaded
Summary
As we’ve seen, using cert-manager to automate Istio CA rotation makes it easy to efficiently handle a critical operations function. Configuring Istio to automatically reload its CA removes the need to manually restart Istio, removing a potential source of human error.
Service mesh is a powerful tool for implementing zero trust security practices and increasing business agility and continuity at scale. Building effective operational practices for service mesh is critical to harnessing that power. As founders and core contributors to Istio and Envoy, we help our customers do that every day.
###
If you’re new to service mesh, Tetrate has a bunch of free online courses available at Tetrate Academy that will quickly get you up to speed with Istio and Envoy.
Are you using Kubernetes? Tetrate Enterprise Gateway for Envoy (TEG) is the easiest way to get started with Envoy Gateway for production use cases. Get the power of Envoy Proxy in an easy-to-consume package managed by the Kubernetes Gateway API. Learn more ›
Getting started with Istio? If you’re looking for the surest way to get to production with Istio, check out Tetrate Istio Subscription. Tetrate Istio Subscription has everything you need to run Istio and Envoy in highly regulated and mission-critical production environments. It includes Tetrate Istio Distro, a 100% upstream distribution of Istio and Envoy that is FIPS-verified and FedRAMP ready. For teams requiring open source Istio and Envoy without proprietary vendor dependencies, Tetrate offers the ONLY 100% upstream Istio enterprise support offering.
Need global visibility for Istio? TIS+ is a hosted Day 2 operations solution for Istio designed to simplify and enhance the workflows of platform and support teams. Key features include: a global service dashboard, multi-cluster visibility, service topology visualization, and workspace-based access control.
Get a Demo