In DevSecOps the reality is that everything is configurable– absolutely everything. A good analogy here: Imagine if you had a pen that had– instead of one button to retract the ballpoint– a number of knobs and switches– a color adjuster, pressure switch, ink density tuning wheel, etc. Many people would find it difficult to sign their name because not many of us are ink experts. A similar situation is happening when application developers or infrastructure specialists need to become cryptographers to set up mTLS certificates.
In this article we really want to touch on the basics, which is honestly all that most of us need to know to set up a properly operating and secure environment. Let’s dig into the details of how our imaginary “security pen” is assembled where the only button we have at the end is to retract the ballpoint.
Some of this material is covered in Tetrate’s blog post already and a video that explains how Istio uses external CAs. Additionally, the source of truth can be found here.
Let’s examine mTLS in baby steps. By the end you can probably go back to your environment and fix any trust issues in a blink of an eye.
Everything started with TLS
Imagine this scenario. In the mid-2000s a user goes to a website and enters their credit card information to make a purchase. Then the bank calls the user to inform them that someone on the other end of the planet is using his/her credit card extensively to fill up gas.
The obvious two causes behind this case of credit card fraud:
- the website that the user accessed was not the website user expected – but looked perfectly fine (the type of attack is usually called phishing)
- the website the user went to was a legitimate business, but the data between the user’s device and the website was hijacked (man in the middle attack)
TLS is supposed to resolve both of these problems. Let’s have a look at TLS setup:
As the picture illustrates, the user is accessing a website via the HTTPs protocol. The website provides the certificate that contains the Subject Alternative Name (SAN) field and its value matches what the user entered in the browser (store.santa.com in this case). Also the certificate contains the signature that protects the integrity of the certificate’s data. If someone tried to change the SAN field, the certificate signature would become invalid. Also, the certificate has the information of the certificate issuer.
After the certificate is provided to the user, it gets validated using the issuer Root certificate from the trusted CA Bundle. CA Bundle is very compact and every browser and operating system has such a bundle installed (usually by operating system processes). Users can add more CA certificates to the trusted bundle. In the typical use case, an organization would have an internal website and the certificate would be signed by the Enterprise Root CA. There is no intent for this website to accept the connections coming from regular callers – only from enterprise users. The additional Enterprise CA certificates can be added manually or automatically (via a secure and controlled process) to the end user devices.
When the website is confirmed to be trusted on the user end, the encryption key exchange is triggered (using the public encryption key that is provided together with the certificate from the server). The rest of the communication that is happening is encrypted (usually using the short-lived, one-time encryption key provided during TLS handshake).
So now both concerns are addressed. The server is validated and allows us to prevent the phishing attack. The payload is encrypted, addressing the man-in-the-middle threat.
Intermediate CAs
In the previous scenario we had only two certificate entities– a Root Certificate that validates end-user certificates. In many cases more certificates are required.
Imagine an organization has a Root CA that, if compromised and revoked, will invalidate all the certificates in the Enterprise. To minimize the number of automation and manual interactions with the Root CA, intermediate CAs are used. For example: Every team can have its own intermediate CA. As part of the enterprise PKI system, all of them should trust each other; the same Root CA is used to sign the Intermediate CAs issued. Then Intermediate CAs are used to sign leaf certificates for the specific servers.
Or maybe there is more of a hierarchical relationship. Let’s say the European region has IT, Finance, and HR all managing their own certs, but the Asia region has only a Business Operation unit. Then the structure will look something like this:
With the exception of the Root Certificate, every other certificate will have “Signer” pointing to the certificate above. When that signer is verified, the validation process moves a level up to check the Signer certificate against the issuer of the next level.
To simplify the previous diagram, if a specific leaf certificate tree is analyzed, it becomes clear that the next-level certificate is needed for the validation process at every step. Validation with more than one certificate in the hierarchy requires multiple iterations to be validated.
TLS communication
A few things need to be in place to make the communication work. Below is a not-working scenario that will help us to understand the requirements of working, trusted TLS communication.
When a client connects to the server it gets the server leaf certificate. That certificate will be validated against the Root CA bundle on the client machine. If the provided leaf certificate signed by an Intermediate CA cert is being validated against the Root CA, the validation will fail as the Intermediate certificate(s) is/are missing.
As demonstrated, the client requires all the certificates to be available to complete the validation. The idea of the client storing all intermediate certificates of all possible targets seems difficult to manage on every client device. It’s much more manageable for a client to have a relatively short list of top-level trusted certificates and for the server to provide the rest. The group of the certificates that are signed sequentially is called a certificate chain.
When the certificate chain is provided to the client, the whole sequence can be verified and the top level of the chain (*.service.local) is validated against the Root CA (*.local in our example). In this case,trust is established all the way, while multiple levels of the certificate management are available to the service owner.
As part of this blog we provide a simple script that creates the chain generated in this example. Hopefully you can use the example to understand the details and compare against your own certs.
Below is an openssl command that would verify the chain and show every certificate
$ > openssl verify -show_chain -CAfile test-full-correct-chain.crt orders.crt
orders.crt: OK
Chain:
depth=0: C = US, ST = CA, O = Tetrateio, CN = orders.fin.service.local Intermidiate CA (untrusted)
depth=1: C = US, ST = CA, O = Tetrateio, CN = *.fin.service.local Intermidiate CA
depth=2: C = US, ST = CA, O = Tetrateio, CN = *.service.local Intermidiate CA
depth=3: C = US, ST = CA, O = Tetrateio, CN = "*.local Root CA "
$ >The important detail to understand is that the certificate order is important. In short, the chain always starts with the leaf certificate, followed by intermediate certs whose signatures are verified using the next certificate in the chain.
$ > openssl crl2pkcs7 -nocrl -certfile test-full-correct-chain.crt | openssl pkcs7 -print_certs --noout
subject=C = US, ST = CA, O = Tetrateio, CN = orders.fin.service.local Intermidiate CA
issuer=C = US, ST = CA, O = Tetrateio, CN = *.fin.service.local Intermidiate CA
subject=C = US, ST = CA, O = Tetrateio, CN = *.fin.service.local Intermidiate CA
issuer=C = US, ST = CA, O = Tetrateio, CN = *.service.local Intermidiate CA
subject=C = US, ST = CA, O = Tetrateio, CN = *.service.local Intermidiate CA
issuer=C = US, ST = CA, O = Tetrateio, CN = "*.local Root CA "
subject=C = US, ST = CA, O = Tetrateio, CN = "*.local Root CA "
issuer=C = US, ST = CA, O = Tetrateio, CN = "*.local Root CA "Turning TLS to mTLS
mTLS (which stands for mutual-TLS) defines the secured trusted communication when both parties’ identities are verified. Going back to the credit card fraud example, with mTLS both the User and the store website must mutually prove they’re trustworthy, when we looked at TLS you can easily see that it is a one way street– meaning the client makes sure that the service that it’s calling to is trusted, but the service doesn’t have the ability to validate who’s calling it. There are plenty of situations where one-way trust is all that is needed. At Tetrate we work with service mesh which means dealing with a significant share of east-west traffic where the client and server are both services and equals, mTLS addresses that requirement.
In the mTLS case, both ends need to be validated, which means:
- The connection only happens when both identities– the calling service and the receiving service– are validated by each other.
- Both services present the leaf and chain (unless the leaf is signed by Root CA).
- Services have a Root CA that was used to sign the chain provided by the other party and can use Root CA to validate the chain presented by the other side.
The drawing above demonstrates how mTLS is functioning:
- Bundles are presented by each party.
- The provided bundles are validated on another end.
- Only if both validations are successful will the communication proceed.
When mTLS is used after the identities of the parties are validated, the data payloads are encrypted in the same fashion as with regular TLS.
Note: the Root Certificate bundle can include multiple certs and if one of those certs validates the certificate provided by another party, the connection is considered to be successful. Another important point is that the Root CA for both services can be the same or different,as long as the Root certificate is available to validate the provided chain.
Summary
This article provided the architectural overview of TLS and its extended version called mTLS.
As demonstrated here, the successful TLS validation requires (a) the complete chain to be validated and (b) the validating party usually has only a Root Certificate to validate the chain.
The very important part is the sequence in the chain – starting with the leaf and then followed by leaf signer, signer of the leaf signer, etc.
Openssl is the most widely used tool with the most examples of usage shared on the internet, but it’s not the ideal tool that certainly requires some practice.
We wish you a lot of luck in the beginning of TLS/mTLS implementation. But when the framework is established and understood, adding additional components in your infrastructure becomes a cookie cutter operation that can be highly automated.