Securing Istio: Addressing Critical Security Gaps and Best Practices

Introduction

Recently, the Wiz research team released a blog post that uncovered tenant isolation vulnerabilities in AI services, generating widespread attention. This study detailed security flaws across several AI service providers, particularly the SAP AI Core platform. Researchers were able to execute arbitrary code through legitimate AI training processes, subsequently moving laterally to take over services and gain access to customers’ private files and cloud credentials. These findings highlight the challenges that cloud services and management platforms face in ensuring isolation and sandbox environments.

In this context, Istio, as a crucial service mesh solution, faces similar security issues, especially in key functionalities like sidecar injection and traffic management. This blog aims to discuss how to protect the security of the Istio service mesh and provide a comprehensive set of mitigation measures. We will also discuss how multi-layer security strategies can effectively enhance Istio’s security to address challenges mentioned in the Wiz report.

Tetrate offers an enterprise-ready, 100% upstream distribution of Istio, Tetrate Istio Subscription (TIS). TIS is the easiest way to get started with Istio for production use cases. TIS+, a hosted Day 2 operations solution for Istio, adds a global service registry, unified Istio metrics dashboard, and self-service troubleshooting.
Get access now ›

Overview

Istio primarily manages east-west traffic within Kubernetes, offering detailed traffic management features such as request routing, load balancing, and fault recovery policies. While Istio offers essential security features such as traffic encryption, authentication, and authorization, it should not be viewed as a standalone firewall solution. To maintain robust security for services within the Istio mesh, it is crucial to complement Istio’s security capabilities with additional measures from the underlying network and infrastructure, such as Container Network Interface (CNI) plugins and secure container implementations. Moreover, micro-segmentation technology can be employed to achieve finer-grained isolation and enhance security.

Whether in Sidecar or Ambient mode, traffic is hijacked from application pods to data plane proxies for processing and forwarding. If application traffic is not successfully intercepted or is impersonated by a rogue application masquerading as Istio, security vulnerabilities can arise.

The diagram below illustrates where “security vulnerabilities” due to bypassing or impersonating Istio system users might occur.

Next, we will explore specific situations where “security vulnerabilities” arise and the strategies to address them.

Bypassing Istio Sidecar Injection

At the Namespace Level

• Scenario: Application teams misuse namespace labels to disable Istio Sidecar injection at the namespace level.
• Mitigation Strategy: Platform teams abstract app deployment and restrict access to the raw Kubernetes namespace resources.
• Monitoring: Use policy engines (like OPA Gatekeeper) to ensure compliance with namespace labels, and regularly review namespace configurations.

At the Pod Level

• Scenario: Application teams misuse Pod labels to disable Istio Sidecar injection at the Pod level.
• Mitigation Strategy: Platform teams abstract app deployment and restrict access to the raw Kubernetes Pod resources.
• Monitoring: Employ Admission Webhooks to enforce Sidecar injection, prohibit exclusion labels, and regularly scan and audit all pods to ensure every required pod has a Sidecar injected.

Bypassing Traffic Redirection to Istio Sidecar

Misuse of Traffic Redirection Annotations

• Scenario: Application teams misuse Pod annotations to exclude certain inbound or outbound ports or IPs, thereby bypassing traffic redirection.
• Mitigation Strategy: Platform teams abstract app deployment and restrict access to the raw Kubernetes Pod resources.
• Monitoring: Use policy engines to detect and alert on non-compliant annotation use, regularly review Pod annotations.

Misuse of Pod UID

• Scenario: Application teams misuse UID 1337 (the ID of the sidecar proxy) to bypass Istio’s Iptables redirection rules.
• Mitigation Strategy:
  • Force all Pods to specify a UID that is not 1337.
  • Inspect all container images to check for UID 1337 and reject those images. This inspection can be performed using an admission webhook or by a central team managing the image registry.
• Monitoring: Prohibit or restrict the use of UID 1337, regularly audit Pod UID configurations to ensure no bypassing occurs.

Misuse of Pod Capabilities (NET_ADMIN, NET_RAW)

• Scenario: Application teams misuse NET_ADMIN and NET_RAW capabilities to remove Istio Iptables rules.
• Mitigation Strategy: Platform teams enable Istio CNI (to avoid granting elevated privileges to application teams) and restrict access to the raw Kubernetes Pod resources.
• Monitoring: Regularly review and monitor Pod permission configurations to ensure no over-privileged actions are taken.

Bypassing Inbound Traffic Constraints

Misuse of PeerAuthentication

• Scenario: Application teams create a PeerAuthentication resource for each namespace/workload, enabling the PERMISSIVE authentication mode.
• Mitigation Strategy: Platform teams restrict access to the raw Istio PeerAuthentication resources.
• Monitoring: Regularly review PeerAuthentication configurations to ensure all inbound traffic is encrypted as required.

Bypassing Outbound Traffic Constraints

Misuse of ServiceEntry

• Scenario: Application teams create a ServiceEntry to directly access external services without going through an Egress gateway.
• Mitigation Strategy: Platform teams restrict access to the raw Istio ServiceEntry resources.
• Monitoring: Regularly review ServiceEntry configurations to ensure no bypassing occurs.

Misuse of ExternalName Services

• Scenario: Application teams create a Kubernetes Service of type ExternalName to directly access external services without going through an Egress gateway.
• Mitigation Strategy: Platform teams restrict access to the raw Kubernetes Service resources.
• Monitoring: Regularly review the types of Kubernetes Service configurations to ensure no bypassing occurs.

Uncontrollably Changing Istio Sidecar Configuration

Misuse of Sidecar Resources

• Scenario: Application teams create an Istio Sidecar resource for each workload and set the outboundTrafficPolicy field to ALLOW_ANY (overriding the possible global value REGISTRY_ONLY).
• Mitigation Strategy: Platform teams restrict access to the raw Istio Sidecar resources.
• Monitoring: Regularly review Sidecar resource configurations to ensure no global settings are overridden.
• Note: It is important to understand that ALLOW_ANY and REGISTRY_ONLY are related to service discovery and not security. To ensure security, implement appropriate authorization policies in addition to these discovery settings.

Misuse of EnvoyFilter

• Scenario: Application teams create an EnvoyFilter that conflicts with existing Istio objects, potentially causing DoS attacks or violating security policies.
• Mitigation Strategy: Platform teams restrict access to the raw Istio EnvoyFilter resources.
• Monitoring: Regularly review EnvoyFilter configurations to ensure no improper use occurs.

Service Mesh as Part of a Layered Defense

The service mesh is described as a supplemental layer to existing security models, enhancing microservice security by adding finer-grained security policies on top of traditional security controls. However, the article emphasizes that service meshes cannot independently ensure comprehensive security for microservices but should be part of an overall security strategy.

Service meshes primarily manage and control network traffic by deploying a lightweight proxy (sidecar) next to each service instance. This allows for precise traffic control and policy enforcement at the network level, such as traffic encryption, authentication, and authorization. Although service meshes offer features like traffic control, service discovery, and circuit breakers, these are essentially management of network traffic and are not sufficient to address all security issues. For instance, they cannot replace traditional security measures like application layer firewalls, intrusion detection systems, and data security.Furthermore, service meshes rely on correct configuration and management, and improper configuration can lead to security vulnerabilities. Therefore, while service meshes are an indispensable part of modern microservices architectures, they should be combined with traditional security measures to form a comprehensive, multi-layered security strategy framework. Refer to How Service Mesh Layers Microservices Security with Traditional Security to Move Fast Safely for further insights on strengthening service mesh security.

Long-Term Solutions and Community Collaboration

The Istio community conducts a security audit almost every year, see the results from 2021 and 2022. From these results, we can see that Istio’s security posture has greatly improved. Ensure that your Istio service mesh adheres to security best practices. Additionally, keep an eye on the Istio CVE Bulletins or use tools like Tetrate Istio Subscription that can scan for various CVEs in the Istio service mesh, deploying Istio versions that are FIPS compliant and FIPS certified.

Conclusion

Service meshes provide an additional layer of security for microservices architectures by managing control flows outside of the applications. This allows for enhanced communication security between services without impacting application performance. When deploying service meshes, it is recommended to use Istio’s Egress Gateway to manage outbound traffic, in conjunction with Kubernetes’ NetworkPolicy, to ensure all outbound traffic must pass through the gateway, thus preventing potential data leaks and other security threats.

References

• How to enforce egress traffic using Istio’s authorization policies – tetrate.io
• Istio Security Best Practice – istio.io
• Optimize Traffic Management and Security with These Service Mesh Best Practices – tetrate.io
• Istio publishes results of 2022 security audit – istio.io
• Announcing the results of Istio’s first security assessment – istio.io

###

If you’re new to service mesh, Tetrate has a bunch of free online courses available at Tetrate Academy that will quickly get you up to speed with Istio and Envoy.

Are you using Kubernetes? Tetrate Enterprise Gateway for Envoy (TEG) is the easiest way to get started with Envoy Gateway for production use cases. Get the power of Envoy Proxy in an easy-to-consume package managed by the Kubernetes Gateway API. Learn more ›

Getting started with Istio? If you’re looking for the surest way to get to production with Istio, check out Tetrate Istio Subscription. Tetrate Istio Subscription has everything you need to run Istio and Envoy in highly regulated and mission-critical production environments. It includes Tetrate Istio Distro, a 100% upstream distribution of Istio and Envoy that is FIPS-verified and FedRAMP ready. For teams requiring open source Istio and Envoy without proprietary vendor dependencies, Tetrate offers the ONLY 100% upstream Istio enterprise support offering.

Need global visibility for Istio? TIS+ is a hosted Day 2 operations solution for Istio designed to simplify and enhance the workflows of platform and support teams. Key features include: a global service dashboard, multi-cluster visibility, service topology visualization, and workspace-based access control.

Get a Demo