We love open source, but recognize the reality that security can be hard. Recently the Log4j vulnerability (CVE-2021-44228) made everyone realize how wide-spread and severe a security incident can be when it comes to security root causes in the open source. In this blog we will first briefly explain the Log4J CVE as background, then demonstrate how Tetrate Service Bridge can help by providing a built-in Web Application Firewall engine.
Disclaimer: this is a work-in-progress feature. To help mitigate the issue right now, check out the blog post Tetrate Log4J Announcement.
Log4j Shell Attack
Below is a diagram from govercert.ch to illustrate how Log4j remote execution works.
Log4j is a popular Java logging library. Besides basic logging facilities, Log4j provides a Lookup feature: ${variable} can be used to reference some runtime variables. When the lookup value is a JNDI(Java Naming and Directory Interface) runtime reference, Log4j makes outbound requests to resolve the value. For example, ${jndi://ldap://evial.xa/path}. The Log4j application sends a request to the LDAP server at evial.xca/path.
In the real world, an attacker can take advantage of the remote lookup feature. By having a malicious LDAP server to return a specially crafted output, the Log4J Java program can be tricked into executing some malware code!
Attackers are actually actively scanning the exploitable network endpoint for the attack. After the log4j CVE was released, well known technology companies such as AWS and Cisco were reported to have been affected by this security incident.
So I understand this is bad, so what should we do with respect to the TSB?
We Tetrate are committed to a high bar of security. Right after the incident report, we shortly announced a TSB release, to update all TSB Java components.
Got it, so we are fully covered. Then what else?
In the announcement blog, we offer an EnvoyFilter based solution. EnvoyFilter is breaking glass Istio API, allowing you to change the sidecar proxies’ configuration directly. First, the approach is not manual and tedious. In this way it’s preferable to SecurityGroup or GatewayGroup, where you would have to directly select your workloads by labels, and duplicate config in each cluster. Second, the EnvoyFilter alone only provides a limited set of the generic HTTP requests libraries as Lua code, which is not specially designed for web application securities.
So this blog is about the future. How can TSB help you to protect from a severe security risk like Log4jShell systematically?
The answer is by offering a native web application firewall. TSB is adding web application firewall capabilities (WAF). And as we can see, just with a small amount of configuration using WAF, you can have TSB to protect your application.
Bring WAF to a Service Mesh
Web application firewall is a generic, application-level firewall technique to filter, monitor, and block suspicious HTTP requests. For example, a web form POST request can contain a malicious string to achieve SQL injection. WAF is aimed to reject these requests at an early stage, including but not limited to SQL injection, XSS, etc.
ModSecurity is a popular open source web application engine. ModSecurity has its own syntax rules. Users can configure the rules such that a request is examined and matched against some defined patterns, and further handled, such as being audited or rejected, if matched.
TSB implements ModSecurity as part of the Envoy Wasm extension that runs in the sidecar proxy. The enhanced envoy proxies are able to extract the HTTP requests attributes, feed into the Wasm-based Modsecurity plugin, and enforce the rule configuration. Since ModSecurity runs in the sidecar proxies, WAF can be enforced to all traffic between pods, not just traffic going through gateways. This is a powerful advantage of bringing an integrated WAF to a service mesh. Any attack originating from any service can be mitigated, resulting in a zero-trust WAF.
On top of the data plane enhancement, we offer an easy TSB API to configure and inject the WAF rules protection at the right place.
End-to-End Setup
Now we will walk through an end to end setup to illustrate TSB web application firewall protection. The demo shows that after WAF is configured, suspicious Log4J shell attack requests will be denied.
Step 1, Install TSB and Deploy Workloads
In our example, we deploy a httpbin app and expose it with an ingress gateway.
Use tctl to create them in TSB.
Now we send a few potential malicious requests to httpbin.
As you can see, the requests succeed. The JNDI path is printed out as the httpbin response.
Step 2, Enable WAF Rule to Protect Log4j Attack
Here we use a simple WAF rule that blocks all requests containing the `${jndi:` pattern. For the detailed pattern to use, check coreruleset.org. When matched, the rules block and log the requests at the envoy proxies.
We first created a WAF rule set custom resource.
Next, configure your SecurityGroup and SecuritySettings to enable the WAF rule set we just defined.
You can adjust the WAF protection to suit your needs. For example, you can only enforce the WAF at the Tier1Gateway or IngressGateway resources, which are the entry points for the untrusted traffic. Alternatively, you can retristrict enforcement for certain namespaces, or a particular app that is written in Java.
Step 3, Check Protection Enforced
Now the WAF rule block-log4j has been enabled on the ingress gateway. We are sending the same requests again.
As we can see, the requests are now denied with 403. In the httpbin sidecar log, you can also see the matching lines.
Further Reading
To summarize, the Tetrate Service Bridge WAF engine makes a big difference for security admin to handle a security incident like this. Definitely we should still follow the right security handling process: performing threat analysis, releasing and upgrading to fix the issue from the root cause. But TSB WAF allows you to mitigate the issue immediately, and buys the time to handle the incident properly and correctly.
Additionally, WAF is designed to be a powerful rule engine to handle security risks beyond Log4j.
Currently we are working on polishing the API and final user experience for the WAF integration. You should expect this feature to be GA in the next release of TSB in March 2022.
References
- Tetrate announcement for security release to fix the log4j cve: https://tetrate.io/blog/tsb-log4j-security-announcement /
- Coreruleset analysis and rule set for Log4j attack: https://coreruleset.org/20211213/crs-and-log4j-log4shell-cve-2021-44228/
- Tetrate introduction about Istio WASM feature: https://tetrate.io/blog/istio-wasm-extensions-and-ecosystem/
