Summary

A critical vulnerability (CVE-2021-44228, CVSS score 10) was identified in the Java logging library Apache Log4j 2. Apache Log4j2 2.14.1 and below are susceptible to a remote code execution vulnerability where a remote attacker can leverage this vulnerability to take full control of a vulnerable machine.

Apache Log4j is used in many Java-based applications, making this vulnerability potentially affecting lots of organizations. As we continue to gain a deeper understanding of the impact of this threat, we will publish technical information to help you detect, investigate, and mitigate attacks. We will provide updates with more information and protection details as they become available.

Update at 2021-12-14: New, related CVE-2021-45046 has been disclosed and mitigations are included in this post.

TSB Security Updates

TSB observability stack, Skywalking (image name spm-central and spm-user and deployed as oap in cluster), is using Log4j for its logging library and is affected by this vulnerability. To address CVE-2021-44228 and CVE-2021-45046 we encourage TSB users to upgrade TSB to version 1.2.13 or 1.4.3. This security patch release will update Skywalking to use Log4j version 2.16.0.

If upgrading TSB is not possible, TSB users can change SPM image (spm-user and spm-central) version to the version tag that is available in TSB 1.2.13 or 1.4.3.

Disabling log4j

TSB users can disable log4j logging to mitigate attacks. However, this will completely disable logging and you will not see any logs in Skywalking when you want to troubleshoot issues. This can be done by editing oap-config ConfigMap and setting log level to OFF <.. level="OFF">. You will need to restart oap container.

TSB 1.4.x

For the management plane, edit oap-config ConfigMap in tsb namespace. Change log level to OFF and restart oap-deployment. You might want to keep a backup of the existing configmap before updating.

For the control plane, edit oap-dynamic-config ConfigMap in istio-system namespace. Change log level to OFF and restart oap-deployment. You might want to keep a backup of the existing configmap before updating

TSB 1.2.x

For the management plane, the ConfigMap is the same as 1.4.x above.

For the control plane, log4j config is in oap-config ConfigMap. edit oap-config ConfigMap in istio-system namespace. Change log level to OFF and restart oap-deployment. You might want to keep a backup of the existing configmap before updating

Please contact your Tetrate Account Manager to get more information.

Application Security Updates

To address these vulnerabilities in your applications, Tetrate recommends TSB users apply the latest Log4j security updates to remediate this vulnerability. The recommended action is to update Log4j 2 to 2.16.0.

Workarounds

Until more complete security updates can be applied, you should consider the following mitigation steps. Note that these workarounds should not be considered a complete solution to resolve this vulnerability. 

Disable Log4j JNDI Lookup plugins

1. If you are using Log4j v2.10 or above, and cannot upgrade, then set the property:
   log4j2.formatMsgNoLookups=true

   Additionally, an environment variable can be set for these same affected versions:  LOG4J_FORMAT_MSG_NO_LOOKUPS=true

   Note that this only applies to CVE-2021-44228. For CVE-2021-45046 you have to remove the JndiLookup class from the classpath that is shown below.  To configure these values in TSB, you need to edit the ControlPlane and ManagementPlane resources, set the `env` value in the OAP section of the resources (managementplane in the TSB cluster, controlplane in the control plane clusters).

   Management Plane Clusters:

   
   

   Control Plane Clusters:

   Note: When you edit the `env` section of the operator, it is merged into the existing environment variables defined in the container, so you do not need to list every environment variable just to set this one.

2. For Log4j releases from 2.0-beta9 to 2.10.0, remove the JndiLookup class from the classpath. For example, you can run the following command to remove the class from the log4j-core.

   zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

Detecting JNDI Request Header

You can Validate all parts of the request header against the attack and also check for evasions of the detection. You can use WAF to detect Log4j exploit payload in request URL, headers, or body. For example, see modsec or you can contact your WAF provider.

If you are using Istio, you can achieve this by applying EnvoyFilter in the gateway or sidecar to detect the attack in the request header. 

The following example enables Envoy’s Lua filter for all inbound HTTP calls arriving at the service pod with labels “app: httpbin”, in the default namespace. The lua filter checks for jndi string in the header and will reject the request if found.