A critical vulnerability (CVE-2021-44228, CVSS score 10) was identified in the Java logging library Apache Log4j 2. Apache Log4j2 2.14.1 and below are susceptible to a remote code execution vulnerability where a remote attacker can leverage this vulnerability to take full control of a vulnerable machine.
Apache Log4j is used in many Java-based applications, making this vulnerability potentially affecting lots of organizations. As we continue to gain a deeper understanding of the impact of this threat, we will publish technical information to help you detect, investigate, and mitigate attacks. We will provide updates with more information and protection details as they become available.
Update at 2021-12-14: New, related CVE-2021-45046 has been disclosed and mitigations are included in this post.
TSB Security Updates
TSB observability stack, Skywalking (image name
spm-user and deployed as oap in cluster), is using Log4j for its logging library and is affected by this vulnerability. To address CVE-2021-44228 and CVE-2021-45046 we encourage TSB users to upgrade TSB to version 1.2.13 or 1.4.3. This security patch release will update Skywalking to use Log4j version 2.16.0.
If upgrading TSB is not possible, TSB users can change SPM image (
spm-central) version to the version tag that is available in TSB 1.2.13 or 1.4.3.
TSB users can disable log4j logging to mitigate attacks. However, this will completely disable logging and you will not see any logs in Skywalking when you want to troubleshoot issues. This can be done by editing
oap-config ConfigMap and setting log level to OFF <
.. level="OFF">. You will need to restart oap container.
For the management plane, edit
oap-config ConfigMap in
tsb namespace. Change log level to OFF and restart oap-deployment. You might want to keep a backup of the existing configmap before updating.
For the control plane, edit
oap-dynamic-config ConfigMap in
istio-system namespace. Change log level to OFF and restart oap-deployment. You might want to keep a backup of the existing configmap before updating
For the management plane, the ConfigMap is the same as 1.4.x above.
For the control plane, log4j config is in
oap-config ConfigMap. edit
oap-config ConfigMap in istio-system namespace. Change log level to OFF and restart oap-deployment. You might want to keep a backup of the existing configmap before updating
Please contact your Tetrate Account Manager to get more information.
Application Security Updates
To address these vulnerabilities in your applications, Tetrate recommends TSB users apply the latest Log4j security updates to remediate this vulnerability. The recommended action is to update Log4j 2 to 2.16.0.
Until more complete security updates can be applied, you should consider the following mitigation steps. Note that these workarounds should not be considered a complete solution to resolve this vulnerability.
Disable Log4j JNDI Lookup plugins
If you are using Log4j v2.10 or above, and cannot upgrade, then set the property:
Additionally, an environment variable can be set for these same affected versions:
Note that this only applies to CVE-2021-44228. For CVE-2021-45046 you have to remove the JndiLookup class from the classpath that is shown below. To configure these values in TSB, you need to edit the ControlPlane and ManagementPlane resources, set the `env` value in the OAP section of the resources (managementplane in the TSB cluster, controlplane in the control plane clusters).
Management Plane Clusters:
Control Plane Clusters:
Note: When you edit the `env` section of the operator, it is merged into the existing environment variables defined in the container, so you do not need to list every environment variable just to set this one.
For Log4j releases from 2.0-beta9 to 2.10.0, remove the JndiLookup class from the classpath. For example, you can run the following command to remove the class from the log4j-core.
zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
Detecting JNDI Request Header
You can Validate all parts of the request header against the attack and also check for evasions of the detection. You can use WAF to detect Log4j exploit payload in request URL, headers, or body. For example, see modsec or you can contact your WAF provider.
If you are using Istio, you can achieve this by applying EnvoyFilter in the gateway or sidecar to detect the attack in the request header.
The following example enables Envoy’s Lua filter for all inbound HTTP calls arriving at the service pod with labels “app: httpbin”, in the default namespace. The lua filter checks for jndi string in the header and will reject the request if found.
TID Security Updates
Tetrate Istio Distribution and the OSS Istio components are not affected by these (CVE-2021-44228 and CVE-2021-45046) log4j vulnerabilities.