top of page

Experienced Technology Product Manager adept at steering success throughout the entire product lifecycle, from conceptualization to market delivery. Proficient in market analysis, strategic planning, and effective team leadership, utilizing data-driven approaches for ongoing enhancements.

  • Twitter
  • LinkedIn
White Background

Workaround instructions to address CVE-2021-44228 in vRA 8.x and vRO 8.x with screenshots |21-Dec-21

Updated: Apr 24, 2022



 

This blog complies with the new version released on 21st December 2021

 

Note

All instructions and procedures are taken from VMware KB: https://kb.vmware.com/s/article/87120 , all i am trying to do is to add some screenshots and outputs by implementing this workaround in my lab


Download this Note which has screenshots and detailed snippets






VMware updated KB article 87120 with new commands on 21st December 2021. This blog article complies with it


Both the PDF document attached and the blog screenshots are taken after the new commands are tested

 

Symptoms

CVE-2021-44228 has been determined to impact vRA and vRO from 8.0 to 8.6.1 via the Apache Log4j open source component it ships. This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

CVE-2021-44228 - VMSA-2021-0028 (link: https://www.vmware.com/security/advisories/VMSA-2021-0028.html) Notice: On December 14, 2021 the Apache Software Foundation notified the community that their initial guidance for CVE-2021-44228 workarounds was not sufficient. We believe the instructions in this article to be an effective mitigation for CVE-2021-44228, but in the best interest of our customers we must assume this workaround may not adequately address all attack vectors.  We expect to fully address both CVE-2021-44228 and CVE-2021-45046 by updating log4j to version 2.16 in forthcoming releases of 8.6.2, as outlined by our software support policies. VMSA-2021-0028 will be updated when these releases are available. In the interim, we will be updating this Knowledge Base article with revised guidance to remove all JndiLookup classes per Apache Software Foundation guidance. Please subscribe to this article to be informed when updates are published. The workarounds described in this document are meant to be a temporary solution only. Upgrades documented in the aforementioned advisory should be applied to remediate CVE-2021-44228 when available. Long-term resolution will be available in vRA and vRO versions 8.6.2 or later.

 

Purpose


CVE-2021-44228 and CVE-2021-45046 have been determined to impact vRA and vRO from 8.0 to 8.6.1 via the Apache Log4j open source component it ships. This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

  • CVE-2021-44228, CVE-2021-45046 - VMSA-2021-0028



 

Impact and Risks


Please note the following prior to executing the workaround procedure:

  • vRA and vRO versions 8.0, 8.0.1 and 8.1 are no longer supported at the time of this article's publication.

  • Re-apply this KB, If you have previously applied the workaround released prior to 12/20/2021. If not re-applied, upgrading to a future release will not be possible.

  • This change is persistent over upgrades and the KB should not be re-applied.

  • This change applies to vRA and vRO (both standalone and embedded).

  • Can be applied to all vRA and vRO deployments versions 8.1 through 8.6.1.

  • For clustered setups, in vRO Control Center/Cluster management page, a warning message "Local changes detected" may appear after applying this KB. This message should be disregarded.

Note: Automated vulnerability scanners may report that vRA/vRO products are still vulnerable to CVE-2021-44228 and CVE-2021-45046 after this KB article has been applied. These findings can be safely ignored.



 

Resolution


The workarounds described in this document can be considered as permanent solution as they update log4j libraries in the VA to 2.17.0.


Future releases will include log4j 2.17.0 or later


 

Workaround



For each vRA and vRO deployments with versions from 8.1 to 8.6.1, execute the following procedure


Pre-requisites


Take simultaneous VM snapshots without memory of all nodes in the cluster


For this task i would leverage vRSLCM as shown below in the screenshots


Select the product and click on the day-2 actions pane on the product and choose Create Snapshot




I will choose an option which would create snapshot after shutting down the appliances. This is the best option and recommended option too. If you cannot shutdown the production then that is fine , but take snapshots through LCM


Run the Precheck



Ensure precheck is successful



Snapshot task does not take a long time but shutting down the application and bringing it back on takes a little bit of time but it's worth it





Procedure


Note: This workaround applies to vRA and vRO (both standalone and embedded).

Note: To be applied to all vRA and vRO deployments versions 8.2 through 8.6.1.


SSH login or virtual machine console into one of the nodes in the vRA / vRO cluster.

Ensure all the pods are in running state




 


Step:1


Upload <87120-kb-v2.tar.gz> and <87120-kb-v2-validate.tar.gz> under /root on all nodes.




 


Step:2


Validate whether the system is vulnerable by running the command below on all nodes. Error reports related to log4j will show up for the affected artifacts that are vulnerable.



cd /root; base64 -d <<< "W1sgIiQoc2hhMjU2c3VtIDg3MTIwLWtiLXYyLXZhbGlkYXRlLnRhci5neiB8IGN1dCAtZiAxIC1kICcgJykiID09ICJmY2IxZjQ0YWRkMmNjZTg3MzNjNzAzOTYzNjg4Njk3NjU4ZjA3NTkzMzYyZTRhMzU0MjZlYjc5OGFjMTM5YzRlIiBdXSAmJiAocm0gLXJmIC90bXAvODcxMjAta2I7IG1rZGlyIC1wIC90bXAvODcxMjAta2IvOyB0YXIgLXh2ZiA4NzEyMC1rYi12Mi12YWxpZGF0ZS50YXIuZ3ogLUMgL3RtcC84NzEyMC1rYjsgY2htb2QgK3ggL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi12YWxpZGF0ZS5zaDsgL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi12YWxpZGF0ZS5zaDsgcm0gLXJmIC90bXAvODcxMjAta2IpIHx8IGVjaG8gIkZpbGUgbm90IGZvdW5kIDg3MTIwLWtiLXYyLXZhbGlkYXRlLnRhci5neiBvciBjaGVja3N1bSBtaXNtYXRjaCIK" | bash -






Complete Output looks like this ... 



root@vra [ ~ ]# cd /root; base64 -d <<< "W1sgIiQoc2hhMjU2c3VtIDg3MTIwLWtiLXYyLXZhbGlkYXRlLnRhci5neiB8IGN1dCAtZiAxIC1kICcgJykiID09ICJmY2IxZjQ0YWRkMmNjZTg3MzNjNzAzOTYzNjg4Njk3NjU4ZjA3NTkzMzYyZTRhMzU0MjZlYjc5OGFjMTM5YzRlIiBdXSAmJiAocm0gLXJmIC90bXAvODcxMjAta2I7IG1rZGlyIC1wIC90bXAvODcxMjAta2IvOyB0YXIgLXh2ZiA4NzEyMC1rYi12Mi12YWxpZGF0ZS50YXIuZ3ogLUMgL3RtcC84NzEyMC1rYjsgY2htb2QgK3ggL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi12YWxpZGF0ZS5zaDsgL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi12YWxpZGF0ZS5zaDsgcm0gLXJmIC90bXAvODcxMjAta2IpIHx8IGVjaG8gIkZpbGUgbm90IGZvdW5kIDg3MTIwLWtiLXYyLXZhbGlkYXRlLnRhci5neiBvciBjaGVja3N1bSBtaXNtYXRjaCIK" | bash - 
87120-kb-v2-validate.sh 
Scanning blueprint-webapp_private:latest at: 
> Scanning /tmp/patc-verify-container.rvwtc 
ERROR: found ./opt/vmware/lib/log4j-core-2.14.1.jar 
Scanning catalog-service_private:latest at: 
> Scanning /tmp/patc-verify-container.R7Doo 
>> Scanning ./opt/service/cs-host.jar 
ERROR: found ./BOOT-INF/lib/log4j-core-2.14.1.jar 
Scanning codestream_private:latest at: 
> Scanning /tmp/patc-verify-container.6WwhN 
ERROR: found ./opt/codestream/lib/log4j-core-2.13.3.jar 
Scanning content-service_private:latest at: 
> Scanning /tmp/patc-verify-container.GovpH 
ERROR: found ./opt/vmware/lib/log4j-core-2.14.1.jar 
Scanning identity-service_private:latest at: 
> Scanning /tmp/patc-verify-container.P5Zyo 
>> Scanning ./opt/bc-fips/bctls-fips-1.0.12.1.jar 
>> Scanning ./opt/bc-fips/bcpkix-fips-1.0.5.jar 
>> Scanning ./opt/bc-fips/bcmail-fips-1.0.3.jar 
>> Scanning ./opt/bc-fips/bc-fips-1.0.2.1.jar 
>> Scanning ./jdk/lib/jrt-fs.jar 
>> Scanning ./identity-service/lib/identity-service-1.5.4-SNAPSHOT.jar 
ERROR: found ./BOOT-INF/lib/log4j-core-2.14.1.jar 
Scanning provisioning-service_private:latest at: 
> Scanning /tmp/patc-verify-container.mYkqN 
ERROR: found ./admiral/log4j-core-2.13.3.jar 
Scanning relocation-service_private:latest at: 
> Scanning /tmp/patc-verify-container.yesgE 
>> Scanning ./opt/relocation/relocation-service.jar 
ERROR: found ./BOOT-INF/lib/log4j-core-2.12.1.jar 
Scanning vco_private:latest at: 
> Scanning /tmp/patc-verify-container.YSm3Q 
>> Scanning ./var/opt/apache-tomcat/lib/websocket-api.jar 
>> Scanning ./var/opt/apache-tomcat/lib/tomcat-websocket.jar 
>> Scanning ./var/opt/apache-tomcat/lib/tomcat-util.jar 
>> Scanning ./var/opt/apache-tomcat/lib/tomcat-util-scan.jar 
>> Scanning ./var/opt/apache-tomcat/lib/tomcat-jni.jar 
>> Scanning ./var/opt/apache-tomcat/lib/tomcat-jdbc.jar 
>> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-zh-CN.jar 
>> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-ru.jar 
>> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-ko.jar 
>> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-ja.jar 
>> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-fr.jar 
>> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-es.jar 
>> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-de.jar 
>> Scanning ./var/opt/apache-tomcat/lib/tomcat-dbcp.jar 
>> Scanning ./var/opt/apache-tomcat/lib/tomcat-coyote.jar 
>> Scanning ./var/opt/apache-tomcat/lib/tomcat-api.jar 
>> Scanning ./var/opt/apache-tomcat/lib/servlet-api.jar 
>> Scanning ./var/opt/apache-tomcat/lib/jsp-api.jar 
>> Scanning ./var/opt/apache-tomcat/lib/jaspic-api.jar 
>> Scanning ./var/opt/apache-tomcat/lib/jasper.jar 
>> Scanning ./var/opt/apache-tomcat/lib/jasper-el.jar 
>> Scanning ./var/opt/apache-tomcat/lib/el-api.jar 
>> Scanning ./var/opt/apache-tomcat/lib/ecj-4.6.3.jar 
>> Scanning ./var/opt/apache-tomcat/lib/catalina.jar 
>> Scanning ./var/opt/apache-tomcat/lib/catalina-tribes.jar 
>> Scanning ./var/opt/apache-tomcat/lib/catalina-storeconfig.jar 
>> Scanning ./var/opt/apache-tomcat/lib/catalina-ha.jar 
>> Scanning ./var/opt/apache-tomcat/lib/catalina-ant.jar 
>> Scanning ./var/opt/apache-tomcat/lib/annotations-api.jar 
>> Scanning ./var/opt/apache-tomcat/bin/tomcat-juli.jar 
>> Scanning ./var/opt/apache-tomcat/bin/commons-daemon.jar 
>> Scanning ./var/opt/apache-tomcat/bin/bootstrap.jar 
>> Scanning ./var/opt/apache-ant/lib/maven-ant-tasks-2.1.3.jar 
>> Scanning ./var/opt/apache-ant/lib/ant.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-xz.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-testutil.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-swing.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-netrexx.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-launcher.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-junitlauncher.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-junit4.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-junit.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-jsch.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-jmf.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-jdepend.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-javamail.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-jai.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-imageio.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-commons-net.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-commons-logging.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-apache-xalan2.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-apache-resolver.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-apache-regexp.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-apache-oro.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-apache-log4j.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-apache-bsf.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-apache-bcel.jar 
>> Scanning ./var/opt/apache-ant/lib/ant-antlr.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/security/policy/unlimited/local_policy.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/security/policy/unlimited/US_export_policy.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/security/policy/limited/local_policy.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/rt.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/resources.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/management-agent.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/jsse.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/jfr.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/jce.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/zipfs.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/sunpkcs11.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/sunjce_provider.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/sunec.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/nashorn.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/localedata.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/jaccess.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/dnsns.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/cldrdata.jar 
>> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/charsets.jar 
>> Scanning ./tmp/dejavu-fonts-ttf-2.37.zip
 Scanning /data/vco at: 
> Scanning /tmp/patc-verify-directory.baK9I/ 
ERROR: found ./vco/usr/lib/vco/configuration/webapps/vco-controlcenter/WEB-INF/lib/log4j-core-2.13.3.jar 
./vco/usr/lib/vco/app-server/temp/dars/o11nplugin-configurator.dar/lib/log4j-core-2.13.3.jar 
./vco/usr/lib/vco/app-server/temp/dars/o11nplugin-multi-node.dar/lib/log4j-core-2.13.3.jar 
./vco/usr/lib/vco/app-server/temp/dars/o11nplugin-vsphere.dar/lib/log4j-core-2.13.3.jar 
./vco/usr/lib/vco/app-server/lib/log4j-core-2.13.3.jar 
Scanning Java processes 
Done.

 

Step:3


If the system is vulnerable, install the KB by executing the following command on all nodes:




cd /root; base64 -d <<< "W1sgIiQoc2hhMjU2c3VtIDg3MTIwLWtiLXYyLnRhci5neiB8IGN1dCAtZiAxIC1kICcgJykiID09ICJkNDdlYzc4ZWJmZjY1M2JjNTg3Y2I5NTMwOTkwNDhlMTBhMzc5OTUzZGFjZWZlMGY5ODMzODRiOTRiMDAyZTFiIiBdXSAmJiAocm0gLXJmIC90bXAvODcxMjAta2I7IG1rZGlyIC1wIC90bXAvODcxMjAta2IvOyB0YXIgLXh2ZiA4NzEyMC1rYi12Mi50YXIuZ3ogLUMgL3RtcC84NzEyMC1rYjsgY2htb2QgK3ggL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi5zaDsgL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi5zaDsgcm0gLWZyIC90bXAvODcxMjAta2IpIHx8IGVjaG8gIkZpbGUgbm90IGZvdW5kIDg3MTIwLWtiLXYyLnRhci5neiBvciBjaGVja3N1bSBtaXNtYXRjaCIK" | bash -



 

Step:4


Make the installation effective by executing /opt/scripts/deploy.sh from the node that is typically used as primary (e.g. where /var/log/deploy.log file exists from previous runs). This is run only once across the vRA/vRO cluster nodes.








 

Step:5


Verify the KB is active on the system by running the verification command below on all nodes. There should be no error reports related to log4j.



cd /root; base64 -d <<< "W1sgIiQoc2hhMjU2c3VtIDg3MTIwLWtiLXYyLXZhbGlkYXRlLnRhci5neiB8IGN1dCAtZiAxIC1kICcgJykiID09ICJmY2IxZjQ0YWRkMmNjZTg3MzNjNzAzOTYzNjg4Njk3NjU4ZjA3NTkzMzYyZTRhMzU0MjZlYjc5OGFjMTM5YzRlIiBdXSAmJiAocm0gLXJmIC90bXAvODcxMjAta2I7IG1rZGlyIC1wIC90bXAvODcxMjAta2IvOyB0YXIgLXh2ZiA4NzEyMC1rYi12Mi12YWxpZGF0ZS50YXIuZ3ogLUMgL3RtcC84NzEyMC1rYjsgY2htb2QgK3ggL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi12YWxpZGF0ZS5zaDsgL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi12YWxpZGF0ZS5zaDsgcm0gLXJmIC90bXAvODcxMjAta2IpIHx8IGVjaG8gIkZpbGUgbm90IGZvdW5kIDg3MTIwLWtiLXYyLXZhbGlkYXRlLnRhci5neiBvciBjaGVja3N1bSBtaXNtYXRjaCIK" | bash -






 

Step:6


Validate Pods and see it's all running




 

vRealize Automation has the workaround implemented now. Go ahead and perform respective tests and it's all BAU now



 




 

201 views0 comments

Comments

Rated 0 out of 5 stars.
No ratings yet
Add a rating
bottom of page