LogJam: Workaround instructions to address CVE-2021-44228 in vCenter Server and vCenter Cloud Gateway (87081)

The new log4j or LogJam Expolits targets many systems, one there is a severe need to patch is VMWARE, but as of time of writing on this article, there is no patch available yet, it is being worked on, BUT here is a workaround posted, here is how to fix various version of vCenter server:


Source : (Please use link, as the article is being updated often!)

Workaround instructions to address CVE-2021-44228 in vCenter Server and vCenter Cloud Gateway (87081) (vmware.com)

 

 

 Purpose

CVE-2021-44228 has been determined to impact vCenter Server 7.0.x, vCenter 6.7.x & vCenter 6.5.x via the Apache Log4j open source component it ships.  This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

 Impact / Risks
  • VCHA needs to be removed before executing the steps in this KB article.
  • Environments with external PSCs need to have the steps taken on both vCenter and PSC appliances.
 Resolution
The workarounds described in this document are meant to be a temporary solution only.

Upgrades documented in the aforementioned advisory should be applied to remediate CVE-2021-44228 when available.

 Workaround

To apply the workaround for CVE-2021-44228 to vCenter Server 7.x, vCenter 6.7.x & vCenter 6.5.x there are workaround sections to update on the vCenter Appliance.

Note: For vCenter Cloud Gateway, only the steps for the vMon Service and Analytics Service are necessary.

Click here for vCenter Server Appliance 7.0.x workaround

Click here for vCenter Server Appliance 6.7.x workaround

Click here for vCenter Server Appliance 6.5.x workaround

Click here for vCenter Server Appliance 6.0.x workaround

vCenter Server Appliance 7.0.x Workaround
vMON Service

  1. Backup the existing java-wrapper-vmon file
cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak
  1. Update the java-wrapper-vmon file with a text editor such as vi
vi /usr/lib/vmware-vmon/java-wrapper-vmon
  1. At the very bottom of the file, replace the very last line with 2 new lines
Execute this step number ‘3’ based on the vCenter version running in your environment.
NOTE :- The below update (mentioned in Step number 3) applies ONLY to the vCenter versions listed below :-
  • vCenter 7.0 Update 3, 3a
  • vCenter 7.0 Update 2, 2a, 2b, 2c, 2d
Original
exec $java_start_bin $jvm_dynargs $security_dynargs $original_args
Updated
log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg $security_dynargs $original_argsNOTE :- The below update (mentioned in Step number 3) applies ONLY to the vCenter versions listed below :-

  • vCenter 7.0 GA, 7.0.0a, 7.0.0b, 7.0.0c, 7.0.0d
  • vCenter 7.0 Update 1, U1a, U1c, U1d
Original
exec $java_start_bin $jvm_dynargs “$@”Updated
log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg “$@”

  1. Ensure the file permissions are set correctly with the below commands:
    1. chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
    2. chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon
  2. Restart vCenter Services
service-control –stop –all
service-control –start –all

Update Manager Service

  1. Back up the existing start.ini file
cp -rfp /usr/lib/vmware-updatemgr/bin/jetty/start.ini /usr/lib/vmware-updatemgr/bin/jetty/start.ini.bak
  1. Update the start.ini file
vi /usr/lib/vmware-updatemgr/bin/jetty/start.ini
  1. Add the following line to the end of the file
-Dlog4j2.formatMsgNoLookups=true
  1. Restart the Update Manager Service

service-control –restart vmware-updatemgr

Analytics Service

  1. Back up the log4j-core-2.8.2.jar file
cp -rfp /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak
  1. Run the zip command to disable the class
zip -q -d /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
  1. Restart the Analytics service
service-control –restart vmware-analytics

DBCC Utility

  1. Back up the log4j-core-2.8.2.jar file
cp /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar.bak
  1. Run the zip command to disable the class
zip -q -d /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
Note: This command may fail with “zip error: Nothing to do!”. If so, proceed to the verification section to ensure the step was not needed. No service needs to be restarted for DBCC.

Verify the changes

Once all sections are complete, use the following steps to confirm if they were implemented successfully.

  1. Verify if the vMon services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:

ps auxww | grep formatMsgNoLookups

Check if the processes include -Dlog4j2.formatMsgNoLookups=true

  1. Verify the Update Manager changes are shown under “System Properties” in the output of the following two commands:

cd /usr/lib/vmware-updatemgr/bin/jetty/
java -jar start.jar –list-config

System Properties:
——————
log4j2.formatMsgNoLookups = true (/usr/lib/vmware-updatemgr/bin/jetty/start.ini
)

  1. Verify the Analytics Service changes:
grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l

This should return 0 lines

  1. Verify the DBCC Utility changes:

grep -i jndilookup /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar | wc -l

This should return 0 lines

vCenter Server Appliance 6.7.x Workaround

vMON Service

  1. Backup the existing java-wrapper-vmon file
cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak
  1. Update the java-wrapper-vmon file with a text editor such as vi
vi /usr/lib/vmware-vmon/java-wrapper-vmon
  1. At the very bottom of the file, replace the very last line with 2 new lines
    • Original
      exec $java_start_bin $jvm_dynargs “$@”

      Updated
      log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
      exec $java_start_bin $jvm_dynargs $log4j_arg “$@”

  2. Restart vCenter Services
service-control –stop –all
service-control –start –all

Note: If the services do not start, ensure the file permissions are set correctly with these commands:

  • chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
  • chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon

 

Analytics Service

NOTE:- The below workaround (Analytics service) is applicable for vCenter Server Appliance 6.7 Update 3o and Older versions only.  The JAR is already updated to 2.11 on the later versions.

  1. Back up the log4j-core-2.8.2.jar file
cp -rfp /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak
  1. Run the zip command to disable the class
zip -q -d /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
  1. Restart the Analytics service
service-control –restart vmware-analytics

CM Service

  1. Back up the log4j-core.jar file
cp -rfp /usr/lib/vmware-cm/lib/log4j-core.jar /usr/lib/vmware-cm/lib/log4j-core.jar.bak
  1. Run the zip command to disable the class
zip -q -d /usr/lib/vmware-cm/lib/log4j-core.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
  1. Restart the CM service
service-control –restart vmware-cm

Secure Token Service

  1. Back up and edit the the vmware-stsd file
cp /etc/rc.d/init.d/vmware-stsd /root/vmware-stsd.bak
vi /etc/rc.d/init.d/vmware-stsd
  1. Find the section labeled start_service(). Insert a new line near line 266, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
start_service()
{
perform_pre_startup_actionslocal retval
JAVA_MEM_ARGS=`/usr/sbin/cloudvm-ram-size -J vmware-stsd`
$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Dauditlog.dir=/var/log/audit/sso-events  \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start

  1. Restart the vmware-stsd service
service-control –stop vmware-stsd
service-control –start vmware-stsd

Identity Management Service

  1. Back up and edit the the vmware-sts-idmd file
cp /etc/rc.d/init.d/vmware-sts-idmd /root/vmware-sts-idmd.bak
vi /etc/rc.d/init.d/vmware-sts-idmd
  1. Insert a new line near line 177 before “$DEBUG_OPTS \” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
$JSVC_BIN -procname $SERVICE_NAME \
-wait 120 \
-server \
<snip>
-Dlog4j.configurationFile=file://$PREFIX/share/config/log4j2.xml \
-Dlog4j2.formatMsgNoLookups=true \
$DEBUG_OPTS \
$DAEMON_CLASS
  1. Restart the vmware-sts-idmd service
service-control –stop vmware-sts-idmd
service-control –start vmware-sts-idmd

Verify the changes

Once all sections are complete, use the following steps to confirm if they were implemented successfully.

  1. Verify if the stsd, idmd, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:

ps auxww | grep formatMsgNoLookups

Check if the processes include -Dlog4j2.formatMsgNoLookups=true

  1. Verify the Analytics Service changes:
grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l
This should return 0 lines
  1. Verify the CM Service changes:

grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l

This should return 0 lines

vCenter Server Appliance 6.5.x Workaround 

vMON Service

  1. Backup the existing java-wrapper-vmon file
cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak
  1. Update the java-wrapper-vmon file with a text editor such as vi
vi /usr/lib/vmware-vmon/java-wrapper-vmon
  1. At the very bottom of the file, replace the very last line with 2 new lines
    • Original
      exec $java_start_bin $jvm_dynargs “$@”

      Updated
      log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
      exec $java_start_bin $jvm_dynargs $log4j_arg “$@”

  2. Restart vCenter Services
service-control –stop –all
service-control –start –all

Note: If the services do not start, ensure the file permissions are set correctly with these commands:

  • chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
  • chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon

CM Service

  1. Back up the log4j-core.jar file
cp -rfp /usr/lib/vmware-cm/lib/log4j-core.jar /usr/lib/vmware-cm/lib/log4j-core.jar.bak
  1. Run the zip command to disable the class
zip -q -d /usr/lib/vmware-cm/lib/log4j-core.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
  1. Restart the CM service
service-control –restart vmware-cm

Secure Token Service

  1. Back up and edit the the vmware-stsd file
cp /etc/rc.d/init.d/vmware-stsd /root/vmware-stsd.bak
vi /etc/rc.d/init.d/vmware-stsd
  1. Find the section labeled start_service(). Insert a new line near line 266, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
start_service()
{
perform_pre_startup_actionslocal retval
$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Dauditlog.dir=/var/log/audit/sso-events  \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start

  1. Restart the vmware-stsd service
service-control –stop vmware-stsd
service-control –start vmware-stsd

Identity Management Service

  1. Back up and edit the the vmware-sts-idmd file
cp /etc/rc.d/init.d/vmware-sts-idmd /root/vmware-sts-idmd.bak
vi /etc/rc.d/init.d/vmware-sts-idmd
  1. Insert a new line near line 177 before “$DEBUG_OPTS \” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
$JSVC_BIN -procname $SERVICE_NAME \
-wait 120 \
-server \
<snip>
-Dlog4j.configurationFile=file://$PREFIX/share/config/log4j2.xml \
-Dlog4j2.formatMsgNoLookups=true \
$DEBUG_OPTS \
$DAEMON_CLASS
  1. Restart the vmware-sts-idmd service
service-control –stop vmware-sts-idmd
service-control –start vmware-sts-idmd

PSC Client Service

  1. Back up and edit the vmware-psc-client file
cp -rfp /etc/rc.d/init.d/vmware-psc-client /root/vmware-psc-client.bak
vi /etc/rc.d/init.d/vmware-psc-client
  1. Insert a new line near line 300, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:

$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Djava.io.tmpdir=”$CATALINA_BASE/temp” \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start

  1. Restart the vmware-psc-client service
service-control –stop vmware-psc-client
service-control –start vmware-psc-client

Verify the changes

Once all sections are complete, use the following steps to confirm if they were implemented successfully.

  1. Verify if the stsd, idmd, psc-client, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:

ps auxww | grep formatMsgNoLookups

Check if the processes include -Dlog4j2.formatMsgNoLookups=true

  1. Verify the CM Service changes:

grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l

This should return 0 lines

vCenter Server Appliance 6.0 U3j Workaround

vCenter Server Appliance 6.0 U3j is no longer in general support but has also been identified as vulnerable to CVE-2021-44228 due to the Performance Charts service. Mitigation steps have been identified as follows:

  1. Back up and edit /usr/lib/vmware-perfcharts/wrapper/conf/wrapper.conf on the appliance and add a new line just below “wrapper.java.additional.13=-Dlog4j.configurationFile=file:/etc/vmware-perfcharts/log4j2.xml” (line 72) with the following content:

wrapper.java.additional.14=-Dlog4j2.formatMsgNoLookups=true

  1. Save the file, stop the service and then start it through service-control:

service-control –stop vmware-perfcharts
service-control –start vmware-perfcharts

NotevCenter Server Appliance versions 6.0GA – 6.0U3i are not vulnerable. However, versions 6.0 U3a/b/c/d/e/f were found to contain the following unused vulnerable jar files. No impact on the product has been observed after removing these jar files.

  • /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-p2p/lib/log4j-core-2.1.jar
  • /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-p2p/lib/log4j-api-2.1.jar
  • /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-cs/lib/log4j-core-2.1.jar
  • /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-cs/lib/log4j-api-2.1.jar
 Related Information
To revert the workarounds, replace the modified files with the backups created in each step.
VCHA needs to be disabled before executing the steps in this KB.Change log:

  • December 12th 2021 – 11:20 PST: Updated Knowledge Base article advising workarounds for 6.5/6.7 partially address the vulnerability.
  • December 12th 2021 – 16:00 PST: Added steps for stsd, idmd and psc-client services to the workaround sections for 6.5 and 6.7.
  • December 12th 2021 – 17:00 PST: Added workaround for VCSA 6.0U3j.
  • December 12th 2021 – 18:30 PST: Added additional information for older versions of VCSA 6.0.
  • December 12th 2021 – 20:30 PST: Updated versions with vulnerable jar files in VCSA 6.0.
  • December 13th 2021 – 1:57 PST: Update the steps in the vMON service commands for permission validation

Leave a Comment

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close