The new log4j or LogJam Expolits targets many systems, one there is a severe need to patch is VMWARE, but as of time of writing on this article, there is no patch available yet, it is being worked on, BUT here is a workaround posted, here is how to fix various version of vCenter server:
Source : (Please use link, as the article is being updated often!)
CVE-2021-44228 has been determined to impact vCenter Server 7.0.x, vCenter 6.7.x & vCenter 6.5.x via the Apache Log4j open source component it ships. This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:
- CVE-2021-44228 – VMSA-2021-0028
- VCHA needs to be removed before executing the steps in this KB article.
- Environments with external PSCs need to have the steps taken on both vCenter and PSC appliances.
Upgrades documented in the aforementioned advisory should be applied to remediate CVE-2021-44228 when available.
To apply the workaround for CVE-2021-44228 to vCenter Server 7.x, vCenter 6.7.x & vCenter 6.5.x there are workaround sections to update on the vCenter Appliance.
Note: For vCenter Cloud Gateway, only the steps for the vMon Service and Analytics Service are necessary.
Click here for vCenter Server Appliance 7.0.x workaround
Click here for vCenter Server Appliance 6.7.x workaround
Click here for vCenter Server Appliance 6.5.x workaround
Click here for vCenter Server Appliance 6.0.x workaround
vCenter Server Appliance 7.0.x Workaround
vMON Service
- Backup the existing java-wrapper-vmon file
- Update the java-wrapper-vmon file with a text editor such as vi
- At the very bottom of the file, replace the very last line with 2 new lines
NOTE :- The below update (mentioned in Step number 3) applies ONLY to the vCenter versions listed below :-
- vCenter 7.0 Update 3, 3a
- vCenter 7.0 Update 2, 2a, 2b, 2c, 2d
exec $java_start_bin $jvm_dynargs $security_dynargs $original_args
log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg $security_dynargs $original_argsNOTE :- The below update (mentioned in Step number 3) applies ONLY to the vCenter versions listed below :-
- vCenter 7.0 GA, 7.0.0a, 7.0.0b, 7.0.0c, 7.0.0d
- vCenter 7.0 Update 1, U1a, U1c, U1d
exec $java_start_bin $jvm_dynargs “$@”Updated
log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg “$@”
- Ensure the file permissions are set correctly with the below commands:
- chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
- chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon
- Restart vCenter Services
service-control –start –all
Update Manager Service
- Back up the existing start.ini file
- Update the start.ini file
- Add the following line to the end of the file
- Restart the Update Manager Service
service-control –restart vmware-updatemgr
Analytics Service
- Back up the log4j-core-2.8.2.jar file
- Run the zip command to disable the class
- Restart the Analytics service
DBCC Utility
- Back up the log4j-core-2.8.2.jar file
- Run the zip command to disable the class
Verify the changes
Once all sections are complete, use the following steps to confirm if they were implemented successfully.
- Verify if the vMon services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:
ps auxww | grep formatMsgNoLookups
Check if the processes include -Dlog4j2.formatMsgNoLookups=true
- Verify the Update Manager changes are shown under “System Properties” in the output of the following two commands:
cd /usr/lib/vmware-updatemgr/bin/jetty/
java -jar start.jar –list-config
System Properties:
——————
log4j2.formatMsgNoLookups = true (/usr/lib/vmware-updatemgr/bin/jetty/start.ini)
- Verify the Analytics Service changes:
This should return 0 lines
- Verify the DBCC Utility changes:
grep -i jndilookup /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar | wc -l
This should return 0 lines
vCenter Server Appliance 6.7.x Workaround
vMON Service
- Backup the existing java-wrapper-vmon file
- Update the java-wrapper-vmon file with a text editor such as vi
- At the very bottom of the file, replace the very last line with 2 new lines
-
Originalexec $java_start_bin $jvm_dynargs “$@”
Updated
log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg “$@”
-
- Restart vCenter Services
service-control –start –all
Note: If the services do not start, ensure the file permissions are set correctly with these commands:
- chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
- chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon
Analytics Service
NOTE:- The below workaround (Analytics service) is applicable for vCenter Server Appliance 6.7 Update 3o and Older versions only. The JAR is already updated to 2.11 on the later versions.
- Back up the log4j-core-2.8.2.jar file
- Run the zip command to disable the class
- Restart the Analytics service
CM Service
- Back up the log4j-core.jar file
- Run the zip command to disable the class
- Restart the CM service
Secure Token Service
- Back up and edit the the vmware-stsd file
- Find the section labeled start_service(). Insert a new line near line 266, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
{
perform_pre_startup_actionslocal retval
JAVA_MEM_ARGS=`/usr/sbin/cloudvm-ram-size -J vmware-stsd`
$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Dauditlog.dir=/var/log/audit/sso-events \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start
- Restart the vmware-stsd service
service-control –start vmware-stsd
Identity Management Service
- Back up and edit the the vmware-sts-idmd file
- Insert a new line near line 177 before “$DEBUG_OPTS \” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
-wait 120 \
-server \
<snip>
-Dlog4j.configurationFile=file://$PREFIX/share/config/log4j2.xml \
-Dlog4j2.formatMsgNoLookups=true \
$DEBUG_OPTS \
$DAEMON_CLASS
- Restart the vmware-sts-idmd service
service-control –start vmware-sts-idmd
Verify the changes
Once all sections are complete, use the following steps to confirm if they were implemented successfully.
- Verify if the stsd, idmd, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:
ps auxww | grep formatMsgNoLookups
Check if the processes include -Dlog4j2.formatMsgNoLookups=true
- Verify the Analytics Service changes:
- Verify the CM Service changes:
grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l
This should return 0 lines
vCenter Server Appliance 6.5.x Workaround
vMON Service
- Backup the existing java-wrapper-vmon file
- Update the java-wrapper-vmon file with a text editor such as vi
- At the very bottom of the file, replace the very last line with 2 new lines
-
Originalexec $java_start_bin $jvm_dynargs “$@”
Updated
log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg “$@”
-
- Restart vCenter Services
service-control –start –all
Note: If the services do not start, ensure the file permissions are set correctly with these commands:
- chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
- chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon
CM Service
- Back up the log4j-core.jar file
- Run the zip command to disable the class
- Restart the CM service
Secure Token Service
- Back up and edit the the vmware-stsd file
- Find the section labeled start_service(). Insert a new line near line 266, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
{
perform_pre_startup_actionslocal retval
$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Dauditlog.dir=/var/log/audit/sso-events \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start
- Restart the vmware-stsd service
service-control –start vmware-stsd
Identity Management Service
- Back up and edit the the vmware-sts-idmd file
- Insert a new line near line 177 before “$DEBUG_OPTS \” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
-wait 120 \
-server \
<snip>
-Dlog4j.configurationFile=file://$PREFIX/share/config/log4j2.xml \
-Dlog4j2.formatMsgNoLookups=true \
$DEBUG_OPTS \
$DAEMON_CLASS
- Restart the vmware-sts-idmd service
service-control –start vmware-sts-idmd
PSC Client Service
- Back up and edit the vmware-psc-client file
vi /etc/rc.d/init.d/vmware-psc-client
- Insert a new line near line 300, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Djava.io.tmpdir=”$CATALINA_BASE/temp” \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start
- Restart the vmware-psc-client service
service-control –start vmware-psc-client
Verify the changes
Once all sections are complete, use the following steps to confirm if they were implemented successfully.
- Verify if the stsd, idmd, psc-client, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:
ps auxww | grep formatMsgNoLookups
Check if the processes include -Dlog4j2.formatMsgNoLookups=true
- Verify the CM Service changes:
grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l
This should return 0 lines
vCenter Server Appliance 6.0 U3j Workaround
vCenter Server Appliance 6.0 U3j is no longer in general support but has also been identified as vulnerable to CVE-2021-44228 due to the Performance Charts service. Mitigation steps have been identified as follows:
- Back up and edit /usr/lib/vmware-perfcharts/wrapper/conf/wrapper.conf on the appliance and add a new line just below “wrapper.java.additional.13=-Dlog4j.configurationFile=file:/etc/vmware-perfcharts/log4j2.xml” (line 72) with the following content:
wrapper.java.additional.14=-Dlog4j2.formatMsgNoLookups=true
- Save the file, stop the service and then start it through service-control:
service-control –stop vmware-perfcharts
service-control –start vmware-perfcharts
Note: vCenter Server Appliance versions 6.0GA – 6.0U3i are not vulnerable. However, versions 6.0 U3a/b/c/d/e/f were found to contain the following unused vulnerable jar files. No impact on the product has been observed after removing these jar files.
- /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-p2p/lib/log4j-core-2.1.jar
- /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-p2p/lib/log4j-api-2.1.jar
- /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-cs/lib/log4j-core-2.1.jar
- /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-cs/lib/log4j-api-2.1.jar
VCHA needs to be disabled before executing the steps in this KB.Change log:
- December 12th 2021 – 11:20 PST: Updated Knowledge Base article advising workarounds for 6.5/6.7 partially address the vulnerability.
- December 12th 2021 – 16:00 PST: Added steps for stsd, idmd and psc-client services to the workaround sections for 6.5 and 6.7.
- December 12th 2021 – 17:00 PST: Added workaround for VCSA 6.0U3j.
- December 12th 2021 – 18:30 PST: Added additional information for older versions of VCSA 6.0.
- December 12th 2021 – 20:30 PST: Updated versions with vulnerable jar files in VCSA 6.0.
- December 13th 2021 – 1:57 PST: Update the steps in the vMON service commands for permission validation
Related Posts
Microsoft 365: Get started with the Migration Manager
VMWARE: [Errno 28] No space left on device on ESXi 6.7
vCenter Server Appliance: ROOT password expired (6.x)
Sophos UTM: How to reboot specific Sophos Access Point
Sophos XG Firewall: How to set up Sophos authentication for thin client (SATC)
Exchange Server 2019 on Windows Server 2019 Core