UPDATE 23/7-2024: Solved in: VMware vCenter Server 8.0 Update 3a Release Notes
When updating vCenter server to version 8.0 U3, it will mostly fail, and showing you this:
- Log file
/var/log/vmware/applmgmt/Patchrunner.log
will show below entries:/var/log/vmware/applmgmt/Patchrunner.log:
vmidentity:Expand INFO vmidentity.utils Executing: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'list', '--store', 'TRUSTED_ROOTS']
vmidentity:Expand INFO vmidentity.utils get all certs completed.
vmidentity:Expand INFO vmidentity Found <> certs in TRUSTED_ROOTS store
vmidentity:Expand INFO vmidentity CN of cert_3=ssoserver
vmidentity:Expand INFO vmidentity Found ssoserver cert in TRUSTED_ROOTS, This will be deleted from store
vmidentity:Expand INFO vmidentity.utils Deleting cert from TRUSTED_ROOTS VECS store
vmidentity:Expand ERROR vmidentity.utils Failed to execute command '['/usr/lib/vmware-vmafd/bin/dir-cli', 'trustedcert', 'unpublish', '--cert', '/storage/seat/software-updateub8jty50/stage/scripts/patches/payload/components-script/vmidentity/<Cert_filename.pem>', '--login', '<VC FQDN>']'
vmidentity:Expand ERROR vmidentity.utils dir-cli failed. Error 1168: Operation failed with error ERROR_NOT_FOUND (1168)
vmidentity:Expand ERROR vmidentity Something went wrong while reading certs from TRUSTED_ROOTS or deleting ssoserver cert: %s
vmidentity:Expand ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vmidentity:Expand' failed.
Traceback (most recent call last):
File "/storage/seat/software-updateub8jty50/stage/scripts/patches/payload/components-script/vmidentity/utils.py", line 275, in delete_ssoserver_cert
raise Exception("Failed to execute dir-cli unpublish cert")
Exception: Failed to execute dir-cli unpublish cert
This is a known issue with vCenter Server 8.0 U3, VMWARE engineering is actively working towards fixing this issue.
Workaround
To workaround this issue, remove the Non-CA Certificate with CN “ssoserver” from TRUSTED_ROOTS of vCenter Server by following any of the below methods:
Scripted Workaround
To workaround this issue using the included fix_ssoserver_cert.sh
script, follow the steps below:
- Download the attached
fix_ssoserver_cert.sh
and copy it to the vCenter Appliance - To list certificates in the TRUSTED_ROOTS store, run the script with the “list” flag
bash ./fix_ssoserver_cert.sh list
- To clean up any certificates with ssoserver anywhere in the subject, run the script with the “clean” flag.
bash ./fix_ssoserver_cert.sh clean
- Verify the ssoserver certificate is removed by re-running the script with the “list” flag
- Resume the vCenter Server update.
Manual Workaround
To workaround this issue using vecs-cli to remove the Non-CA Certificate with CN “ssoserver
” from TRUSTED_ROOTS
, use the below steps :
- SSH to the vCenter Server
- List the Certificates and identify the Alias of Non-CA Certificate with CN=ssoserver
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | egrep 'Alias|ssoserver|Key Usage' -A 1 | egrep -v 'Entry type|--'
- Take a backup of the certificate
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias <Alias> --output /var/tmp/non_ca_ssoserver.crt
Note: Replace
<Alias>
with the Alias ID identified from above step. - Remove the certificate from VECS Store
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias <Alias> -y
- Perform Certificate list again and make sure the certificate is removed
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | egrep 'Alias|ssoserver|Key Usage' -A 1 | egrep -v 'Entry type|--'
- Resume the vCenter Server update