vCenter, VMware

PSC 6.0U3 not respecting certool.cfg settings when generating VMCA CSR

After a very successful and quick migration from Windows SSO 5.5 U3e installation to a Platform Services Controller v6.0U3 appliance I was ready to get my VMCA into action.

We have a corporate internal Microsoft CA with the VMware certificate templates already created as per VMware KB 2112009. Everything was coming up Milhouse, until CSR generation time using the ‘certificate-manager’ on the PSCs.

After stepping through the ‘certificate-manager’ wizard and having the CSR and private key files sent to a directory of my choosing, I quickly inspected the CSR using openssl to make sure I was on the right track:

openssl req -in vmca_issued_csr.csr -noout -text

My CSR still had the old self-signed details of the PSC node! Sure, it was marked as a certificate authority, but contained all the default VMware self-signed details.

I had a look in the VMware pubs (specifically this bit) and found that it’s possible to generate the CSR with my own config file. Using the “certool.cfg” template config file in /usr/lib/vmware-vmca/share/config, I quickly spun out a config file to match my VMCA node details and stuck it in /tmp for the time being.

Here is how you use certool command:

/usr/lib/vmware-vmca/bin/certool –gencsr –privkey={destination of private key} –pubkey={destination of public key} –csrfile={destination of new CSR} –config={the config file I created}

And here is what I ran:

/usr/lib/vmware-vmca/bin/certool –gencsr –privkey=/root/vmca_private.key –pubkey=/root/vmca_public.key –csrfile=/root/vmca_req.csr –config=/tmp/vmca.cfg

Obviously, you can name the files whatever you like.

While this seems like it should’ve worked and should churn out a VMCA compatible intermediate CSR, it doesn’t. It only creates a CSR for a normal ‘machine’ certificate (compared to what I wanted which was a CA signing cert). I couldn’t figure out the config requirements to generate a CSR for a CA. But how was the certificate-manager doing it?

Certificate-manager is actually generating a CSR from an existing certificate while using a config file to overwrite most of the parameters. The certificate it uses is the default VMCA self-signed root certificate, and the config file is made up from your answers in the certificate-manager wizard. Cool! Maybe I’ll try this manually using the certool instead, thinking certificate-manager has regressed in Update 3. Referencing my previously crafted cartoon.cfg file in /tmp, here’s what I ran:

/usr/lib/vmware-VMCA/bin/certool –gencsrfromexistingcert –privkey=/root/vmca_private.key –pubkey=/root/vmca_public.key –csrfile=/root/vmca_req.csr –certfile=/etc/vmware-vmca/*************

Unfortunately, this didn’t work either. I still ended up with a CSR with all the details of a self signed VMCA. It definitely looks like the 6.0U3 certool has regressed and is experiencing a similar bug to 6.0U1 (6.0U1 release notes).

The only way I was able to get around it was using a temporary 6.0U2 PSC machine and using the certificate-manager tool to create the CSR and private key. The CSR and key were taken off the temporary PSC, submitted and approved to my enterprise CA with great success. I was able to use the 6.0U3 certool to install the new VMCA intermediate certificate.

Let me know in the comments if you found a fix or are experiencing the same issue.

vCenter, VMware

Empty inventory after SSO v5.5 to PSC v6.0 U3 migration

After performing the vSphere v5.5 to vSphere 6.0 migration in our testing environment with great success, I began work on our production environment. First things first, migrating Windows SSO to PSC appliance.

I had successfully converted the first machine, and started doing some testing. Things like logging into the thick client and checking all vCenter servers and basic login services.

Problem

Out of 6 vCenter servers, only 1 was having issues. Logging in with the SSO administrator account I was able to see entire inventory and all services were running just fine. However, attempting to login with my org’s domain account was met with some generic “You do not have permissions to login”. Quickly jumping over to the SSO administrator session, the permissions for the affected vCenter were completely gone, only the SSO admin was listed as an administrator.

Cause

All vCenter servers have a security setting called Active Directory Validation. Essentially, this setting will perform a synchronization of AD users and groups every X minutes with the domain that vCenter is connected to. If vCenter is unable to perform the validation (SSO is unavailable, for example) then vCenter will remove all invalidated users and groups. For my environment, vCenter was set to sync every 24 hours. This timer begins when the vCenter service starts.

In what may be the worst timing ever, I had restarted the vCenter server roughly 24 hours before I had performed my SSO->PSC migration. This resulted in vCenter attempting to validate just as SSO had become unavailable during the migration. Goodbye user and group permissions.

Fix

To get this vCenter usable, I ended up just re-adding the required ACLs to vCenter for the time being. Although, I did find a VMware KB article on how to restore your permissions from a vCenter DB backup: KB2086548

If you want to prevent this from happening on your vCenter servers, just disable the AD validation setting until you’ve finished your migrations.