vCenter, VMware

Empty inventory after SSO v5.5 to PSC v6.0 U3 migration

After performing the vSphere v5.5 to vSphere 6.0 migration in our testing environment with great success, I began work on our production environment. First things first, migrating Windows SSO to PSC appliance.

I had successfully converted the first machine, and started doing some testing. Things like logging into the thick client and checking all vCenter servers and basic login services.

Problem

Out of 6 vCenter servers, only 1 was having issues. Logging in with the SSO administrator account I was able to see entire inventory and all services were running just fine. However, attempting to login with my org’s domain account was met with some generic “You do not have permissions to login”. Quickly jumping over to the SSO administrator session, the permissions for the affected vCenter were completely gone, only the SSO admin was listed as an administrator.

Cause

All vCenter servers have a security setting called Active Directory Validation. Essentially, this setting will perform a synchronization of AD users and groups every X minutes with the domain that vCenter is connected to. If vCenter is unable to perform the validation (SSO is unavailable, for example) then vCenter will remove all invalidated users and groups. For my environment, vCenter was set to sync every 24 hours. This timer begins when the vCenter service starts.

In what may be the worst timing ever, I had restarted the vCenter server roughly 24 hours before I had performed my SSO->PSC migration. This resulted in vCenter attempting to validate just as SSO had become unavailable during the migration. Goodbye user and group permissions.

Fix

To get this vCenter usable, I ended up just re-adding the required ACLs to vCenter for the time being. Although, I did find a VMware KB article on how to restore your permissions from a vCenter DB backup: KB2086548

If you want to prevent this from happening on your vCenter servers, just disable the AD validation setting until you’ve finished your migrations.

vCenter, vCloud Director, VMware

SYSTEM_REFRESH_VIMSERVER – Could not register vCloud Director as an extension to vCenter Server

While trying to troubleshoot another problem, we tried Refreshing vCloud to vCenter which includes registering/updating the extension. This is when we hit a beauty we’d never seen before:

vimserver refresh

Alright, calm down. Probably something with the network, right? And  if it’s not the network then it’s probably DNS. Right? Wrong.

I dug around in the vCenter MOB and found the vCloud Director extension. As expected it already had a “vCloud Director-1” named extension. What I found odd was the last heartbeat time was back in 2013. Interestingly enough the last version recorded was also v5.1.2. I say interestingly because we are running v8.10.1 for SP.

Jumping into our test environment, I performed a Refresh of our test vCloud instance to vCenter and lo and behold it happened there too! I couldn’t find anything in the vCloud logs reporting the why behind this failure, but I needed to get this running and quick, too.

Knowing that the vCloud DB stores its own references to the vCenter MOB, and that vCloud would try to register itself as vCloud Director-1 again, I theorised that we could remove the existing extension and perform another Refresh without causing any issues.

So, that’s what I did right in the test environment. It went without a hitch. Rolled the same change out in production and it went beautifully.

If you’re getting this error, I’d suggest taking a backup of your vCenter server/DB and removing the existing vCloud Director extension.

Removing the extension (from KB1025360):

  1. In a web browser, navigate to http://vCenter_Server_name_or_IP/mob.
    Where vCenter_Server_name_or_IP/mob is the name of your vCenter Server or its IP address.
  2. Click Content.
  3. Click ExtensionManager.
  4. Select and copy the name of the plug-in you want to remove from the list of values under Properties. For a list of default plug-ins, see the Additional Information section of this article.
  5. Click UnregisterExtension. A new window appears.
  6. Paste the key of the plug-in and click Invoke Method. This removes the plug-in and results in void.
  7. Close the window.
  8. Refresh the Managed Object Type:ManagedObjectReference:ExtensionManager window to verify that the plug-in is removed successfully

Now go back to vCloud and perform a Refresh against your vCenter server. You should be back in action now!

vCenter, VMware

vSphere 6 certificate templates with SHA256 encryption

I was just in the middle of configuring a PSC 6.0 node’s VMCA as an intermediate CA and, in traditional fashion, went to request a certificate from a 2008 R2 Microsoft CA using the web enrollment form (as per this VMware KB article).

Oddly enough though my brand spanking new vSphere 6.0 machine and intermediate CA certificate templates were missing from the template selection drop down.

I had a look around online and found that MS CA v3 certificate templates are not supported in the web enrollment form. Why is this relevant? Well, this VMware KB states that if you use SHA256 encryption in your environment you must select Windows Server 2008 Enterprise as your certificate template version. That instantly sets your certificate templates to v3.

Damn. How was I going to submit my CSR to this Microsoft CA and get back my certificates?! The Certificate Management snap-in doesn’t allow CSR files to be submitted. It’s just not an option.

Luckily we have the trusty certreq tool. I was easily able to submit my CSR file to the Microsoft CA and get a certificate back in a simple command:

Certreq -submit -attrib "certificateTemplate:vSphere6.0VMCA" vmca_issued_csr.csr

Make sure you specify the correct certificate template. In my example above, I was after the VMCA intermediate CA template. The file specified was in my cmd working directory and is the same file the PSC’s spit out when you’re using the certificate manager tool.

PowerCLI, vCenter, VMware

Copy-VMGuestFile returns 403 Forbidden error

Got this error just today and couldn’t figure out why. I was trying to copy from my management server to a test VM with no network connectivity, and was receiving the following error:

Copy-VMGuestFile : 5/01/2017 11:47:40 AM Copy-VMGuestFile The remote server returned an error: (403) Forbidden. 
At line:1 char:1
+ Copy-VMGuestFile -Source "REDACTED" -Destination " ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CategoryInfo : NotSpecified: (:) [Copy-VMGuestFile], ViError
 + FullyQualifiedErrorId : Client20_VmGuestServiceImpl_UploadFileToGuest_UploadError,VMware.VimAutomation.ViCore.Cmdlets.Commands.CopyVMGuestFile

My privileges and network connectivity to vCenter and the ESXi hosts was looking good. Until I tripled checked my network port access to the ESXi host.

I was missing port 903 from the management server to the ESXi host the VM was sitting on. Opening that up allowed me to execute the command.

Check your ports people.

vCenter, VMware

Update Export Failed – Converting Windows SSO to PSC Appliance

This isn’t a be all and end all post on converting your Windows-based SSO server to the Platform Services Controller appliance, although I found an issue when performing the migration.

We kept receiving an “Update export failed” message when the appliance was deployed by the conversion wizard. We couldn’t understand why, and the appliance updaterunner.log file gave us no clues as to what it could be.

Turns out, you must run the vcsa_setup.html wizard with the same domain user/admin account that you started the Migration-Assistance.exe process with.