Cross vCenter vMotion Utility

vSphere

Whilst upgrading the homelab I also decided to rebuild from scratch. There were some challenges to overcome because I have running VM’s I don’t want to shut while migrating.

My current homelab setup and the go to setup is documented here (work in progress). Basically it comes down to:

  • Original setup: three hosts backed with iSCSI storage for running the VM’s
  • Temporary setup:
    • New vCenter with two of the three hosts configured for vSAN with connection to the iSCSI datastores
    • Old vCenter with one remaining host running all of the VM’s
  • Destination setup: new vCenter with vSAN datastore

To migrate the virtual machines from the old environment (from the last remaining host to the two new hosts) I decided to take a look at the ‘Cross vCenter vMotion Utility‘. There is not a lot of documentation available at first sight but it is straightforward to set up and configure. Although I did find some things that are worth noting.

Step 1 : Running the jar

To start the Cross vCenter vMotion Utility one must run a jar file: ‘java -jar xvm-2.6.jar’.

I am running linux (Pop!_OS 18.04) as my OS. I have java version 8 and 11 installed with version 11 as default. Version 11 is not listed on the fling site as supported (Java Runtime Environment 1.8-10: See requirements). Running with version 11 (sudo java -jar xvm-2.6.jar) starts the local website on port 8080 (http://localhost:8080) but does not report back on the CLI.

Under the assumption that the java application started and failed rightaway, I decided to run it on my windows box which has Java Runtime environment 8 installed. The last line of feedback ‘Initialized controller with empty state’ was the same as on my linux machine. Navigating to localhost:8080 showed the Cross vCenter vMotion Utility web interface. I could now configure the application and run migrations.

It is only later when I closed the running instance on my linux box and restarting it that it showed me output on the CLI that the application started succesfully.

ps -df | grep -i java
kill -HUP 9159

Output after restart:

Step 2 : Configuration

  • Register connections
    1. Source vCenter
    2. Destination vCenter

Step 3 : Migration

  • Add migrations
    1. Source Site: source vCenter
    2. Target Site: destination vCenter
    3. Source Datacenter
    4. Virtual Machine(s): Select one or more virtual machines
    5. Placement Target: Cluster or Host
    6. Target Datastore
    7. Network Mapping(s): the utility will detect the source networks for all selected virtual machines and display a selection field for the target network

Issues

Storage vMotion?

Storage vMotion does not seem to be supported. I tried to svMotion my machines from their iSCSI based datastores to the newly created vSAN datastore but it failed.

Target Datastore: Shared datastore (same as source)

Choosing ‘Shared datastore (same as source)’ as Target Datastore fails and throws the following error:

I added the destination host and tried again but it also failed with several issues:

  • destination networks were not listed, only a subset were – although all were added to the distributed vSwitch
  • matching datastore was not found on the destination host

I could migrate to the new environment but had to select a destination datastore. This posed not much of a problem in my environment because the end goal was to get the virtual machine on the vSAN datastore.

Now after migrating most of the virtual machines, only two types of virtual machines were leftit felt like I could take a step back if needed. The a to migrate, the vCenter VM’s and the firewall VM’s. The old vCenter is not needed anymore, the new vCenter and the firewall VM’s are and once those are migrated I can go break down the last part of the old setup. The last host will be reset to default settings via the DCUI after which it can be added to the vSAN cluster and I can make the vSAN cluster setup complete. A tmp_vSAN_policy with no redundancy is not the way you (or me) want to run your environment, even if it is a lab environment.

Conclusion

I could not migrate from the old environment to the new environment while also doing a Storage vMotion, I did needed to go in steps.

Nevertheless I’m happy to have used the Cross vCenter vMotion Utility. It did save me a lot of work, required little setup and configuration. I didn’t need to change anything to the setup of my old nor my new environment.

Configuring Tesla M60 cards for NVIDIA GRID vGPU

vSphere

There are a couple of steps which need to be taken to configure the Tesla M60 cards with NVIDIA GRID VGPU in a vSphere / Horizon environment. I have listed them here quick and dirty. They are an extract of the NVIDIA Virtual GPU Software User Guide.

  • On the host(s):
    • Install the vib
      • esxcli software vib install -v directory/NVIDIA-vGPUVMware_ESXi_6.0_Host_Driver_390.72-1OEM.600.0.0.2159203.vib
    • Reboot the host(s)
    • Check if the module is loaded
      • vmkload_mod -l | grep nvidia
    • Run the nvidia-smi command to verify the correct communictation with the device
    • Configuring Suspend and Resume for VMware vSphere
      • esxcli system module parameters set -m nvidia -p “NVreg_RegistryDwords=RMEnableVgpuMigration=1”
    • Reboot the host
    • Confirm that suspend and resume is configured
      • dmesg | grep NVRM
    • Check that the default graphics type is set to shared direct
    • If the graphics type were not set to shared direct, execute the following commands to stop and start the xorg and nv-hostengine services
      • /etc/init.d/xorg stop
      • nv-hostengine -t
      • nv-hostengine -d
      • /etc/init.d/xorg start
  • On the VM / Parent VM:
    • Configure the VM, beware that once the vGPU is configured that the console of the VM will not be visible/accessible through the vSphere Client. An alternate access method should already be foreseen
    • Edit the VM configuration to add a shared pci device, verify that NVIDIA GRID vGPU is selected
    • Choose the vGPU profile
      more info on the profiles can be found here under section ‘1.4.1 Virtual GPU Types’: https://docs.nvidia.com/grid/6.0/grid-vgpu-user-guide/index.html
    • Reserve all guest memory
  • On the Horizon pool
    • Configure the pool to use the NVIDIA GRID vGPU as 3D Renderer

Unsupported upgrade of VCSA 6.5 U2 to 6.7

vSphere

We will upgrade the vCenter Server Appliance from 6.5 U2 to 6.7 though it is not supported. As this is not supported you will NOT want go ahead with this in a production environment. Maybe I will have regrets later on too … but this is my lab environment so the alternative is to redeploy a new VCSA.

I have applied the following knowledge base articles on the source VCSA

The first KB was applied because the installer is failing due to a lack of disk space on the source appliance. The installer gives the opportunity to supply a location on the source VCSA to export the necessary files that facilitate the upgrade.

The second KB was applied because the VMware Directory failed during the firstboot phase after the upgrade succeeded.

I downloaded the sources for VCSA 6.7.0 but had to go and download the sources for VCSA 6.7.0a. The VCSA 6.7.0 sources stalled at 5% on VMware Identity Management Service.

I also went to change the root password expiration to no and set the administrator@vsphere.local account password to only include alphabet characters.

The installer will also fail after the first phase if the VAMI port is not reachable, the first phase will finish succesfully though. I forgot to add an exception to my firewall. You can then continue the installer by going to the VAMI interface on port 5480.

vShield Endpoint SVM status vCenter alarm

vCenter is showing an alarm on the TrendMicro Deep Security Virtual Appliance (DSVA): ‘vShield Endpoint SVM status

Checking vShield for errors: The DSVA VA console window shows: (as to where it should show a red/grey screen)

Let’s go for some log file analysis
To get a login prompt: Alt + F2
Login with user dsva and password dsva (this is the default)
less /var/log/messages (why less is more: you get almost all the vi commands)
G to go to the last line

For some reason the ovf file is not like it is expected. The appliance is not able to set some ovf settings, in this case the network interfaces. q (to exit the log file display) sudo –s (to gain root privileges) enter the dsva user password  

test

  (to create the dsva-ovf.env file, if necessary delete the file first) reboot (to reboot the appliance, once rebooted give it 5 minutes and the alarm should clear automatically)

vCenter is showing an alarm on the TrendMicro Deep Security Virtual Appliance (DSVA): ‘vShield Endpoint SVM status Checking vShield for errors: The DSVA VA console window shows: (as to where it should show a red/grey screen) Let’s go for some log file analysis To get a login prompt: Alt + F2 Login with user dsva and password dsva (this is the default) less /var/log/messages (why less is more: you get almost all the vi commands) G to go to the last line For some reason the ovf file is not like it is expected. The appliance is not able to set some ovf settings, in this case the network interfaces. q (to exit the log file display) sudo –s (to gain root privileges) enter the dsva user password  

test

  (to create the dsva-ovf.env file, if necessary delete the file first) reboot (to reboot the appliance, once rebooted give it 5 minutes and the alarm should clear automatically)

[code language=”css”] your code here [/code]

Start or stop ESXi services using PowerCLI

vSphere

Start the ssh service on all hosts:

Get-VMHost | Foreach {
   Start-VMHostService -HostService ($_ | Get-VMHostService | Where { $_.Key -eq "TSM-SSH"} )
}

Thanks to Alan Renouf at virtu-al.net, where I found this snippet: http://www.virtu-al.net/2010/11/23/enabling-esx-ssh-via-powercli/

If you want to start the ssh service on a single host, change ESXiHostName to your ESXi FQDN:

Get-VMHost -Name ESXiHostName | Foreach {
   Start-VMHostService -HostService ( $_ | Get-VMHostService | Where { $_.Key -eq "TSM-SSH" } )
}

If you want to stop the ssh service on all hosts:

Get-VMHost | Foreach {
   Stop-VMHostService -HostService ($_ | Get-VMHostService | Where { $_.Key -eq "TSM-SSH"} )
}

If you have multiple cluster in vCenter, are connected to multiple vCenters, be sure to launch the command only to the necessary hosts:

  • Get-Cluster -Name ClusterName will filter to the specified Cluster
  • Get-VMHost -Name ESXiHostName will filter to the specified ESXi
  • Get-VMHost -Server vCenterServerName will filter to the specified vCenter server
Get-Cluster -Name ClusterName | Get-VMHost -Name ESXiHostName -Server vCenterServerName | Foreach {
   Stop-VMHostService -HostService ($_ | Get-VMHostService | Where { $_.Key -eq "TSM-SSH"} )
}

These are other services I frequently use:

  • DCUI (Direct Console UI)
  • lwsmd (Active Directory Service)
  • ntpd (NTP Daemon)
  • sfcbd-watchdog (CIM Server)
  • snmpd (SNMP Server)
  • TSM (ESXi Shell)
  • TSM-SSH (SSH)
  • vmsyslogd (Syslog Server)
  • vmware-fdm (vSphere High Availability Agent)
  • vpxa (VMware vCenter Agent)
  • xorg (X.Org Server)

There are other services available but I have never used them in this context (yet):

  • lbtd (Load-Based Teaming Daemon)
  • pcscd (PC/SC Smart Card Daemon)
  • vprobed (VProbe Daemon)

Change the startup policy for a service:

  • Automatic: Start automatically if any ports are open, and stop when all ports are closed
  • On: Start and stop with host
  • Off: Start and stop manually
get-vmhost | Foreach {Set-VMHostService -HostService ($_ | Get-VMHostService | where {$_.key -eq "tsm-ssh"}) -policy On}