When you connect to your ESXi host and you launch esxtop. You look at the esxtop output and it is not displaying as it should. Instead, it is displaying like in the below screenshot:
Your esxtop output will be displayed correctly if you are using a terminal emulator that defaults to xterm as the TERM environment variable. Some terminal emulators will use another terminal emulator value by default, eg. xterm-256color. ESXi does not map xterm-256color to one of the values it knows, so it doesn’t know how to display the output.
There is a KB article that explains how to resolve:
The value of the environment variable TERM is used by the server to control how input is recognized by the system, and what capabilities exist for output.
Let us have a look first what the TERM variable is in my case:
I am receiving the following output:
My terminal emulator tries to connect to the endpoint (ESXi) with xterm-256color. Now let’s take a look at what values this endpoint does support:
So all of the above is possible to assign to TERM. The value my terminal emulator uses is not among the supported terminfo types. So the ESXi host cannot map to any of the known and thus does not know how to display the esxtop info correctly.
When we update the TERM environment variable to xterm and try to run esxtop again, the output will show nicely formatted.
Let’s check esxtop again to make sure the outcome is as expected:
This post is not as a end-to-end upgrade guide but a methodology guide. Everything is more or less straight forward if one uses the correct methodology.
As this is a home lab I have chosen not to add complexity by adding additional nodes to the various components for High Availability but depend on vSphere HA. This eases and shortens my upgrade path. In a production environment every application should be evaluated and talked through with the stakeholders whether it can rely on vSphere HA or some form of application HA should be introduced.
The release of vSphere 7 introduced a starting point for an update/upgrade round in the lab. I have not always used the methodology when upgrading components as I should have in the lab. When I used this methodology in the current upgrade round it has come to light that some components were not interoperable with each other. Why is this important? If you have a problem with your environment and you call VMware support, they will go through the logs and verify the environment is on par with the documentation.
So where do I start?
Reading the docs first takes time but it will save a lot of time in the long run. Can’t stress it enough RTFM !!
I always create my own high level upgrade guide. This will include the research that I have done and it also includes the upgrade path to follow.
Phase 1: Information gathering
Determine the components and versions
Be thorough in determining the components and versions (BOM or Bill Of Materials).
VMware vSphere 6.7U3
vRealize Log Insight 4.5
vRealize Operations 7.5
VMware Horizon 7.11
You will see below that I had forgotten to include VMware Horizon in my component set. This could have been catastrophic as some components might stop to work when you don’t follow the correct upgrade procedure.
Gather the documentation
After you determine which components are used in the environment you can go and search for the necessary documentation. Find the release notes, upgrade guides and other relevant documentation
What I learned from these documents was that I was not sure the NVMe drives in my hosts would be compatible. After all they are consumer grade NVMe drives and are not on the VMware HCL. I recently installed two NVMe drives, a CT500P1SSD8 and a CT1000P1SSD8 and at the time of install the Crucial CT500P1SSD8 was not recognized. A quick googling showed me a blog post from William Lam that replaced the vSphere 6.7 U3 nvme driver with one from a vSphere 6.5U2 install. I will discuss how I determined if there would be issues around this in ESXi 7.0 in a later phase.
All components should be checked with the vendor and the VMware HCL. Be aware that the vendor and VMware might not always agree and that the VMware HCL might not always be in sync with the the documentation of the hardware vendor. You should always follow the VMware HCL but be aware of the following KB. If vSAN hardware is involved it is advised to use extreme caution as this has a specific section of the VMware HCL for vSAN Ready Nodes and for Build Your Own
VMware Product Interoperability Matrix overview
I did a research on which versions would be compatible with vSphere 6.7 and vSphere 7.0 as I was not yet sure if would be able to upgrade to vSphere 7.0
One product I forgot about was VMware Horizon. I’m currently on 7.10 and the VMware Product Interoperability Matrix show that at least 7.12 is needed to be supported. As I currently use Horizon with a full clone this should not pose much problems and I am planning to upgrade this as well. If I would be using Linked Clones or Instant Clones this could have been worse.
Update sequence for vSphere 7.0 and its compatible VMware products
Now that we checked all the info on whether everything would be compatible and supported after the upgrade, it is time to check the knowledge base article on the update sequence. This article shows what must be upgraded before another component can be upgraded to keep a supported environment.
Phase 2: The upgrade
vRealize Log Insight upgrade
vRealize Log Insight was running version 4.6. Digging back through the release notes showed me that I had to upgrade from 4.6 > 4.7(.1) > 4.8 > 8.1. The VMware Product Interoperability Matrix showed that vRealize Log Insight 8.1 was compatible with either vSphere 6.7U3 and vSphere 7.0
The upgrade process was painless. It just took a lot of time. The process itself is straight forward. Go to Administration > Management > Cluster, upload the pak file and follow the screens. In my case again and again because I did not upgrade Log Insight a long time.
vRealize Operations Manager upgrade
Upgrading the vRealize Operations Manager node is a breeze too, mainly because it is a simple setup with only one master node. vRealize Operations Manager was running version 7.5. I missed the 8.1 release so I upgraded to 8.0 first.
There are a couple of things that need some attention.
Always run the vRealize Operations Manager Pre-Upgrade Readiness Assessment Tool (APUAT pak)
Make sure to upgrade the OS through the OS pak files first, then the vROps pak file
As I upgraded to 8.0 I had to switch files to execute the 8.1 upgrade
The Pre-Upgrade Readiness Assessment Tool showed me warnings for two items:
Validating product version Make sure to run vRealize Operations Manager – 6.6.1, 6.7, 7.0 and 7.5 Virtual Appliance upgrade, as product version is 7.5.0 Ensure product and upgrade versions meet the requirements.
Checking /dev/sda partition size. The size of the partition is less than 20GB. Increase the size of the partition to be greater or equal to 20 GB (https://kb.vmware.com/s/article/75298).
Both are easily addressed. The first one gives a warning to use the correct pak file when upgrading. The second one refers to a KB article that has only a couple of steps:
take vRealize Operations offline
shut down guest OS
increase hard disk in vCenter
boot Virtual Machine
take vRealize Operations online
After addressing both warnings I was able to upgrade.
Simple and easy when you are already on the VCSA with an embedded Platform Services Controller (PSC).
Run the installer and choose upgrade. Supply the source vCenter information and destination vCenter information and click Next – Next – Finish. Grab a drink and wait. It is a two part process. The first part will deploy the new machine with the chosen information and the second part will migrate the data from the old vCenter to the new vCenter.
Before the actual upgrade could take place I needed to be sure that everything would work after the upgrade. Within the vExpert vCommunity I had seen a nice and easy way to do this. I am sorry that I can’t give credits to the person that I got the idea from.
Create a bootable USB ESXi installer or use the iDRAC or equivalent technology to boot your server from the ESXi installer
Find an empty USB flash drive
Put your server in Maintenance Mode
Shutdown and boot your server from the USB installer or the iDRAC or equivalent technology
Install to the empty USB drive – BE CAREFUL not to install to the wrong location
Upgrade check workaround
ESXi will create a VMFS volume from the remaining local space where ESXi is installed by default. After installing I tried to add the ESXi host to vCenter but failed because it had detected the local VMFS volume from the original install and that was conflicting with the one that was still present in vCenter but disconnected. I rebooted the ESXi host, booted into the original drive, verified nothing was on the local drive in the original install and deleted the datastore. Rebooted again into the USB drive and now could add the USB installed ESXi 7.0 to vCenter. Now I was able to get a glimpse of how everything was seen from an ESXi 7.0 install. The NVMe drives I was worried about were showing all fine.
Again this is a home lab and not all components are on the VMware HCL so this adds some extra steps like checking from an actual install. This would not be necessary in a production environment where everything has green checks on the VMware HCL.
The actual upgrade
Upgrading ESXi hosts is done easiest through VMware vSphere – Lifecycle Manager (VMware Update Manager has been rebranded)
I imported the Dell customized ISO, created a baseline and did a Host Compliance Check. The Host Compliance Check was Incompatible and led me to the following two knowledge base articles:
I had to remove everything based on the qedi and qedf drivers
VMware Tools / VM Hardware Compatibility
Upgrading the VMware Tools and the VM Hardware Compatibility is the last part in the process. Determine the viability of each VM to upgrade the VMware Tools and the VM Hardware Compatibility. For most VMs this won’t pose a problem. Nevertheless there are some vendor appliances that will need to run a specific version.
Although vSAN is not really a separate component to upgrade, you will need to upgrade the on-disk format. This is an online upgrade that will not impact the running VMs.
Sometimes you want/need use iPerf to test the nic speed between two ESXi hosts. I did because I was seeing a NIC with low throughput in my lab.
How can we test raw speeds between the two hosts? iPerf comes to the rescue. I was looking on how to do this on an ESXi host. I doesn’t come as a surprise that I found the solution here at William Lams’ virtuallyghetto.com. Apparently iperf has been added to ESXi since 6.5 U2. You used to have to copy iperf to iperf.copy. In ESXi 7.0 that has been done for you, although you will need to look for /usr/lib/vmware/vsan/bin/iperf3.copy
ESXi host 1 (iperf server)
Disable the firewall:
esxcli network firewall set--enabled false
Change to the directory containing the iperf binary
Execute iPerf as server
Overview of the used parameters:
will start iperf as server
defines the IP the iperf server will listen to
Disable the firewall
esxcli network firewall set--enabled false
ESXi host 2 (iperf client)
Change to the directory containing the iperf binary
Execute iPerf as client
Overview of the used parameters:
will determine the interval of reporting back
time iperf will be running
client ip, will force the usage of the correct vmkernel interface
defaults to kbit/s, adding m will use mbit/s
Don’t forget to re-enable the firewall on both systems.
Whilst upgrading the home lab I also decided to rebuild from scratch. There were some challenges to overcome because I have running VMs I don’t want to shut while migrating.
My current home lab setup and the go to setup is documented here (work in progress). Basically it comes down to:
Original setup: three hosts backed with iSCSI storage for running the VMs
New vCenter with two of the three hosts configured for vSAN with connection to the iSCSI datastores
Old vCenter with one remaining host running all of the VMs
Destination setup: new vCenter with vSAN datastore
To migrate the virtual machines from the old environment (from the last remaining host to the two new hosts) I decided to take a look at the ‘Cross vCenter vMotion Utility‘. There is not a lot of documentation available at first sight but it is straightforward to set up and configure. Although I did find some things that are worth noting.
Step 1 : Running the jar
To start the Cross vCenter vMotion Utility one must run a jar file: ‘java -jar xvm-2.6.jar’.
I am running linux (Pop!_OS 18.04) as my OS. I have java version 8 and 11 installed with version 11 as default. Version 11 is not listed on the fling site as supported (Java Runtime Environment 1.8-10: See requirements). Running with version 11 (sudo java -jar xvm-2.6.jar) starts the local website on port 8080 (http://localhost:8080) but does not report back on the CLI.
Under the assumption that the java application started and failed right away, I decided to run it on my windows box which has Java Runtime environment 8 installed. The last line of feedback ‘Initialized controller with empty state’ was the same as on my linux machine. Navigating to localhost:8080 showed the Cross vCenter vMotion Utility web interface. I could now configure the application and run migrations.
It is only later when I closed the running instance on my linux box and restarting it that it showed me output on the CLI that the application started successfully.
Output after restart:
Step 2 : Configuration
Step 3 : Migration
Source Site: source vCenter
Target Site: destination vCenter
Virtual Machine(s): Select one or more virtual machines
Placement Target: Cluster or Host
Network Mapping(s): the utility will detect the source networks for all selected virtual machines and display a selection field for the target network
Storage vMotion does not seem to be supported. I tried to svMotion my machines from their iSCSI based datastores to the newly created vSAN datastore but it failed.
Target Datastore: Shared datastore (same as source)
Choosing ‘Shared datastore (same as source)’ as Target Datastore fails and throws the following error:
I added the destination host and tried again but it also failed with several issues:
destination networks were not listed, only a subset were – although all were added to the distributed vSwitch
matching datastore was not found on the destination host
I could migrate to the new environment but had to select a destination datastore. This posed not much of a problem in my environment because the end goal was to get the virtual machine on the vSAN datastore.
After migrating most of the virtual machines, only two types of virtual machines were left, it felt like I could take a step back if needed. The following types were left to migrate, the vCenter VMs and the firewall VMs. The old vCenter is not needed anymore, the new vCenter and the firewall VMs are and once those are migrated I can go break down the last part of the old setup. The last host will be reset to default settings via the DCUI after which it can be added to the vSAN cluster and I can make the vSAN cluster setup complete. A tmp_vSAN_policy with no redundancy is not the way you (or me) want to run your environment, even if it is a lab environment.
I could not migrate from the old environment to the new environment while also doing a Storage vMotion, I needed to go in steps.
Nevertheless I’m happy to have used the Cross vCenter vMotion Utility. It did save me a lot of work, required little setup and configuration. I didn’t need to change anything to the setup of my old nor my new environment.
There are a couple of steps which need to be taken to configure the Tesla M60 cards with NVIDIA GRID VGPU in a vSphere / Horizon environment. I have listed them here quick and dirty. They are an extract of the NVIDIA Virtual GPU Software User Guide.
I decided to upgrade the vCenter Server Appliance from 6.5 U2 to 6.7 though it is not supported, see VMware KB. As this is not supported you will NOT want go ahead with this in a production environment. Maybe I will have regrets later on too … but this is my lab environment so the alternative is to redeploy a new VCSA.
While it was not possible at the time of writing of this post. This VMware KB has been updated with the following information.
Important: vSphere 6.5 Update 2d and higher are not supported to upgrade to vSphere 6.7 Update 1. vSphere 6.5 Update 2d and higher are supported to upgrade to vSphere 6.7 Update 2. For more information, see VMware Product Interoperability Matrices.
I have applied the following knowledge base articles on the source VCSA:
The first KB was applied because the installer is failing due to a lack of disk space on the source appliance. The installer gives the opportunity to supply a location on the source VCSA to export the necessary files that facilitate the upgrade.
The second KB was applied because the VMware Directory failed during the firstboot phase after the upgrade succeeded.
I downloaded the sources for VCSA 6.7.0 but had to go and download the sources for VCSA 6.7.0a. The VCSA 6.7.0 sources stalled at 5% on VMware Identity Management Service.
I also went to change the root password expiration to no and set the email@example.com account password to only include alphabet characters.
The installer will also fail after the first phase if the VAMI port is not reachable, the first phase will finish succesfully though. I forgot to add an exception to my firewall. You can then continue the installer by going to the VAMI interface on port 5480.