While going through the latest lab upgrade round, I found myself running into an error when upgrading NSX. The NSX Edge Transport Nodes (ETN) upgrade successfully, however, the NSX Host Transport Nodes (HTN) portion fails.
Not that the solutions is so special but it had me running around a bit, therefore I wanted to share.
The upgrade returns the following error:
A general system error occurred: Image is not valid. Component NSX LCP Bundle(NSX LCP Bundle(4.1.0.2.0-8.0.21761693)) has unmet dependency nsx-python-greenlet-esxio because providing component(s) NSX LCP Bundle(NSX LCP Bundle(4.1.0.2.0-8.0.21761693)) are obsoleted.
At the same time the same error is listed on vCenter:
NSX LCP upgrade error on vCenter
When analysing the vLCM configuration, there was nothing that pointed to the fact that the NSX LCP Bundle was causing an issue.
To be honest, I have been complaining some over the last year, or so, about the NSX Advanced Load Balancer documentation. Mostly that it was not easy to be found, and one was having to fall back on the avinetworks.com site, which was not great either.
On docs.vmware.com the navigation links were not existing. However, if and when you knew the page titles, you could search for them through search engines. That showed that a lot of those documentation pages were there, in fact, but only not visible with non-existing links.
However, since a couple weeks, there is a banner on the avinetworks.com site that 22.1.4 is the latest release that was documented on avinetworks.com.
NSX ALB documentation deprecation on avinetworks.com
This means that the single source of truth will be on the NSX Advanced Load Balancer page on docs.vmware.com (the link does redirect you to that location 😀).
Quick tip: if you want to search within a site through a browser, e.g. chrome, use the following as an example:
When you connect to your ESXi host and you launch esxtop. You look at the esxtop output and it is not displaying as it should. Instead, it is displaying like in the below screenshot:
Your esxtop output will be displayed correctly if you are using a terminal emulator that defaults to xterm as the TERM environment variable. Some terminal emulators will use another terminal emulator value by default, eg. xterm-256color. ESXi does not map xterm-256color to one of the values it knows, so it doesn’t know how to display the output.
There is a KB article that explains how to resolve:
The value of the environment variable TERM is used by the server to control how input is recognized by the system, and what capabilities exist for output.
Let us have a look first what the TERM variable is in my case:
Shell
1
echo$TERM
I am receiving the following output:
My terminal emulator tries to connect to the endpoint (ESXi) with xterm-256color. Now let’s take a look at what values this endpoint does support:
So all of the above is possible to assign to TERM. The value my terminal emulator uses is not among the supported terminfo types. So the ESXi host cannot map to any of the known and thus does not know how to display the esxtop info correctly.
When we update the TERM environment variable to xterm and try to run esxtop again, the output will show nicely formatted.
Default
1
2
TERM=xterm
echo$TERM
Let’s check esxtop again to make sure the outcome is as expected:
Sometimes you want/need use iPerf to test the nic speed between two ESXi hosts. I did because I was seeing a NIC with low throughput in my lab.
How can we test raw speeds between the two hosts? iPerf comes to the rescue. I was looking on how to do this on an ESXi host. I doesn’t come as a surprise that I found the solution here at William Lams’ virtuallyghetto.com. Apparently iperf has been added to ESXi since 6.5 U2. You used to have to copy iperf to iperf.copy. In ESXi 7.0 that has been done for you, although you will need to look for /usr/lib/vmware/vsan/bin/iperf3.copy
ESXi host 1 (iperf server)
Disable the firewall:
Shell
1
esxcli network firewall set--enabled false
Change to the directory containing the iperf binary
Shell
1
cd/usr/lib/vmware/vsan/bin/
Execute iPerf as server
Shell
1
./iperf3.copy-s-B10.11.6.171
Overview of the used parameters:
-s
will start iperf as server
-B
defines the IP the iperf server will listen to
Disable the firewall
Shell
1
esxcli network firewall set--enabled false
ESXi host 2 (iperf client)
Change to the directory containing the iperf binary
Shell
1
cd/usr/lib/vmware/vsan/bin/
Execute iPerf as client
Shell
1
./iperf3.copy-i1-t10-c10.11.6.171-fm
Overview of the used parameters:
-i
will determine the interval of reporting back
-t
time iperf will be running
-c
client ip, will force the usage of the correct vmkernel interface
-fm
defaults to kbit/s, adding m will use mbit/s
Don’t forget to re-enable the firewall on both systems.
Whilst upgrading the home lab I also decided to rebuild from scratch. There were some challenges to overcome because I have running VMs I don’t want to shut while migrating.
My current home lab setup and the go to setup is documented here (work in progress). Basically it comes down to:
Original setup: three hosts backed with iSCSI storage for running the VMs
Temporary setup:
New vCenter with two of the three hosts configured for vSAN with connection to the iSCSI datastores
Old vCenter with one remaining host running all of the VMs
Destination setup: new vCenter with vSAN datastore
To migrate the virtual machines from the old environment (from the last remaining host to the two new hosts) I decided to take a look at the ‘Cross vCenter vMotion Utility‘. There is not a lot of documentation available at first sight but it is straightforward to set up and configure. Although I did find some things that are worth noting.
Step 1 : Running the jar
To start the Cross vCenter vMotion Utility one must run a jar file: ‘java -jar xvm-2.6.jar’.
I am running linux (Pop!_OS 18.04) as my OS. I have java version 8 and 11 installed with version 11 as default. Version 11 is not listed on the fling site as supported (Java Runtime Environment 1.8-10: See requirements). Running with version 11 (sudo java -jar xvm-2.6.jar) starts the local website on port 8080 (http://localhost:8080) but does not report back on the CLI.
Under the assumption that the java application started and failed right away, I decided to run it on my windows box which has Java Runtime environment 8 installed. The last line of feedback ‘Initialized controller with empty state’ was the same as on my linux machine. Navigating to localhost:8080 showed the Cross vCenter vMotion Utility web interface. I could now configure the application and run migrations.
It is only later when I closed the running instance on my linux box and restarting it that it showed me output on the CLI that the application started successfully.
Shell
1
2
ps-df|grep-ijava
kill-HUP9159
Output after restart:
Step 2 : Configuration
Register connections
Source vCenter
Destination vCenter
Step 3 : Migration
Add migrations
Source Site: source vCenter
Target Site: destination vCenter
Source Datacenter
Virtual Machine(s): Select one or more virtual machines
Placement Target: Cluster or Host
Target Datastore
Network Mapping(s): the utility will detect the source networks for all selected virtual machines and display a selection field for the target network
Issues
Storage vMotion?
Storage vMotion does not seem to be supported. I tried to svMotion my machines from their iSCSI based datastores to the newly created vSAN datastore but it failed.
Target Datastore: Shared datastore (same as source)
Choosing ‘Shared datastore (same as source)’ as Target Datastore fails and throws the following error:
I added the destination host and tried again but it also failed with several issues:
destination networks were not listed, only a subset were – although all were added to the distributed vSwitch
matching datastore was not found on the destination host
I could migrate to the new environment but had to select a destination datastore. This posed not much of a problem in my environment because the end goal was to get the virtual machine on the vSAN datastore.
After migrating most of the virtual machines, only two types of virtual machines were left, it felt like I could take a step back if needed. The following types were left to migrate, the vCenter VMs and the firewall VMs. The old vCenter is not needed anymore, the new vCenter and the firewall VMs are and once those are migrated I can go break down the last part of the old setup. The last host will be reset to default settings via the DCUI after which it can be added to the vSAN cluster and I can make the vSAN cluster setup complete. A tmp_vSAN_policy with no redundancy is not the way you (or me) want to run your environment, even if it is a lab environment.
Conclusion
I could not migrate from the old environment to the new environment while also doing a Storage vMotion, I needed to go in steps.
Nevertheless I’m happy to have used the Cross vCenter vMotion Utility. It did save me a lot of work, required little setup and configuration. I didn’t need to change anything to the setup of my old nor my new environment.
