VMware I/O Analyzer is a tool to launch orchestrated tests against a storage solution available from the VMware flings website. It can be used as a single appliance where the worker process and the analytics is done within. Additional appliances can be deployed to act as Worker VMs. The Analyzer VM launches IOmeter tests (on the Worker VMs) and after test completion it collects the data. All configuration is done from a web interface on the Analyzer VM.
This post is describing how I deployed VMware I/O Analyzer and how I got to a test with maximized IOs. The first tests were conducted launching a IOmeter from within a virtual machine on the vSAN datastore and showed more or less 300 IOs being generated. In the end 18 Worker VMs with 8 disks each on a 6 host vSAN cluster were used generating 340K+ IOPS. The purpose was to create a baseline for a VSAN datastore maximum IOPs.
Hardware used
6 hosts 1 disk group 1 800GB SSD drive5 1,2 TB 10K SAS vSphere 5.5 U3
General
The VM OS disks should not be put on the vSAN datastore you want to test, if not the generated IOPs will be part of your report. To keep the Analyser VM IOPS out of the performance graphs, put it on a different datastore.
Deploy one Analyser VM. Deploy a Worker VM per ESXi host. You should end up with as much Worker VMs as you have hosts in your cluster.
I changed the IP of all VMs to static as there was no DHCP server available in the subnet. This means that no DNS entries were required.
Preferably you will want to change the Analyser VM to a static IP as you will manage the solution from a web browser. The Worker VMs you can leave as is if there is DHCP server available. You will need dns entries and change the configuration used here.
To work easily set the Worker VMs on static IPs or create dns aliases as you will be doing a lot of work on the Worker VMs. I prefer static IPs because they add no complexity due to name resolving, etc…
vi/etc/sysconfig/network/routes(The filewill be created ifit doesn’texist)
Add / Change the following line:
Shell
1
2
Default192.168.1.1--
(Defaultspace GW space hyphen space hyphen)
Save and close the file (:wq)
Restart the network service:
Shell
1
service network restart
Check if the VM is reachable.
Now shutdown the VM.
Deploying the Worker VM:
Clone the Analyser VM.
Add a Hard Disk of 1GB.
Choose advanced and put the 1GB disk on the VSAN datastore.
I needed to configure static IPs on the Worker VMs, so I had to start each VM and change the IP address. After changing the network settings, shut down the VM and create a new clone. Not changing the IPs will give duplicate IPs.
Ease of access configuration
Two ease of access configurations were applied. The first is configured for easy copying from the Analyzer VM to the Worker VMs. The second because all appliances need to be logged onto for the VMware IO Analyzer solution to work. All commands are executed on the Analyzer VM and then copied to the Worker VMs.
Setup ssh keyless authentication
Generate a key pair
Shell
1
ssh-keygen(with an empty passphrase)
ssh-copy-id will copy your public key to the target machine
Shell
1
2
3
4
5
6
ssh-copy-id-iid_rsa.pubroot@192.168.1.21
ssh-copy-id-iid_rsa.pubroot@192.168.1.22
ssh-copy-id-iid_rsa.pubroot@192.168.1.23
ssh-copy-id-iid_rsa.pubroot@192.168.1.24
ssh-copy-id-iid_rsa.pubroot@192.168.1.25
ssh-copy-id-iid_rsa.pubroot@192.168.1.26
The root account password of the destination will need to be supplied for each of the above lines.
BE AWARE: This has the following security downside. If the root account is compromised on the Analyzer vm all worker vms should be considered compromised too.
Autologon
Change autologon=”” to autologon=”root” in the displaymanager (/etc/sysconfig/displaymanager) file with the following command:
TIP: Create affinity rules in vCenter to keep the Worker VMs on dedicated hosts, otherwise the configuration on the VMware I/O Analyzer dashboard will be outdated soon. The consequence is that certain Worker VMs will not be launching their IOmeter profiles and therefor the reports will not be correct.
Configuration
Prerequisites
Enable the SSH service on the ESXi hosts via the vSphere (Web) Client or through Powershell.
The powershell way: (be aware to filter your hosts if needed). There is a dedicated post about starting and stopping ESXi services through powershell here.
I found that looking at the console of the Worker VMs is interesting for troubleshooting. You can see the IOmeter tests being launched. This was very usefull in the process of creating the IOmeter profile. You don’t need to wait untill the test is finished to see it has failed. Stopping IOmeter tests from the console gives the opportunity to look at, edit and save the launched profile.
VMware SRM 5.8.1 Embedded Database refuses to uninstall. Clicking uninstall in ‘Control Panel – Programs and Features’ showed a progress bar going forward and then rolling back. Afterwards ‘Control Panel – Programs and Features’ showed that the embedded PostgreSQL was still installed.
So I tried it through the command line with the purpose of generating a log file:
Going up in the log file, somewhere halfway there is a remark that the “C:\ProgramData\VMware\VMware vCenter Site Recovery Manager Embedded Database\data\postgresql.conf” file cannot be found.
I created the postgresql.conf file in the “C:\ProgramData\VMware\VMware vCenter Site Recovery Manager Embedded Database\data\” and tried the msiexec uninstall again. Now the uninstall succeeded.
UCS Manager is showing 309 errors because a Firmware Packages have been deleted but the references in the Host Firmware Packages (HFP) still exist.
Appears that all of the errors show a cause of ‘image-deleted’. In the ‘Affected object’ the path where the error is originating is shown. In the first error it shows ‘org-root/fw-host-pack-HFP-2.2.7/pack-image-Cisco Systems|R200-1120402W|blade-controller’ The first portion ‘org-root/fw-host-pack-HFP-2.2.7’ is important because this is the path. The second part ‘pack-image-Cisco Systems|R200-1120402W|blade-controller’ is the component image which is missing.
A HFP resides in the ‘Servers’ tab. The referenced one can be found in ‘Servers – Policies – root – Host Firmware Packages – HFP-2.2.7’
Going to the referenced ‘Host Firmware Packages’ some of the components have a presence status ‘Missing’
Below is a screenshot of the existing ‘Firmware Packages’. You can see that the ‘Firmware Package’ 2.2.6f exists for the ‘B Series’ and for the ‘Infrastructure’ but not for the ‘C Series’.
Important to notice is that ‘Rack Package’ 2.2.7b’ is not present for the ‘C Series’ as you can see in the next screenshot.
Going to the Host Firmware Package general page and looking in the assigned versions. You can see that ‘Rack Package’ 2.2(7b)C is assigned. In the above screenshot we saw that this package is not in the UCS Manager anymore.
The rack package is empty. It was on ‘Rack Package’ 2.2(7b)C but because the Firmware Package was removed from UCS Manager this is showing blank.
Use ‘Show Policy Usage’ to look if the Host Firmware Package is used somewhere.
The Host Firmware Package is used in Service Profile Template ‘HP_FW_TEST_Cisco_Support_Case’
Navigate to the Service Profile Template
Verify the Policy Usage
It is not in use, so it is safe to delete the Service Profile Template
Delete the Service Profile Template
Going back to the Host Firmware Package and looking again at the policy usage
It is not in use anymore
Delete the Host Firmware Package
The previous actions made UCS Manager go down to 193 errors. The next ones are about the Host Firmware Package ‘default’. I don’t want to delete this Host Firmware Package ‘default’, so I will adapt this one so it doesn’t throw any errors anymore.
The following screenshot is not entirely correct as the package was already changed to the correct one (‘2.2(5d)C’ was ‘2.2(6f)C’ as you will see in a later screenshot) but I still wanted to show that I checked the Policy Usage first:
The Host Firmware Package ‘default’ is not used in a Service Profile Template, so it safe to change the assigned ‘Rack Package’.
Modify the Package version from the Rack package that was deleted
Set it to one that still exists
Going back to the errors lead me to the next Host Firmware Package package, which was on a sub-organisation level. Looking at the components in the Host Firmware Package I see a presence of ‘Missing’ again
First I was going to modify the package version but I went to look if it was in use first.
It was in use by a Service Profile Template and went to see if the Service Profile Template was in use
It was not so I deleted the Service Profile Template and went back up the chain. The Host Firmware Package was not in use anymore. So I deleted the Host Firmware Package
All references to Firmware Packages were corrected