Failed to clear bootbank content /altbootbank: [Errno 9] Bad file descriptor: ‘/altbootbank/state.xxxxxxx’

In a VSAN project the VMware Compatibility Guide mentioned a different driver version for the raid controller than the one that was installed. So I tried to install a driver update for the raid controller through the CLI. This did not work out as expected because the /altbootbank was in a corrupted state. There were two ways to go ahead, either reinstall from scratch or try to rebuild the /altbootbank from the /bootbank contents. This was not a production server so I had the freedom to apply a more experimental approach and therefor I chose the not supported, not recommended approach to rebuild the /altbootbank from the /bootbank contents.

I ran the following command to install the driver:

I got the following error message:

I found the following two links describing the issue:

https://communities.vmware.com/thread/413441?start=0&tstart=0

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2033564

The vmware KB is going through the steps to solve this, which in this case didn’t. The better solution is to repair or reinstall but this is a time consuming task.

The steps in the KB didn’t solve it, so I tried to delete it with:

The ghost file/directory would not delete. The first command returned ‘This is not a file’, the second ‘This is not a directory’.
I repeated the same commands after a reboot with the same results. As the server was still booting well I knew the /bootbank was still ok. I wanted to replace the /altbootbank with the contents of the /bootbank partition.

THE FOLLOWING IS NOT RECOMMENDED NOR SUPPORTED! DO NOT EXECUTE ON A PRODUCTION ENVIRONMENT !

Identity the naaID and partition number of the /altbootbank:

Scratch the partition through recreating the file system:

Remove the /altbootbank folder:

Create a symlink to the newly created vFat volume with /altbootbank:

Copy all the contents from /bootbank to /altbootbank:

Change the bootstate=3 in /altbootbank/boot.cfg

Run /sbin/autobackup.sh script to update the changes

Reconfigure diagnostic partition

Reconfigure diagnostic partition with PowerCLI using Get-EsxCli

The following Get-EsxCli command will unconfigure your diagnostic partition and reconfigure with smart selection. This was needed because the install partition uuid had changed due to an option in the NetApp system while doing system testing.

Many thanks to http://www.virten.net/2014/02/howto-use-esxcli-in-powercli/

PowerCLI goodies

These are powercli goodies I use on a regular base. I have collected them here to find them easily. Some I wrote myself, some are copied from other sites. If I didn’t reference the source, I don’t know anymore where I found it.

Change portgroup on a lot of vm’s

Testing with one vm

Reconfigure all vm’s with the string “vdi” in the name

Find all vm’s with more than 8 CPUs

Reload Syslog via esxcli

While executing the NetApp MetroCluster testplan, the syslog service stops logging to the presented syslog datastore. To restart the logging reload the syslog service on all impacted hosts. The following command will reload the syslog service on all hosts in the connected vCenters. Check the $global:defaultviservers to know which vCenters are connected.

Speed-up the initialization of PowerCLI

This needs to be done for each registered version of PowerCLI. This one worked for me on Windows Server 2012 R2

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\ngen.exe install “VimService55.XmlSerializers, Version=5.5.0.0, Culture=neutral, PublicKeyToken=10980b081e887e9f” /ExeConfig:c:\windows\system32\WindowsPowerShell\v1.0\PowerShell_ISE.exe

Change the Power Management Policy for each host in cluster

Update:

Apparently there were some errors (curly brackets missing or in the wrong place) in the previous code.

It also ran several times per host because per host there was a Get-VMHost in the ForEach iteration. So If you had three hosts it would run three times per host.

The updated and optimized code:

show connected vcenters/esxi’s

connect to previously connected vcenters/esxi’s

move vm storage vmotion

Mounting and unmounting NFS datastores

Mount NFS datastore

Remove NFS datastore

Change the default ‘ESX Admins’ AD group to your group name

ESXi servers will by default search for the group ‘ESX Admins’ in Active Directory. The following command will change this to ‘my_group_name’.