In a VSAN project the VMware Compatibility Guide mentioned a different driver version for the raid controller than the one that was installed. So I tried to install a driver update for the raid controller through the CLI. This did not work out as expected because the /altbootbank was in a corrupted state. There were two ways to go ahead, either reinstall from scratch or try to rebuild the /altbootbank from the /bootbank contents. This was not a production server so I had the freedom to apply a more experimental approach and therefor I chose the not supported, not recommended approach to rebuild the /altbootbank from the /bootbank contents.
I ran the following command to install the driver:
esxcli software vib install -d /vmfs/volumes/datastore/patch.zip
I got the following error message:
Failed to clear bootbank content /altbootbank: [Errno 9] Bad file descriptor: '/altbootbank/state.xxxxxxx'
Please refer to the log file for more details.
I found the following two links describing the issue:
The vmware KB is going through the steps to solve this, which in this case didn’t. The better solution is to repair or reinstall but this is a time consuming task.
The steps in the KB didn’t solve it, so I tried to delete it with:
rm –rf /altbootbank/state.5824665/
The ghost file/directory would not delete. The first command returned ‘This is not a file’, the second ‘This is not a directory’.
I repeated the same commands after a reboot with the same results. As the server was still booting well I knew the /bootbank was still ok. I wanted to replace the /altbootbank with the contents of the /bootbank partition.
Identity the naaID and partition number of the /altbootbank:
vmkfstools -Ph /altbootbank
Scratch the partition through recreating the file system:
vmkfstools -C vfat /dev/disks/naaID:partitionNumber
Remove the /altbootbank folder:
rm –rf /altbootbank
Create a symlink to the newly created vFat volume with /altbootbank:
ln –s /vmfs/volumes/volumeGUID /altbootbank
Copy all the contents from /bootbank to /altbootbank:
cp /bootbank/* /altbootbank
Change the bootstate=3 in /altbootbank/boot.cfg
Run /sbin/autobackup.sh script to update the changes