In a VSAN project the VMware Compatibility Guide mentioned a different driver version for the raid controller than the one that was installed. So I tried to install a driver update for the raid controller through the CLI. This did not work out as expected because the /altbootbank was in a corrupted state. There were two ways to go ahead, either reinstall from scratch or try to rebuild the /altbootbank from the /bootbank contents. This was not a production server so I had the freedom to apply a more experimental approach and therefor I chose the not supported, not recommended approach to rebuild the /altbootbank from the /bootbank contents.
I ran the following command to install the driver:
esxcli software vib install -d /vmfs/volumes/datastore/patch.zip
I got the following error message:
[InstallationError] Failed to clear bootbank content /altbootbank: [Errno 9] Bad file descriptor: '/altbootbank/state.xxxxxxx' Please refer to the log file for more details.
I found the following two links describing the issue.
The vmware KB is going through the steps to solve this, which in this case didn’t. The better solution is to repair or reinstall but this is a time consuming task.
The steps in the KB didn’t solve it, so I tried to delete it with:
rm –rf /altbootbank/state.5824665/
The ghost file/directory would not delete. The first command returned ‘This is not a file’, the second ‘This is not a directory’.
I repeated the same commands after a reboot with the same results. As the server was still booting well I knew the /bootbank was still ok. I wanted to replace the /altbootbank with the contents of the /bootbank partition.
THE FOLLOWING IS NOT RECOMMENDED NOR SUPPORTED! DO NOT EXECUTE ON A PRODUCTION ENVIRONMENT !
Identity the naaID and partition number of the /altbootbank:
vmkfstools -Ph /altbootbank
Scratch the partition through recreating the file system:
vmkfstools -C vfat /dev/disks/naaID:partitionNumber
Remove the /altbootbank folder:
rm –rf /altbootbank
Create a symlink to the newly created vFat volume with /altbootbank:
ln –s /vmfs/volumes/volumeGUID /altbootbank
Copy all the contents from /bootbank to /altbootbank:
cp /bootbank/* /altbootbank
Change the bootstate=3 in /altbootbank/boot.cfg
Run /sbin/autobackup.sh script to update the changes