Deleting the datastore where a content library is hosted is probably not the best idea

Deleting the datastore where a content library is hosted is probably not the best idea but … yes stupid error and now what. If you are not faint of heart (and now how to take a snapshot), you can rectify this. You should contact GSS as there is not documented solution and this might break.

Take a snapshot and verify if the vCenter backups are in a healthy status. Yes? Ok go ahead.

Log on to the vCenter and create a new Content Library and name it ‘i-made-an-error’. Use the new datastore you want to use and keep the rest of the settings default as these don’t really matter.

Open an SSH session to the vCenter and connect to the Postgress DB ‘VCDB’

To show which tables are present within the database:

Show an overview of the Content Libraries added ( make sure to add the trailing ;):

Show the Content Library entries in the vCenter database

Now that we have an overview of the Content Libraries, with the one that is throwing an error highlighted.

In the following overview we find the library id from the new Content Library we just added and also the corresponding storage id.

Database Content Library storage ids

I will update the storage id from the faulty one we found on the previous screenshot with the one we found for the new Content Library.

Update the storage id for the faulty Content Library

There are a couple of places that helped me in solving this:

https://communities.vmware.com/t5/VMware-vCenter-Discussions/Content-library-item-delete-issue/td-p/2266050

https://tinkertry.com/how-to-remove-vmware-vsphere-zombie-datastore

https://vmninja.wordpress.com/2019/04/05/remove-inaccessible-datastore-from-inventory/

VCSA 7 U1 available updates error

Today I deployed a new VCSA 7 U1 and as U2 has GA’d recently I wanted to update the environment first. So I headed to the VAMI interface > Available Updates page. Immediately there was an error:

Error in method invocation ({‘id’: ‘com.vmware.appliance.update.manifest_verification_failed’, ‘default_message’: ‘Manifest verification failed’, ‘args’: []}, ‘Verification Failure\n’, ”)

I found some blogs that showed to delete upgrade status file ‘software_update_state.conf’ at /etc/applmgmt/appliance. While I tested with renaming this file to .old this did not resolve the error.

The file was recreated but held the same info, which is in JSON format and has an the following content:

“UP_TO_DATE”, it clearly is not. So I found this KB article. This is also where I got the solution for my install. I compared the url I found in the KB article with the one that is included by default in the update settings page.

The one on the KB page is the following:

https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.7.0.31000.latest/

The one that is included in the VAMI interface is the following:

https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/7.0.1.00200.latest/

In my case when I alleviated the .latest from that url, updates are detected and I can proceed.

So as you can see in the screenshot below (well not entirely but you will need to take my word for it), I selected ‘Specified’ and supplied the following url:

https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/7.0.1.00200

Update settings link

As soon I clicked ‘SAVE’, the updates became available to install.

Available updates success

boot failure: systemctl status system-fsck-root.service

I had downtime in my lab due to an power failure which resulted in a boot failure of my VCSA 6.5 appliance. Looking on the console showed me a “[FAILED] Failed to start File System Check on /dev/dis…uuid/uuid. See ‘systemctl status system-fsck-root.service’ for details.” message. Therefor it booted into ‘Emergency Shell’ or ‘Emergency mode’.

I ran the command ‘systemctl status systemd-fsck-root’ manually. This showed me that the ‘/dev/sda3’ partition was having issues.

UPDATE: It also states “RUN fsck MANUALLY”. I did not notice this the first time

I tried to run fsck with no options to see if the command was known to the CLI. I then ran the command with the partition as a parameter ‘fsck /dev/sda3’. I answered ‘y(es)’ to all ‘Fix<y>?’ questions.

In the end I received the message ‘FILE SYSTEM WAS MODIFIED’ and tried to reboot. The reboot command gave me an error so I went through the ESXi to reset the virtual machine. Afterwards I was able to login again.