vShield Endpoint SVM status vCenter alarm

vShield Endpoint SVM status vCenter alarm

vCenter is showing an alarm on the TrendMicro Deep Security Virtual Appliance (DSVA): ‘vShield Endpoint SVM status

Checking vShield for errors: The DSVA VA console window shows: (as to where it should show a red/grey screen)

Let’s go for some log file analysis
To get a login prompt: Alt + F2
Login with user dsva and password dsva (this is the default)
less /var/log/messages (why less is more: you get almost all the vi commands)
G to go to the last line

For some reason the ovf file is not like it is expected. The appliance is not able to set some ovf settings, in this case the network interfaces. q (to exit the log file display) sudo –s (to gain root privileges) enter the dsva user password  

test

    (to create the dsva-ovf.env file, if necessary delete the file first) reboot (to reboot the appliance, once rebooted give it 5 minutes and the alarm should clear automatically)

vCenter is showing an alarm on the TrendMicro Deep Security Virtual Appliance (DSVA): ‘vShield Endpoint SVM status Checking vShield for errors: The DSVA VA console window shows: (as to where it should show a red/grey screen) Let’s go for some log file analysis To get a login prompt: Alt + F2 Login with user dsva and password dsva (this is the default) less /var/log/messages (why less is more: you get almost all the vi commands) G to go to the last line For some reason the ovf file is not like it is expected. The appliance is not able to set some ovf settings, in this case the network interfaces. q (to exit the log file display) sudo –s (to gain root privileges) enter the dsva user password  

test

    (to create the dsva-ovf.env file, if necessary delete the file first) reboot (to reboot the appliance, once rebooted give it 5 minutes and the alarm should clear automatically)

[code language=”css”] your code here [/code]

Start or stop ESXi services using PowerCLI

Start the ssh service on all hosts:

Get-VMHost | Foreach {
   Start-VMHostService -HostService ($_ | Get-VMHostService | Where { $_.Key -eq "TSM-SSH"} )
}

Thanks to Alan Renouf at virtu-al.net, where I found this snippet: http://www.virtu-al.net/2010/11/23/enabling-esx-ssh-via-powercli/

If you want to start the ssh service on a single host, change ESXiHostName to your ESXi FQDN:

Get-VMHost -Name ESXiHostName | Foreach {
   Start-VMHostService -HostService ( $_ | Get-VMHostService | Where { $_.Key -eq "TSM-SSH" } )
}

If you want to stop the ssh service on all hosts:

Get-VMHost | Foreach {
   Stop-VMHostService -HostService ($_ | Get-VMHostService | Where { $_.Key -eq "TSM-SSH"} )
}

If you have multiple cluster in vCenter, are connected to multiple vCenters, be sure to launch the command only to the necessary hosts:

  • Get-Cluster -Name ClusterName will filter to the specified Cluster
  • Get-VMHost -Name ESXiHostName will filter to the specified ESXi
  • Get-VMHost -Server vCenterServerName will filter to the specified vCenter server
Get-Cluster -Name ClusterName | Get-VMHost -Name ESXiHostName -Server vCenterServerName | Foreach {
   Stop-VMHostService -HostService ($_ | Get-VMHostService | Where { $_.Key -eq "TSM-SSH"} )
}

These are other services I frequently use:

  • DCUI (Direct Console UI)
  • lwsmd (Active Directory Service)
  • ntpd (NTP Daemon)
  • sfcbd-watchdog (CIM Server)
  • snmpd (SNMP Server)
  • TSM (ESXi Shell)
  • TSM-SSH (SSH)
  • vmsyslogd (Syslog Server)
  • vmware-fdm (vSphere High Availability Agent)
  • vpxa (VMware vCenter Agent)
  • xorg (X.Org Server)

There are other services available but I have never used them in this context (yet):

  • lbtd (Load-Based Teaming Daemon)
  • pcscd (PC/SC Smart Card Daemon)
  • vprobed (VProbe Daemon)

Change the startup policy for a service:

  • Automatic: Start automatically if any ports are open, and stop when all ports are closed
  • On: Start and stop with host
  • Off: Start and stop manually
get-vmhost | Foreach {Set-VMHostService -HostService ($_ | Get-VMHostService | where {$_.key -eq "tsm-ssh"}) -policy On}

 

Failed to clear bootbank content /altbootbank: [Errno 9] Bad file descriptor: ‘/altbootbank/state.xxxxxxx’

In a VSAN project the VMware Compatibility Guide mentioned a different driver version for the raid controller than the one that was installed. So I tried to install a driver update for the raid controller through the CLI. This did not work out as expected because the /altbootbank was in a corrupted state. There were two ways to go ahead, either reinstall from scratch or try to rebuild the /altbootbank from the /bootbank contents. This was not a production server so I had the freedom to apply a more experimental approach and therefor I chose the not supported, not recommended approach to rebuild the /altbootbank from the /bootbank contents.

I ran the following command to install the driver:

esxcli software vib install -d /vmfs/volumes/datastore/patch.zip

I got the following error message:

[InstallationError]

Failed to clear bootbank content /altbootbank: [Errno 9] Bad file descriptor: '/altbootbank/state.xxxxxxx'

Please refer to the log file for more details.

I found the following two links describing the issue.

https://communities.vmware.com/thread/413441?start=0&tstart=0
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2033564

The vmware KB is going through the steps to solve this, which in this case didn’t. The better solution is to repair or reinstall but this is a time consuming task.

The steps in the KB didn’t solve it, so I tried to delete it with:

rm /altbootbank/state.5824665/
rm –rf /altbootbank/state.5824665/

The ghost file/directory would not delete. The first command returned ‘This is not a file’, the second ‘This is not a directory’.
I repeated the same commands after a reboot with the same results. As the server was still booting well I knew the /bootbank was still ok. I wanted to replace the /altbootbank with the contents of the /bootbank partition.

THE FOLLOWING IS NOT RECOMMENDED NOR SUPPORTED! DO NOT EXECUTE ON A PRODUCTION ENVIRONMENT !

Identity the naaID and partition number of the /altbootbank:

vmkfstools -Ph /altbootbank

Scratch the partition through recreating the file system:

vmkfstools -C vfat /dev/disks/naaID:partitionNumber

Remove the /altbootbank folder:

rm –rf /altbootbank

Create a symlink to the newly created vFat volume with /altbootbank:

ln –s /vmfs/volumes/volumeGUID /altbootbank

Copy all the contents from /bootbank to /altbootbank:

cp /bootbank/* /altbootbank

Change the bootstate=3 in /altbootbank/boot.cfg

vi /altbootbank/boot.cfg

Run /sbin/autobackup.sh script to update the changes

/sbin/autobackup.sh

 

Reconfigure diagnostic partition

Reconfigure diagnostic partition with PowerCLI using Get-EsxCli

The following Get-EsxCli command will unconfigure your diagnostic partition and reconfigure with smart selection. This was needed because the install partition uuid had changed due to an option in the NetApp system while doing system testing.

$server_list = Get-VMhost

Foreach ($srv in $server_list)
{
 $esxcli = Get-EsxCli -VMhost $srv
 #$esxcli.system.coredump.file.add($null,"VMFS_log_partition","$srv.name",$null)
 $esxcli.system.coredump.partition.set($null,$null,$null,$true)
 $esxcli.system.coredump.partition.set($true,$null,$true,$null)
 $esxcli.system.coredump.partition.get()
}

Many thanks to http://www.virten.net/2014/02/howto-use-esxcli-in-powercli/

PowerCLI goodies

PowerCLI goodies

These are powercli goodies I use on a regular base. I have collected them here to find them easily. Some I wrote myself, some are copied from other sites. If I didn’t reference the source, I don’t know anymore where I found it.

Change portgroup on a lot of vm’s

Testing with one vm

get-vm my_vm_name | Get-NetworkAdapter |where {$_.networkname -eq "current_network_label"} | set-networkadapter -portgroup "new_network_label" -confirm:$false

Reconfigure all vm’s with the string “vdi” in the name

get-vm *vdi* | Get-NetworkAdapter |where {$_.networkname -eq "current_network_label"} | set-networkadapter -portgroup "new_network_label" -confirm:$false

Find all vm’s with more than 8 CPUs

(get-view -viewtype virtualmachine).summary.config | where {$_.numcpu -gt 8}

Reload Syslog via esxcli

While executing the NetApp MetroCluster testplan, the syslog service stops logging to the presented syslog datastore. To restart the logging reload the syslog service on all impacted hosts. The following command will reload the syslog service on all hosts in the connected vCenters. Check the $global:defaultviservers to know which vCenters are connected.

$server_list = Get-VMhost

Foreach ($srv in $server_list)
{
   $esxcli = Get-EsxCli -VMhost $srv
   $esxcli.system.syslog.reload()
}

Speed-up the initialization of PowerCLI

This needs to be done for each registered version of PowerCLI. This one worked for me on Windows Server 2012 R2

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\ngen.exe install “VimService55.XmlSerializers, Version=5.5.0.0, Culture=neutral, PublicKeyToken=10980b081e887e9f” /ExeConfig:c:\windows\system32\WindowsPowerShell\v1.0\PowerShell_ISE.exe

Change the Power Management Policy for each host in cluster

#get connected esxi hosts
$vmhostlist = get-vmhost |sort|select name

#for each host show the power management policy setting
foreach ($entry in $vmhostlist) {
   #list power management policy on all connected esxi hosts
   get-vmhost |sort|select name, @{ N="CurrentPolicy"; E={$_.ExtensionData.config.PowerSystemInfo.CurrentPolicy.ShortName}},@{N="CurrenPolicyKey"; E={$_.ExtensionData.config.PowerSystemInfo.CurrentPolicy.Key}},@{N="AvailablePolicies";E={$_.ExtensionData.config.PowerSystemCapability.AvailablePolicy.ShortName}
}

#for each host change the power management policy to high performance
foreach ($entry in $vmhostlist) {
   $view=(get-vmhost $entry.name|get-view);(get-view $view.configmanager.powersystem).configurepowerpolicy(1)}
   #list power management policy on all connected esxi hosts
   get-vmhost |sort|select name, @{ N="CurrentPolicy"; E={$_.ExtensionData.config.PowerSystemInfo.CurrentPolicy.ShortName}},@{N="CurrenPolicyKey"; E={$_.ExtensionData.config.PowerSystemInfo.CurrentPolicy.Key}},@{N="AvailablePolicies";E={$_.ExtensionData.config.PowerSystemCapability.AvailablePolicy.ShortName}
}

Update:

Apparently there were some errors (curly brackets missing or in the wrong place) in the previous code.

It also ran several times per host because per host there was a Get-VMHost in the ForEach iteration. So If you had three hosts it would run three times per host.

The updated and optimized code:

$VMHosts = Get-VMHost

#for each host show the power management policy setting
ForEach ($entry in $VMHosts) {
   #list power management policy on all connected esxi hosts
   $entry | Select Name, @{ N="CurrentPolicy"; E={$_.ExtensionData.Config.PowerSystemInfo.CurrentPolicy.ShortName}},@{N="CurrenPolicyKey"; E={$_.ExtensionData.Config.PowerSystemInfo.CurrentPolicy.Key}},@{N="AvailablePolicies";E={$_.ExtensionData.Config.PowerSystemCapability.AvailablePolicy.ShortName}}
}

#for each host change the power management policy to high performance
ForEach ($entry in $VMHosts) {
   $view=($entry | Get-View);(Get-View $view.ConfigManager.PowerSystem).ConfigurePowerPolicy(1)
   #list power management policy on all connected esxi hosts
   $entry | Select Name, @{ N="CurrentPolicy"; E={$_.ExtensionData.Config.PowerSystemInfo.CurrentPolicy.ShortName}},@{N="CurrenPolicyKey"; E={$_.ExtensionData.Config.PowerSystemInfo.CurrentPolicy.Key}},@{N="AvailablePolicies";E={$_.ExtensionData.Config.PowerSystemCapability.AvailablePolicy.ShortName}}
}

show connected vcenters/esxi’s

$Global:DefaultVIServers

connect to previously connected vcenters/esxi’s

connect-viserver -menu

move vm storage vmotion

get-vm Win* | move-vm -Datastore (Get-Datastore sdc-t*)

Mounting and unmounting NFS datastores

Mount NFS datastore

get-vmhost | New-Datastore -Nfs -Name Datastore_Name -Path /vol/Name_on_NFS_host -NfsHost NFS_host_IP

Remove NFS datastore

get-vmhost | Remove-Datastore -datastore Datastore_Name

Change the default ‘ESX Admins’ AD group to your group name

ESXi servers will by default search for the group ‘ESX Admins’ in Active Directory. The following command will change this to ‘my_group_name’.

get-vmhost | Get-AdvancedSetting "Config.HostAgent.plugins.hostsvc.esxAdminsGroup" | Set-AdvancedSetting -value "my_group_name" -Confirm:$false