Invalid Virtual Machines

Sometimes virtual machines within your VMWare environment may show up as invalid. The machine is in-fact still running at this point; but you are unable to manage the virtual machine. This can happen for a few reasons, but in my experience the most common is when the esx host is unable to access the storage.

I have seen this caused by high latency when accessing an NFS datastore, when you leave an offline datastore mounted for an extended period, and I have also seen this happen when a SAN controller failover event occurs.

This condition happens when the esx host cannot access the vmx file (vmware configuration file) in the datastore, so VMWare just sort of forgets the configuration of the machine. The process which runs the virtual machine is still running on the host, so your VM does not crash. But, you still need to recover since HA, DRS, etc… no longer work when the VM is in this state.

If this is becoming a common occurrence in your environment; you will need to do additional troubleshooting to determine why this is happening to you. But to get you out of the mess you are in right now, you need to re-register the VM. One way to re-register the VM would be to shut down the virtual machine, remove it from inventory, browse the datastore, then re-add it to inventory. But, my preferred method, which does not involve shutting down your VM is to do the following:

  1. Login to vCenter, and locate the VM in your inventory

  2. Click on VM, Select Summary, note the Host the VM is running on

  3. Enable SSH on that Host

    1. Go into the Hosts and Clusters view in vCenter

    2. Select the host

    3. Click the configuration tab

    4. Click on Security Profile

    5. Click on SSH

    6. Click Options

    7. Click Start

  4. SSH into the host and type the following *note* (If you have not created any other users, you will need to use your Root credentials to connect): vim-cmd vmsvc/getallvms | grep invalid The previous command will list all the VM’s which are currently invalid

  5. Note the four digit VM ID number for the VM you wish to repair
  6. Type the following command, replacing <VM ID> with the four digit number from the previous step: vim-cmd vmsvc/reload <VM ID>

  7. Wait 60 seconds, the VM will now re-register, and should no longer display as invalid.

If you perform the steps above, but when you get to step 4 you have trouble finding the VM ID that you need to run this against, you can take out the grep command, and just run vim-cmd vmsvc/getallvms This will list all the VM ID’s running on this ESX host. You might see the name of the VM in question, or you might see on VM ID where it shows an error while retrieving the information. When you see that, it should give you an indication that you have found the VM you are trying to fix.

One thing to watch for if you find the above methods are not working is you may have a corrupt VMX File. One way this can happen is if vCenter/ESX tries to update the VMX flle and loses/times out at an inopportune time, you can end up with a corrupted file. Typically when this happens, the VMX file ends up being zero bytes.

To check for a zero byte VMX file:

  1. Login to vCenter, and switch to the datastores view

  2. Locate the datastore with the virtual machine in question, right click select browse datastore

  3. Browse to the datastore with the virtual machine in it.

  4. Locate the file with a name ending in .vmx

  5. One of the columns is file size, if you see a zero in that column, you have a corrupted vmx file, and you will have to either restore the file from backup, or create a new VMX file.

In the event that you have a corrupted VMX File, and you need to create a new one. The easiest way to do this is build a new virtual machine, but when it prompts you if you want to create a new virtual disk; or use an existing one, select use existing virtual disk; then browse to the virtual disk for the invalid machine. When you are done, you will have a new virtual machine with the virtual hard disks from your old virtual machine. When your machine boots up off the old virtual disk; it will operate as if it were the old virtual machine….one of the magical things about virtualization.