VM that fails consolidation? Locked file? Here’s the fix!


Had a problem recently where I had a VM with a lot of snapshots which somehow failed to get noticed, so following a vCenter upgrade it lit up like a Christmas Tree! So here is how it was fixed with the help from a very kind man at VMware Support

So if you’ve got a warning on your vCenter (6.5 U3) against a particular VM that looks something like this:

You will no doubt just right-click on the VM > Snapshots > Consolidate and vSphere does its magic and fixes it. But what if it doesn’t work and you get something like the error below:

So you do what every good techie does and Google it right? You’ll come across all number of solutions:

  • vMotion to another host and try and consolidate
  • Shutdown the VM, take a snapshot, delete all the snapshots, power it on and consolidate
  • Wait until the moon is full, stand on your head and pray for the VMware Gods to save you.

Unbelievably, none of them work, not even that last one. I half expected Pat Gelsinger to turn up, but nope – he didn’t. So I did what I haven’t done in a long time and put a ticket in, and the next day a lovely man called Ramy from VMware called me and resolved it incredibly quickly. And this is how:

  1. SSH on to the host where the VM is running and log in as root
  2. Find where the VM files are stored
    • find -iname servername
  3. Change to that directory found in the above command
    • cd ./vmfs/volumes/527a3baa-8415c2d4-fa85-b8ca3afxxxx/servername
  4. List the contents of that directory to see what pain you have to go through
    • ls
  5. Check to see what locks are applied to the files in that directory
    • ls | while read x; do vmkfstools -D $x| grep -i "is locked"; done
    • ls | while read x; do vmfsfilelockinfo -p $x| grep -i "is locked"; done
  6. You will see lots of locks associated to a mac address on the host that the VM currently resides
    • "servername-000001-delta.vmdk" is locked in Read-Only mode by host having mac address ['00:ca:00:f8:00:f3']
    • "servername_1-000004-delta.vmdk" is locked in Read-Only mode by host having mac address ['00:ca:00:f8:00:f3']
  7. Eventually you should see an alternative mac address, different to the one listed above. In my case, locked to two mac addresses:
    • "servername_1-flat.vmdk" is locked in Read-Only mode by host having mac address ['00:00:00:00:12:f1', '00:ca:00:f8:00:f3']
  8. In your vCenter, find the host with that mac address. SSH onto that host with root
  9. Run a list open files with the name of your server.
    • lsof | grep servername
  10. At this stage, you may find, like me another server has mounted the VMDKs and that why a lock exists. This is common with backup software like Veeam. If so, you’ll see something like this
    • 71283570 vmx FILE 149 /vmfs/volumes/527a3bc7-747c85a5-958e-b8ca3afxxxx/backupserver/servername_1.vmdk-delta.REDO_0ivOja
  11. You just need to unmount those disks from the backup server in question and then you’ll be able to consolidate your VM.
  12. If its not locked by another VM, then you can release the lock by restarting the management agents.
    • /etc/init.d/hostd restart
    • /etc/init.d/vpxa restart
  13. Hopefully one of those releases your lock and you can then consolidate your VM. To check the progress, on the host that the VM currently resides run the following command to get the name of the consolidating disk process
    • vim-cmd vimsvc/task_list
  14. It will give you the name of the process, something like vim.Task:haTask-469-vim.VirtualMachine.consolidateDisks-69773984
  15. Run the following command and it should give you details of the progress of the consolidation
    • vim-cmd vimsvc/task_info haTask-469-vim.VirtualMachine.consolidateDisks-69773984

I realise that this may well be very basic diagnostics for VMware gurus, but hopefully this is handy for someone who finds it on the Internet randomly. Thanks again to Ramy from VMware Support, you’re excellent at your job!

Leave a Reply