Cincinnerdi Tech Stuff

A mind-numbing read if ever there was one

The 2nd 95% – tracking VMware snapshot removal progress

with 7 comments

“The first 99% of the project flies by. But the 2nd 99%! Sheesh…” – anonymous

If you ever removed a snapshot in VMware ESX / ESXi, you’re presented with the ubiquitous progress meter. It chunks right along, increasing by 5% every so often. Encouraging.

And then it gets to the dreaded 95%. You’d think you’re almost home.

This would make you think you're almost done. Wrong!

But you’re probably nowhere close. Stuck.

This is really kind of dangerous. I’ve been tempted to assume that something is hung up. And that leads to thinking a hard reset of the host is required.

How CAN you see the progress? What follows is not an elegant solution, but you’ll at least be able to see what’s going on.

First, you’ll need to go to the ESXi command line (see other posts on the internet for accessing ESXi via SSH.) In this case, I used PuTTY ( http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html ) to get to the host IP and command line.

Go to the storage directory of the host, usually /vmfs/volumes, then the LUN directory and finally the VM directory.

Use the following linux command to list the files in time order, latest files last:

ls -ltr

This will show you what files have been most recently processed. Repeat this command over time (remember up-arrow to repeat bash commands) and you should notice a progression, disk files progress from lowest to highest, and within a disk, the delta files progress highest to lowes.

For example, if you have a VM called Server with 3 disks, they would be called

Server-flat.vmdk
Server_1-flat.vmdk
Server_2.flat.vmdk

And you’d see that they’d progress (latest file change time) in that order. ┬áThe delta files, created by snapshots, have 6 digit sequence numbers in their names that would progress in reverse order.

Server_1-000005-delta.vmdk
Server_1-000004-delta.vmdk
Server_1-000003-delta.vmdk
Server_1-000002-delta.vmdk
etc.

Not very exciting, true. But at least you can see some progress. I recently removed a snapshot that took 2:40 hrs. It was up to 99% in about :15 of that.

About these ads

Written by scottledyard

2010, January 19th at 9:37 pm

Posted in ESX, ESXi, Linux, Virtualization

Tagged with

7 Responses

Subscribe to comments with RSS.

  1. Experiencing this issue now. Thanks for posting this otherwise I would’ve lost patience an hour ago.

    baroniparson

    2010, November 22nd at 1:17 pm

  2. [...] "The first 99% of the project flies by. But the 2nd 99%! Sheesh…" – anonymous If you ever removed a snapshot in VMware ESX / ESXi, you're presented with the ubiquitous progress meter. It chunks right along, increasing by 5% every so often. Encouraging. And then it gets to the dreaded 99%. You'd think you're almost home. … Read More [...]

  3. Having the same bahaviour now as well … started consolidating 2 hours ago and 95% ever since ;)
    Only thing bothering me is that the last modification of the underlying base-disk was 20 mins ago …

    msmckilrogg

    2011, June 29th at 6:31 am

  4. Thanks for this post. I was sure the remowe snapshot process was hanging, but this post gave me renewed hope.

    I did something silly: I clicked “remove all snapshots” on a server that had approx 10 snapshots. The process indicator went to 95% after some minutes and then stayed there for 6 and a half hours. I dont have ssh-access, so in vSphere client I selected Configuration -> Storage -> right-click on the storage -> Browse Datastore. Then I selected the folder of the machine. Here I could follow the changes in the files, similar to what is described in this article. Without seeing the file activity I would have given up long before the 6 hours. Occasionally during the process the file activity would stop for 10 minutes or so – making me worry, and then pick up again – making me breathe again.

    The virtual disk files kept growing and growing during the process. This is a virtual machine (MS server 2003) with a Thick Provisioned disk with capasity of 40 GB. At its peek during the process the disk usage was up to 120 GB. At that point I only had 60 GB free and was getting worried as I could see it still growing. But in the end it did finish and the machine works fine.

    Lars Olav

    Lars Olav Tveito

    2012, June 1st at 10:58 am

    • Lars, you make a good point that the GUI file display in Browse Datastore also can show you that file activity is taking place.

      scottledyard

      2012, June 2nd at 6:02 pm

  5. thanks for the post. I’ll go get dinner during the dreaded 1%.

    Bob Sendelbach (@rsendelbach)

    2013, February 13th at 5:18 pm

  6. Just removed a snapshot that had a delta grown to 550 gigs. Took 30 hours but finished successfully. Good luck and be patient out there.

    perydell (@perydell)

    2014, February 4th at 2:26 am


Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 172 other followers

%d bloggers like this: