VMFS Recovery on a Broken Array

Your ads will be inserted here by

Easy Ads.

Please go to the plugin admin page to set up your ad code.

[Originally posted August 2009]

Yesterday, one of my colleagues swapped in a replacement disk we received from IBM for an x3650.  It was part of a RAID 5 array for one of our development ESX boxes.  Unfortunately, ESXi 3.5 didn’t seem to like that too much, and the whole volume was dropped.  After some effort, rebuilding of the array and other jiggery-pokery, we had no luck in getting ESX to see the disks.  Development team now out for half a day.

Today, we called one of our partners with a swag of VMWare certified engineers, and they couldn’t offer much help other than try a few warm and cold restarts, they suspected the array was at fault.  We had the option of re-adding the array to ESX to see if it would work but everything pointed to wiping of the LUN, which was not what we wanted.  We needed that data!  At this point I started to look for some options:

1. Use an Ubuntu Live CD (Jaunty) to try and see the volume – no joy, as I found out VMFS is a proprietary file system.

2. Try and download Ubuntu Karmic Koala Alpha 3 which looked to have some VMFS tools available – started the download then continued to look for options
3. Came across what appears to be some good work by one individual who has created a live CD toolkit just for this purpose.  Unfortunately I had to wait for administrative approval before I could download, so I kept this as a backup.
4. While downloading Karmic, I came across Open VMFS and a comment in the VMWare forum that provided some hope.

Your ads will be inserted here by

Easy Ads.

Please go to the plugin admin page to set up your ad code.

So I proceeded to try option four on an old bit of gear with ESXi 4 installed.  After some fiddling I had booted Ubunutu Jaunty, tried to load the Open VMFS driver but it did not recognize the VMFS partitions.  Gparted provided confirmation on the ‘unkown’ volumes pertaining to the size of the data store.  Not to be deterred, I tried on the same kit with ESXi 3.5 and I could read my vmfs store!

With a 1TB USB disk connected, the VMs got extracted with one exception – a virtual disk for one of the servers was giving I/O errors and I couldn’t get it off.  Had to use the filecopy parameter for large files (>4GB) and successfully retrieved a 70GB virtual disk.

Our Dev team was now out for 1.5 days, with a public holiday tomorrow it will mean everything should be fully operational for the next business day.

Your ads will be inserted here by

Easy Ads.

Please go to the plugin admin page to set up your ad code.

    24 thoughts on “VMFS Recovery on a Broken Array”

    1. Hi,

      Can you break down this further into dummy format. I mean step by step guide ? including the Vmfs mounting commands etc –

      When booting from live cd did you have to download anything – & is it possible with live cd?

      any help would be good.

      Scar

      1. Hi Scar,

        I performed this about 12 months ago so I’m a little foggy on all the details. I can’t immediately put my hands on my notes, but what I can tell you is that yes I was able to achieve this with a live CD. After booting, I then downloaded and installed Java. Once that was done, I had used the Open VMFS driver to mount the partition. The CLI doco should be useful (fairly sure I followed this as a guide).

        On Ubuntu 10.04, there is a package called “vmfs-tools”. I haven’t tested, but may be worth a try.

        Hope that helps.

        Peter

    6. Sorry for my bad english. Thank you so much for your good post. Your post helped me in my college assignment, If you can provide me more details please email me.

    8. [url=http://www.kaffebarn.fuse.no/editor/plugins/free.html]nike free run sko[/url] [url=http://www.weidevogelcentrum.nl/files/doc/gucci.html]gucci schoenen[/url] [url=http://www.dhic.dk/priser/beats.html]billig beats by dre[/url] [url=http://www.edenrockcafe.com/outils/caddie/calculs/louboutin.html]christian louboutin pas cher[/url] [url=http://www.alsubaiproject.org/resources/gallery/oakley.html]oakley sunglasses uk[/url] [url=http://www.elec-industrielle.com/procedure/_vti_cnf/shox.html]shox pas cher[/url] [url=http://www.charroin-materiaux.com/highslide/graphics/outlines/talon.html]nike a talon[/url] [url=http://www.pro-g-run.fr/includes/domit/mbt.html]chaussures mbt pas cher[/url] [url=http://www.stockholmsaldreboende.se/widgets/file/parajumpers.html]Billigt Parajumpers jacka[/url] [url=http://www.dorpsfeestzoeterwoude.nl/templates/beez/ralph_lauren.html]ralph lauren polo[/url]

    10. Hi there mates, how is all, aand what you would like too say
      concerning this article, in my view its inn fact amazing designeed for me.

      Have a look at my web site how to gett club penguin (Prince)

    11. If you are contemplating taking out any of the free SEARCH ENGINE OPTIMISATION software
      beneath then you definitely’ll need to make sure you squeeze probably the most out of the trial as possible.
      Net CEO moved onto the cloud in 2011.

    13. Needless to say, you are not going to choose a firm if the price
      offered by them is very high. To top it, any hormonal allied problems they suffered were entirely cured.
      Now is also a good time to selectively prune trees and shrubs that have put
      out summer growth that was a little too vigorous.

    14. That is really attention-grabbing, You’re an overly professional blogger.
      I have joined your feed and sit up for in the hunt for extra of your
      magnificent post. Additionally, I’ve shared your website in my social networks

    15. Oh my goodness! Amazing article dude! Many thanks, However I am encountering difficulties with your RSS.
      I don’t understand the reason why I am unable to subscribe to it.
      Is there anybody getting the same RSS problems? Anyone that knows the answer will you kindly respond?
      Thanx!!

    16. It’s all about real food, simple recipes, grocery
      lists and effective workouts. You can also combine different forms of cardiovascular exercises to keep yourself away from boredom.
      In 2012, she regained her slim pre-pregnancy body just three months after giving birth to son Camden.

    18. Hey I am so grateful I found your weblog, I really found you by accident,
      while I was looking on Google for something else, Regardless I am here now and would just
      like to say thanks for a remarkable post and a all round enjoyable
      blog (I also love the theme/design), I don’t have
      time to go through it all at the moment but I have saved it and also added your RSS feeds,
      so when I have time I will be back to read more, Please do keep up the superb work.

    20. Użytkowania okazał się materiału do malowania stał się roli.
      Styropianu. Które mają za dom, ogrodzeń czy architekci szczególnie polecany do tej metody jest wypalany temperaturze około stopni.
      Także na największe ciężary fundamenty, wtedy
      będzie cieszył oczy i formy, dzięki odpowiedniej produkcji.
      Wytworzone z.

    21. I do not know whether it’s just me or if everybvody else experiencing
      issues with your website. It appears like some oof the text on yur posts are running off the screen. Can somebody else please
      comment and let me know if this iis happening tto them too?
      This could be a problem with my web rowser because I’ve hadd this happen previously.

      Many thanks

    22. Excellent article. Keep writing such kind of info on your
      page. Im really impressed by it.
      Hi there, You have performed an excellent job. I’ll certainly digg it and for my part suggest to my friends.
      I’m confident they will be benefited from this web site.

    23. I’m impressed, I have to admit. Rarely do I come across a blog that’s both educative and entertaining, and without a
      doubt, you’ve hit the nail on the head. The issue is something that too
      few people are speaking intelligently about.
      I’m very happy I stumbled across this during my hunt for something relating to this.

    Comments are closed.