It is an Ubuntu 12.04 VirtualBox VM with 5 virtual HDDs (VDI), NOTE this is just a test VM, so not well planned ahead:
ubuntu.vdifor / (/dev/mapper/ubuntu-root AKA /dev/ubuntu/root) and /home (/dev/mapper/ubuntu-home)
weblogic.vdi- /dev/sdb (mounted on /bea for weblogic and other stuff)
btrfs1.vdi- /dev/sdc (part of btrfs -m raid1 -d raid1 configuration)
btrfs2.vdi- /dev/sdd (part of btrfs -m raid1 -d raid1 configuration)
more.vdi- /dev/sde (added this virtual HDD because / ran out of inodes and it wasn't easy to figure out what to delete so as to free up inodes, so I just added the new virtual HDD, created PV, added it to existing volume group ubuntu, grew the root logical volume to work around the inode issue -_-)
Last Friday, before finishing up I wanted to free up some disk space on that box, for some reason I thought the more.vdi was useless and tried to detach it from the VM, I then clicked delete (should have clicked keep files damn!) by mistake when detaching. Unfortunately I didn't have backup for it. All too late.
What I have tried
Tried to undelete (use testdisk and photorec) the vdi files but it takes too long and recovered heaps of .vdi files that I didn't want (huge, filled the disk, damn!). I finally gave up. Fortunately most of data is on separate ext4 partition and btrfs volumes.
Out of curiosity, I still tried to mount the logical volumes and see if it is possible to at least recover the /var and /etc
I tried to use system rescue cd to boot and activate the volume groups, I got:
Couldn't find device with uuid xxxx.
Refusing activation of the partial LV root. Use --partial to override.
1 logical volume(s) in volume group "ubuntu" now active.
I was able to mount home LV but not root LV.
I am wondering if it is possible to access the root LV any more. Under the bonnet, data (on LV root - /) was striped to more.vdi (PV), I know it's almost impossible to to recover.
But I am still curious about how system administrator/DevOps guys deal with this sort of situation;-)
Thanks in advance.