LVM metadata corruption

I had the biggest PC related scare a couple of days ago. After I had two disks in my RAID5 fail in a very short amount of time and only pure luck saved my data I moved to RAID6 and I felt safer. That was, until two days ago I ran:
# pvs -v
pvs    Scanning for physical volume names
pvs  Incorrect metadata area header checksum
pvs  Incorrect metadata area header checksum
pvs  WARNING: Volume Group vg0 is not consistent
pvs  Incorrect metadata area header checksum
pvs  Incorrect metadata area header checksum
pvs  PV         VG   Fmt  Attr PSize PFree   DevSize PV UUID
pvs  /dev/md2   vg0  lvm2 a-   1.80T 922.19G   1.80T Y9naEo-OKG6-0ZyX-qmZX-u3JP-uCPg-cE1hVX


Ooops. Not looking good.

# vgs -v
vgs    Finding all volume groups
vgs  Incorrect metadata area header checksum
vgs    Finding volume group "vg0"
vgs  Incorrect metadata area header checksum
vgs  Incorrect metadata area header checksum
vgs  VG   Attr   Ext   #PV #LV #SN VSize VFree   VG UUID
vgs  vg0  wz--n- 4.00M   1  15   0 1.80T 922.19G 8uc2fo-0OwD-lPRT-1gdh-87lw-pjQW-Y1n2vd


Hmmmm, lots of errors but at least the volume group is there... now let's see the logical volumes:

# lvs -v
lvs    Finding all logical volumes
lvs  Incorrect metadata area header checksum
lvs  Incorrect metadata area header checksum
lvs  Incorrect metadata area header checksum
lvs  Volume group "vg0" inconsistent
lvs  Incorrect metadata area header checksum
lvs  Incorrect metadata area header checksum
lvs  WARNING: Inconsistent metadata found for VG vg0 - updating to use version 154
lvs  Incorrect metadata area header checksum
lvs  Automatic metadata correction failed
lvs  Internal error: Volume Group vg0 was not unlocked
lvs  Device '/dev/md2' has been left open.
lvs  Device '/dev/md2' has been left open.
lvs  Device '/dev/md2' has been left open.
lvs  Device '/dev/md2' has been left open.


I was expecting 15 logical volumes. The /home directory, some data, mail volume, etc. Thing is, they still worked.


The thing is, even if the metadata on /dev/md2 is corrupted, my kernel still had everything mounted. Since LVM had already created the device mapper devices and the kernel knew at what offsets all the volumes are. Or in simpler terms, the metadata on the disk is corrupted, but the metadata in the kernel is still alive. Therefore, first thing I did was to run 
dmsetup table
and save its output. If bad came to worse, I could still recreate all the device mapper devices using this output. Then I was able to recover the metadata from the lvm backup files, that LVM by defaults dump /etc/lvm/backup but I had set that up as a symlink to /boot/lvm-backup because it makes no sense to keep the LVM backups on an LVM volume (my / is also on an LVM).

Comments

Post a Comment

Popular posts from this blog

FreeIPA cluster with containers

Notes on how to install a FreeIPA cluster on Ubuntu. Mostly following the instructions on https://github.com/freeipa/freeipa-container Start with what would be the primary host. If you install a third replica you would want to review the replica agreements to ensure you have a two-way replication between all pairs of nodes.
Read more

ADSL Router Model CT-5367 user and pass (VIVACOM)

Writing down the username and password for the vivacom ADSL router, in case I need it again. user: root pass: warmWLspot Useful to telnet 192.168.1.1 and then reboot . Originally found at Neo2SHYAlien blog . And here I found passwords for other models, just in case. ZTE ZXDSL 832 username: root password: GSrootaccess ZTE ZXDSL 831 username: root password: GSrootaccess ZTE username: root password: 831access Huawei SmartAX MT882 username: root password: MT882rootaccess ZTE ZXDSL-531b username: root password: rootWLaccess Pirelli – DRG – A124G username: root password: warmWLspot Comtrend CT-5367 username: root password: warmWLspot HG530 Home Gateway username: root password: warmWLspot
Read more

Installing Gentoo with full disk encryption

The purpose of this exercise - install Gentoo where all disk partitions are encrypted. Boot from a system rescue disk and partition the disk - use GPT partition table where the first partition is reserved, second is the boot, and the third one is for the rest of the data. Device Start End Sectors Size Type /dev/sda1 2048 32767 30720 15M BIOS boot /dev/sda2 32768 442367 409600 200M Linux filesystem /dev/sda3 442368 375093902 374651535 178.7G Linux filesystem Encrypt partitions number 2 and 3, and open them cryptsetup luksFormat /dev/sda2 cryptsetup luksFormat /dev/sda3 cryptsetup luksOpen /dev/sda2 crypt_boot cryptsetup luksOpen /dev/sda3 crypt_data Format the boot partition mkfs.btrfs /dev/mapper/crypt_boot Prepare LVM for the rest, and format the top level vgcreate vg0 /dev/mapper/crypt_data lvcreate -l 100%FREE -n data vg0 mkfs.btrfs /dev/vg0/data Mount and set up any subvolumes mount /dev/vg0/data /mnt/gentoo btrfs subvol create /mnt/gentoo/@ /mnt/gentoo/@home umount /mnt/gentoo Now...
Read more