FS#33851 - [lvm2][device-mapper] upgrade renders system unbootable

Attached to Project: Arch Linux
Opened by Xyne (Xyne) - Thursday, 14 February 2013, 02:14 GMT
Last edited by Allan McRae (Allan) - Thursday, 02 January 2014, 05:09 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Thomas Bächler (brain0)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 13
Private No


There is a forum thread about this: https://bbs.archlinux.org/viewtopic.php?id=158019

Upgrading while following all of the points in the news item and then regenerating the ramdisk (mkinitcpio -p linux) leads to the next boot failing with "end_request I/O error dev fd0 sector 0".

The system can be mounted via chroot from a recovery system without issue. No disk errors are reported by fsck.

This occurs for various kernel line parameter combinations as shown in the forum thread.

Downgrading the packages via the recovers system or from within the chroot makes the system bootable again.

Let me know what other information you need or what I should test.
This task depends upon

Closed by  Allan McRae (Allan)
Thursday, 02 January 2014, 05:09 GMT
Reason for closing:  Fixed
Additional comments about closing:  Reported fixed by user
Comment by Xyne (Xyne) - Thursday, 14 February 2013, 02:21 GMT
Sorry, forgot to share hooks and modules:
HOOKS="base udev autodetect block lvm2 filesystems keymap keyboard fsck shutdown"

Also, partition setup is
sda1: primary, boot
sda2: primary, LVM
root, var and home on LVM, unencrypted
Comment by Christoper Daley (luciferin) - Thursday, 14 February 2013, 05:15 GMT
For me this was caused by my BIOS having a floppy drive enabled when there was no floppy drive present. LVM2 seems to time out trying to detect an LVM on the nonexistent drive and fails to continue.
Comment by Wiebe Cazemier (halfgaar) - Friday, 15 February 2013, 19:24 GMT
I just upgraded my system and I have the same problem. I do have a floppy. It errors about it, but it also says:

device-mapper: table: 253:0: linear: dm-linear: device lookup failed

Then it drops me to a recovery shell. My root fs (/dev/mapper/lvmopraid-root) has no device entry. When I type

lvm lvchange -a e lvmopraid # lvmopraid being my volume group

the device entries appear, and I can press ctrl-d to logout, and the system boots.

My setup is:

/dev/sda2 and /dev/sdb2 contain an MD (/dev/md1), which contains the VG lvmopraid. That VG has three LV's, root, home and swap.

LVM: core/lvm2 2.02.98-3
Uname -a: Linux melk 3.7.7-1-ARCH #1 SMP PREEMPT Mon Feb 11 20:20:58 EET 2013 x86_64 GNU/Linux
Comment by Michael Oczak (dsoul) - Saturday, 16 February 2013, 16:41 GMT
I had problems with booting with root on lvm too after arch change to lvmetad. I seems that this due to poor quality of lvmetad in lvm 2.02.98, after building lvm grom git/master everything works ok.
Comment by Wiebe Cazemier (halfgaar) - Sunday, 17 February 2013, 09:48 GMT
My system now boots properly again. I haven't upgraded any packages. The only thing I did, was change /etc/mdadm.conf and renamed the devices to '/dev/md1' and '/dev/md0' (they were /dev/md/1 and /dev/md/0). Everytime I booted, mdadm notified me of a change in MD's. I've had that problem before, so I changed the mdadm.conf and did 'mkinitcpiofs -p linux'. I didn't think it would influence booting, but apparently, it did.
Comment by Markus Holtermann (MarkusH) - Monday, 18 February 2013, 17:42 GMT
@Wiebe Cazemier (halfgaar): Nice, thanks. That works for me as well. Replace /dev/md/0 with /dev/md0
Comment by Bartosz Kwitniewski (Zerg2000) - Friday, 22 February 2013, 17:23 GMT
My machine also didn't boot after last lvm update. I have boot partition on sda1 and root on LVM, no software raid.

HOOKS="base udev autodetect modconf block lvm2 filesystems keyboard fsck"

lvm.conf was updated automatically by pacman after update (use_lvmetad = 1).

After booting from rescuecd and downgrading the following packages, machine booted again:
lvm2 (2.02.98-3 -> 2.02.98-1)
device-mapper (2.02.98-3 -> 2.02.98-1)
mkinitcpio (0.13.0-1 -> 0.12.0-3)
Comment by Jelle van der Waa (jelly) - Friday, 22 February 2013, 19:31 GMT
Did you edit your /etc/lvm/lvm.conf according to the news? And rebuild your initrd with mkinitcpio -p linux in a chroot?
Comment by Bartosz Kwitniewski (Zerg2000) - Friday, 22 February 2013, 20:41 GMT
I have updated packages again and it works this time.

At first update I did "mkinitcpio -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img" instead of "mkinitcpio -p linux" after seeing that /sbin/dmsetup is missing, but I doubt it has something to do with it because linux.preset has the same options. Perhaps I missed something else, sorry.
Comment by David Kowis (BeepDog) - Sunday, 31 March 2013, 04:04 GMT
So I got bit by this as well.

I didn't have to edit my /etc/lvm/lvm.conf, since I hadn't modified it. I verified it according to the news article. I resolved the missing /sbin/dmsetup.

when I boot, during the mdadm hook, I get a lot of errors:

device-mapper: table: 253:4: linear: dm-linear: Device lookup failed

if I run lvm vgchange -ay , it will activate the groups, and the system continues to boot.
Comment by David Kowis (BeepDog) - Sunday, 31 March 2013, 04:20 GMT
So, a resolution for me, from the forums listed is the global_filter fix:

global_filter = [ "r|^/dev/fd[0-9]$|", "r|^/dev/sr[0-9]$|" ]

Setting that in /etc/lvm/lvm.conf and everything works smoothly again. I don't see the "device-mapper: table: 253:4: linear: dm-linear: Device lookup failed" error messages any longer either.
Comment by David Kowis (BeepDog) - Sunday, 07 April 2013, 21:57 GMT
I'll also note, that not having the global_filter makes vgs and lvs take a very long time to complete.

This is on a box that has a mdadm raid10, and two VGs and like 70something logical volumes.

With the global filter, it returns very, very fast.
Comment by Gerardo Exequiel Pozzi (djgera) - Monday, 06 May 2013, 18:53 GMT
Comment by Curtis Shimamoto (WonderWoofy) - Friday, 26 July 2013, 19:50 GMT
This was already mentioned by dsoul above apparently, but I too was able to find a fix in having the lvm2 and device-mapper packages build from the git sources.
Comment by Doug Newgard (Scimmia) - Sunday, 29 September 2013, 20:05 GMT
both device-mapper and lvm2 in [core] have been updated since this report, is it still an issue? How about with the newer update in [testing]?
Comment by Wiebe Cazemier (halfgaar) - Tuesday, 01 October 2013, 16:04 GMT
I fixed it by renaming my md devices in mdadm.conf

Then, at some point, it broke again. I had to run 'mdadm --assemble --scan; exit' in a recovery prompt all the time. This went unfixed, until I found out that I had to replace the hook mdadm with mdadm_udev in mkinitcpio.conf.

So, my system works now, but there should be been better detection/warning on the upgrades to notify the user.
Comment by Curtis Shimamoto (WonderWoofy) - Wednesday, 09 October 2013, 17:11 GMT
I have a file server with only LVM2 (no mdadm, encryption, etc.) and for the last few updates after lvm2 and device-mapper 2.02.98-3, things seem to be fixed.