FS#37469 - [lvm2][device-mapper] Systemd timeout waiting for LV

Attached to Project: Arch Linux
Opened by David Rosenstrauch (darose) - Thursday, 24 October 2013, 15:14 GMT
Last edited by Dave Reisner (falconindy) - Saturday, 26 October 2013, 14:04 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To No-one
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

When I boot into single user mode LVM works perfectly and mounts this LV from fstab; but when I boot normally it fails, telling me that mount failed / dependency failed / lvmetad timed out waiting for device.

journalctl shows the following:

Oct 23 22:13:02 darsys12 systemd[1]: Expecting device dev-vg2-lvshare.device...
-- Subject: Unit dev-vg2-lvshare.device has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit dev-vg2-lvshare.device has begun starting up.

And later:

Oct 23 22:14:31 darsys12 systemd[1]: Job dev-vg2-lvshare.device/start timed out.
Oct 23 22:14:31 darsys12 systemd[1]: Timed out waiting for device dev-vg2-lvshare.device.
-- Subject: Unit dev-vg2-lvshare.device has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/catalog/be02cf6855d2428ba40df7e9d022f03d
--
-- Unit dev-vg2-lvshare.device has failed.
--
-- The result is timeout.
Oct 23 22:14:31 darsys12 systemd[1]: Dependency failed for /usr/local/share.
-- Subject: Unit usr-local-share.mount has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/catalog/be02cf6855d2428ba40df7e9d022f03d
--
-- Unit usr-local-share.mount has failed.
--
-- The result is dependency.
Oct 23 22:14:31 darsys12 systemd[1]: Dependency failed for Local File Systems.
-- Subject: Unit local-fs.target has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/catalog/be02cf6855d2428ba40df7e9d022f03d
--
-- Unit local-fs.target has failed.
--
-- The result is dependency.
Oct 23 22:14:31 darsys12 systemd[1]: Triggering OnFailure= dependencies of local-fs.target.
Oct 23 22:14:31 darsys12 systemd[1]: Dependency failed for File System Check on /dev/vg2/lvshare.
-- Subject: Unit systemd-fsck@dev-vg2-lvshare.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/catalog/be02cf6855d2428ba40df7e9d022f03d
--
-- Unit systemd-fsck@dev-vg2-lvshare.service has failed.
--
-- The result is dependency.


Not sure what the issue is here - some sort of race condition?
This task depends upon

Closed by  Dave Reisner (falconindy)
Saturday, 26 October 2013, 14:04 GMT
Reason for closing:  None
Additional comments about closing:  Not consistently reproducible. Closed at reporter's request.
Comment by Dave Reisner (falconindy) - Thursday, 24 October 2013, 15:25 GMT
Define "single user mode". You also have no logging that supports your claim about lvmetad timing out.

What's in /etc/mkinitcpio.conf? Do you do any LVM assembly there? I'm guessing you do, but the metadata isn't transitioned from the initramfs to the root.
Comment by David Rosenstrauch (darose) - Thursday, 24 October 2013, 15:41 GMT
Single user mode meaning booting with:

kernel /vmlinuz26 root=/dev/sda2 rw single

I'm not sure what "LVM assembly" is, but I'm assuming I'm not doing any in /etc/mkinitcpio.conf. In addition, my /etc/mkinitcpio.conf does not include the "lvm2" hook. But I was under the impression that that's ok, since the lvm2 hook is only needed if your *boot* volume is LVM, which is not the case here.
Comment by David Rosenstrauch (darose) - Thursday, 24 October 2013, 15:45 GMT
As far as "no logging that supports your claim about lvmetad timing out". Perhaps I mis-typed and it's not lvmetad that's timing out. But something is. See this msg:

Oct 23 22:14:31 darsys12 systemd[1]: Job dev-vg2-lvshare.device/start timed out.
Oct 23 22:14:31 darsys12 systemd[1]: Timed out waiting for device dev-vg2-lvshare.device.
...
Comment by Dave Reisner (falconindy) - Thursday, 24 October 2013, 16:02 GMT
Yeah, that isn't lvmetad. That's systemd waiting on udev to add symlinks for the device and that not happening (which is often caused by a wrongly built initramfs).

You're missing a lot of datapoints here... 'lsblk -f' output, /etc/fstab, and I still want your /etc/mkinitcpio.conf. Output from 'udevadm info /dev/vg2/lvshare' and 'systemctl status /dev/vg2/lvshare' would be nice as well.
Comment by David Rosenstrauch (darose) - Thursday, 24 October 2013, 17:08 GMT
[root@darsys12 ~]# lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
sda
├─sda1 ext3 a2704fb6-e751-49e7-98a0-c3fd83971abe /boot
├─sda2 xfs 09091cf5-4565-4781-a886-66af9893d535 /
├─sda3 swap e13194fc-44ec-473a-891b-dd0664178bd7 [SWAP]
└─sda4 LVM2_member jL4T5e-wA2y-1BPY-cMob-z9Ha-Hi0m-dSUcIv
├─vg1-lvabs xfs dca7c694-c231-4bcd-9aa0-5dfb0f100c08 /var/abs
└─vg1-lvcache xfs 880e0d96-688d-4a20-bb3c-9ac3de3449c2 /var/cache
sdb
└─sdb1 ext4 cd81690c-b06e-4ce8-a956-ab8faa50fbef /backup
sdc LVM2_member Ba4OAC-6hYf-5iAC-UEdZ-zlQC-3y1Z-RMRs2X
└─vg2-lvshare xfs 179941e0-b45d-42cb-a85c-c49e0ec978c5 /usr/local/share
sdd LVM2_member fL5VdF-3q1d-pOf2-IPSW-LurC-qFdJ-tHFjCb
└─vg3-lvhome xfs acba2551-7d92-4cd3-aba0-7457ff0b8502 /home
sr0


[root@darsys12 ~]# grep -v "^#" /etc/fstab
shm /dev/shm tmpfs defaults 0 0
tmpfs /tmp tmpfs size=2G 0 0

/dev/sda2 / xfs defaults 0 1
/dev/sda1 /boot ext3 defaults 0 1
/dev/sda3 swap swap defaults 0 0

/dev/vg1/lvcache /var/cache xfs defaults 0 1
/dev/vg1/lvabs /var/abs xfs defaults 0 1
/dev/vg3/lvhome /home xfs defaults 0 1
/dev/vg2/lvshare /usr/local/share xfs defaults 0 1

/dev/sdb1 /backup ext4 defaults 0 1


[root@darsys12 ~]# grep -v "^#" /etc/mkinitcpio.conf
MODULES=""

BINARIES=""

FILES=""

HOOKS="base udev autodetect modconf block filesystems keyboard fsck"


[root@darsys12 ~]# udevadm info /dev/vg2/lvshare
P: /devices/virtual/block/dm-0
N: dm-0
S: disk/by-id/dm-name-vg2-lvshare
S: disk/by-id/dm-uuid-LVM-vfOgvnrtbc79ww0JiGsnO9aLpMf2rYee8bJbWJ9JSa4Kcse3AeSVZwRaqQI1bM1r
S: disk/by-uuid/179941e0-b45d-42cb-a85c-c49e0ec978c5
S: mapper/vg2-lvshare
S: vg2/lvshare
E: DEVLINKS=/dev/disk/by-id/dm-name-vg2-lvshare /dev/disk/by-id/dm-uuid-LVM-vfOgvnrtbc79ww0JiGsnO9aLpMf2rYee8bJbWJ9JSa4Kcse3AeSVZwRaqQI1bM1r /dev/disk/by-uuid/179941e0-b45d-42cb-a85c-c49e0ec978c5 /dev/mapper/vg2-lvshare /dev/vg2/lvshare
E: DEVNAME=/dev/dm-0
E: DEVPATH=/devices/virtual/block/dm-0
E: DEVTYPE=disk
E: DM_ACTIVATION=1
E: DM_LV_NAME=lvshare
E: DM_NAME=vg2-lvshare
E: DM_SUSPENDED=0
E: DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1
E: DM_UDEV_PRIMARY_SOURCE_FLAG=1
E: DM_UDEV_RULES_VSN=2
E: DM_UUID=LVM-vfOgvnrtbc79ww0JiGsnO9aLpMf2rYee8bJbWJ9JSa4Kcse3AeSVZwRaqQI1bM1r
E: DM_VG_NAME=vg2
E: ID_FS_TYPE=xfs
E: ID_FS_USAGE=filesystem
E: ID_FS_UUID=179941e0-b45d-42cb-a85c-c49e0ec978c5
E: ID_FS_UUID_ENC=179941e0-b45d-42cb-a85c-c49e0ec978c5
E: MAJOR=254
E: MINOR=0
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: UDISKS_DM_TARGETS_COUNT=1
E: UDISKS_DM_TARGETS_LENGTH=1048576000
E: UDISKS_DM_TARGETS_PARAMS=8:32\x202048
E: UDISKS_DM_TARGETS_START=0
E: UDISKS_DM_TARGETS_TYPE=linear
E: UDISKS_PRESENTATION_NOPOLICY=1
E: USEC_INITIALIZED=21100


[root@darsys12 ~]# systemctl status /dev/vg2/lvshare
dev-vg2-lvshare.device - /dev/vg2/lvshare
Follow: unit currently follows state of sys-devices-virtual-block-dm\x2d0.device
Loaded: loaded
Active: active (plugged) since Wed 2013-10-23 22:16:03 EDT; 14h ago
Device: /sys/devices/virtual/block/dm-0

Oct 23 22:16:03 darsys12 systemd[1]: Found device /dev/vg2/lvshare.
Comment by Dave Reisner (falconindy) - Thursday, 24 October 2013, 17:12 GMT
Looks to me like you provided that output from a successful bootup...
Comment by David Rosenstrauch (darose) - Thursday, 24 October 2013, 17:16 GMT
Doh! Sorry - yes.

Will have to wait till I get home to get the output from an unsuccessful boot. (Will have to take the device in question out of fstab and reboot.)
Comment by David Rosenstrauch (darose) - Friday, 25 October 2013, 02:48 GMT
Hmmmm .... OK. Well *now* it booted without a problem.

Bit of a WTF going on here. Not sure why today it's working.

Why don't you go ahead and close the issue. I'll request a re-open if it comes back and I'm able to gather some debugging evidence.

Loading...