FS#31236 - [systemd] and dmraid
Attached to Project:
Arch Linux
Opened by Fred Verschueren (fvsc) - Wednesday, 22 August 2012, 08:17 GMT
Last edited by Dave Reisner (falconindy) - Sunday, 30 December 2012, 14:28 GMT
Opened by Fred Verschueren (fvsc) - Wednesday, 22 August 2012, 08:17 GMT
Last edited by Dave Reisner (falconindy) - Sunday, 30 December 2012, 14:28 GMT
|
Details
Description:
I' m using Arch now for several years on a desktop and a laptob without severe problems. Few days ago I changed to systemd both on the laptop and desktop. For the laptop all went fine and I' m able to boot using systemd and the system is up and running as before. The only difference between laptop and desktop is that the /home partition on the desktop is on a FakeRaid (bios raid) My problem: When booting with init=/bin/systemd I have the following phenomenon: Job dev-mapper-nvidia_cjjcaiiep1.device/start timed out and booting stops and I'm in a rescue shell. my /home partition is not mounted. /dev/mapper/nvidia_cjjcaiiep1 exists. Systemctl start dev-mapper-nvidia_cjjcaiiep1.device gives the same result timeout. and my /home is not mounted output of dmraid -t /dev/sdc: nvidia, "nvidia_cjjcaiie", mirror, ok, 398297086 sectors, data@ 0 /dev/sdb: nvidia, "nvidia_cjjcaiie", mirror, ok, 398297086 sectors, data@ 0 Output dmsetup status nvidia_cjjcaiie: 0 398297086 mirror 2 8:16 8:32 3039/3039 1 AA 1 core VolGroup00-lvolswap: 0 8388608 linear VolGroup00-lvoltmp: 0 8388608 linear VolGroup00-lvolroot: 0 31457280 linear VolGroup00-lvolroot: 31457280 10485760 linear VolGroup00-lvolweb: 0 41943040 linear VolGroup00-lvolvar: 0 31457280 linear nvidia_cjjcaiiep1: 0 398292992 linear VolGroup00-lvolvirt: 0 83886080 linear output dmsetup deps nvidia_cjjcaiie: 2 dependencies : (sdc) (sdb) VolGroup00-lvolswap: 1 dependencies : (sda8) VolGroup00-lvoltmp: 1 dependencies : (sda8) VolGroup00-lvolroot: 1 dependencies : (sda8) VolGroup00-lvolweb: 1 dependencies : (sda8) VolGroup00-lvolvar: 1 dependencies : (sda8) nvidia_cjjcaiiep1: 1 dependencies : (nvidia_cjjcaiie) VolGroup00-lvolvirt: 1 dependencies : (sda8) output ls -al /dev/mapper drwxr-xr-x 2 root root 220 Aug 22 10:01 . drwxr-xr-x 19 root root 3.8K Aug 22 10:03 .. lrwxrwxrwx 1 root root 7 Aug 22 10:01 VolGroup00-lvolroot -> ../dm-0 lrwxrwxrwx 1 root root 7 Aug 22 10:01 VolGroup00-lvolswap -> ../dm-4 lrwxrwxrwx 1 root root 7 Aug 22 10:01 VolGroup00-lvoltmp -> ../dm-2 lrwxrwxrwx 1 root root 7 Aug 22 10:01 VolGroup00-lvolvar -> ../dm-1 lrwxrwxrwx 1 root root 7 Aug 22 10:01 VolGroup00-lvolvirt -> ../dm-3 lrwxrwxrwx 1 root root 7 Aug 22 10:01 VolGroup00-lvolweb -> ../dm-5 crw------- 1 root root 10, 236 Aug 22 10:01 control brw------- 1 root root 254, 6 Aug 22 09:58 nvidia_cjjcaiie brw------- 1 root root 254, 7 Aug 22 09:58 nvidia_cjjcaiiep1 When I do a manual mount of the /home partition mount /dev/mapper/nvidia_cjjcaiiep1 /home /home is mounted Systemctl default my system resumes booting and finally comes online. I 'm new to systemd so if someone can be so kind to give me some hints how / where to start searching for the problem or maybe the problem is already known or it is a bug ... Thanks in advance. Fred Additional info: * package version(s) libsystemd 188-2 systemd 188-2 systemd-arch-units 20120704-2 systemd-tools 188-2 dmraid 1.0.0.rc16.3-7 * config and/or log files etc. Please let me know what you need Steps to reproduce: ?home partition on a bias raid (mirror) |
This task depends upon
Closed by Dave Reisner (falconindy)
Sunday, 30 December 2012, 14:28 GMT
Reason for closing: Fixed
Additional comments about closing: dmraid 1.0.0.rc16.3-8
Sunday, 30 December 2012, 14:28 GMT
Reason for closing: Fixed
Additional comments about closing: dmraid 1.0.0.rc16.3-8
FS#30134. I don't have anything more to add to that right now, and I have no ability to test this in any way.1. This is booting without any problem
2. my /home partition is mounted and fully accessable
3. Even dmraid is used
dmraid -r
/dev/sdc: nvidia, "nvidia_cjjcaiie", mirror, ok, 398297086 sectors, data@ 0
/dev/sdb: nvidia, "nvidia_cjjcaiie", mirror, ok, 398297086 sectors, data@ 0
dmsetup status
luks-9391e85a-6420-4074-b78e-2ba420efd864: 0 567976624 crypt
nvidia_cjjcaiie: 0 398297086 mirror 2 8:16 8:32 3039/3039 1 AA 1 core
VolGroup00-lvolswap: 0 8388608 linear
VolGroup00-lvoltmp: 0 8388608 linear
VolGroup00-lvolroot: 0 31457280 linear
VolGroup00-lvolroot: 31457280 10485760 linear
VolGroup00-lvolweb: 0 41943040 linear
VolGroup00-lvolvar: 0 31457280 linear
vgroup-lvol_bu_pc: 0 567967744 linear
nvidia_cjjcaiiep1: 0 398292992 linear
VolGroup00-lvolvirt: 0 83886080 linear
dmsetup deps
luks-9391e85a-6420-4074-b78e-2ba420efd864: 1 dependencies : (8, 50)
nvidia_cjjcaiie: 2 dependencies : (8, 32) (8, 16)
VolGroup00-lvolswap: 1 dependencies : (8, 8)
VolGroup00-lvoltmp: 1 dependencies : (8, 8)
VolGroup00-lvolroot: 1 dependencies : (8, 8)
VolGroup00-lvolweb: 1 dependencies : (8, 8)
VolGroup00-lvolvar: 1 dependencies : (8, 8)
vgroup-lvol_bu_pc: 1 dependencies : (253, 8)
nvidia_cjjcaiiep1: 1 dependencies : (253, 0)
VolGroup00-lvolvirt: 1 dependencies : (8, 8)
ls -al /dev/mapper
total 0
drwxr-xr-x. 2 root root 260 Aug 24 10:16 .
drwxr-xr-x. 21 root root 3880 Aug 24 08:16 ..
crw-------. 1 root root 10, 236 Aug 24 08:16 control
lrwxrwxrwx. 1 root root 7 Aug 24 08:16 luks-9391e85a-6420-4074-b78e-2ba420efd864 -> ../dm-8
brw-rw----. 1 root disk 253, 0 Aug 24 10:16 nvidia_cjjcaiie
lrwxrwxrwx. 1 root root 7 Aug 24 08:16 nvidia_cjjcaiiep1 -> ../dm-1
lrwxrwxrwx. 1 root root 7 Aug 24 08:16 vgroup-lvol_bu_pc -> ../dm-9
lrwxrwxrwx. 1 root root 7 Aug 24 08:16 VolGroup00-lvolroot -> ../dm-2
lrwxrwxrwx. 1 root root 7 Aug 24 08:16 VolGroup00-lvolswap -> ../dm-6
lrwxrwxrwx. 1 root root 7 Aug 24 08:16 VolGroup00-lvoltmp -> ../dm-4
lrwxrwxrwx. 1 root root 7 Aug 24 08:16 VolGroup00-lvolvar -> ../dm-3
lrwxrwxrwx. 1 root root 7 Aug 24 08:16 VolGroup00-lvolvirt -> ../dm-5
lrwxrwxrwx. 1 root root 7 Aug 24 08:16 VolGroup00-lvolweb -> ../dm-7
4. mdadm arrays are empty
cat /proc/mdstat
Personalities :
unused devices: <none>
I want to point out that this fedora configuration is rather complex:
The / is on an encrypted external firewire disk
The /boot is on the first internal disk
The /home is on a FakeRaid (bios raid) with two (sdb , sdc) disks
cat /etc/fstab
#
# /etc/fstab
/dev/mapper/vgroup-lvol_bu_pc / ext4 defaults 1 1
UUID=7bae55c2-a689-4829-8077-181afd23e0cd /boot ext3 defaults 1 2
UUID=76c4b013-fd10-4d39-ba65-e18c97a356fa swap swap defaults 0 0
/dev/mapper/nvidia_cjjcaiiep1
In my humble opinion this seems to indicate that the problem is Arch specific and as nothing to with upstream dmraid nor systemd.
Please let me know if you need some more info
Could anyone else verify whether or not you have "TAGS=:systemd:" on your dmraid device under Arch?
FS#30134Dracut.conf from Fedora
Udevadm from Fedora
udevadm from arch when in emergency shell (udevadm-arch-before)
udevadm from arch after manual mount /dev/mapper/nvidiajjcaiiep1 /home (udevadm-arch-after)
My /root is NOT on a dmraid nor mdraid only /home is on a dmraid
So, the issue is there by using either dmraid or mdadm and it doesn't depend on whether the array was assembled in early userspace or not. Still, the systems work in fedora fc7 (using systemd and dmraid) and fedora 17 (using systemd and mdadm). Any idea?
This works for sysvinit because we don't care about those symlinks existing. We simply run 'mount -a' after some synchronization point (when we believe all storage devices are available) and let mount resolve the tags through libblkid.
IMPORT{builtin}="blkid"
ENV{DM_UDEV_LOW_PRIORITY_FLAG}=="1", OPTIONS="link_priority=-100"
ENV{ID_FS_USAGE}=="filesystem|other|crypto", ENV{ID_FS_UUID_ENC}=="?*", SYMLINK+="disk/by-uuid/$env{ID_FS_UUID_ENC}"
ENV{ID_FS_USAGE}=="filesystem|other", ENV{ID_FS_LABEL_ENC}=="?*", SYMLINK+="disk/by-label/$env{ID_FS_LABEL_ENC}"
with dmraid (I'm not using mdadm)
1. after succesful boot with sysvinit
2. After failed boot with systemd (in emergency shell)
3. After succesful boot with systemd (after mount of /home and systemctl default in emergency shell)
Hope this helps
(the one when I booted with systemd is the same before or after I mounted my home partition by hand .
/ -> /dev/md126p5
swap -> /dev/md126p5
/home -> /dev/md126p7
sytemctl show dev-mapper-nvidia_jjcaiie.mount (dmraid disk)
sytemctl show dev-mapper-nvidia_jjcaiiep1.mount (dmraid disk partition)
From each show command I sent two fies: one when the mount of /home was failed and one after a manual mount of /home (mount /dev/mapper/nvidia_jjcaiiep1 /home)
Diff show no difference for the dev-mapper-nvidia ... files (before and after)
The difference for the home file is showing the mount operation.
By the way, is there somewhere a document describing the the flow of the boot process with systemd.
This would help to pin point a fault.
By the way, is there somewhere a document describing the the flow of the boot process with systemd.
This would help to pin point a fault.
I can only boot after manual mount. (see above)
Intel "RAID" controller (fake type), 2 SSD RAID-0. (Asus P8Z77V Premium)
If booting off of it, or another install on a different single disk, it will see all partitions listed in the fstab on the Intel RAID as "DEAD".
If running LUKS on another parition on the same disk, it will see the LUKS partition fine after it is unlocked. (preboot or postboot does not matter)
If a systemctl daemon-reload is performed, it will then see it as healthy and mount properly.
When changing targets, it will see it as dead again and require another daemon-reload. (I think this eliminates mkinitcpio from equation)
If it is set as "noauto" in fstab, it will work properly when accessed after a daemon-reload after boot in final target.
If "noauto" is not used in fstab, a daemon-reload is used, and then accessing data on the partition for more than a few mins, it will do an fsck and kind of lock the system, basically systemd crashes. I can issue commands, but no systemd, no init, so no logs, and no reboot.
I am using UUID, but path or label work exactly the same.
Doing a ls of /dev shoes the partitions, as well as the UUID and Label in the disk-by, before and after daemon-reload.
Blkid lists all of them before and after daemon-reload.
Udevadm lists disk and partition before and after daemon-reload.
I would give up hope for a solution, but doing a daemon-reload after finished loading a target, and mounting afterwards, it works fine. This indicates it is capable of working, just got a problem with load order, or something tirvial someone with a more in-depth knowledge of systemd and udev can fix.
Systemd is the way things are going. I think it is an improvement, and am hopeful we can work past these little bumps in the road.
My system is still in a state I can wipe everything and test. Single disk 2 partitions, 2 disk fake RAID-0 2 partition. SSDs or HDD.
I am using UEFI to boot with a EF00 boot partition and a LUKS root partition.
Problem is only with the boot partition of the drive on the Intel fake RAID.
Everything is md_raid, NOT dm_raid for these results.
/dev/md127 container
/dev/md126 inside the /dev/md127 container
/dev/md126p1 EF00 /boot
/dev/md126p2 LUKS /
This nested RAID is done by the Intel RAID OROM. It may be the heart of the cause. I would abandon, but the only way the EFI will boot off of the RAID partition. Booting off of the EFI RAID partition works fine until system tries to mount it according to fstab. The "noauto" option in fstab allows a complete successful boot in this configuration. After booting and login, a systemctl daemon-reload, ls /boot, and everything is perfect until a target change.
Please let us know if there is anything else we can contribute to assist in diagnosing the cause.
Thank you for your assistance.
As I did not have enough available ports on the intel controller I built a DDF container. Pretty much the same results occurred.
Steps for anyone to duplicate:
mdadm -CR --verbose /dev/md0 -e ddf -l container -n 2 /dev/sde /dev/sdf
mdadm -CR --verbose /dev/md1 -l 0 -n 2 /dev/md0 -z 200G
gdisk /dev/md1
o y
n 1 entr +512M efoo
n 2 entr entr entr
w
mkfs.vfat -F32 -n testboot /dev/md1p1
mkfs.ext4 -L testroot /dev/md1p2
mkdir /ctr1
mkdir /ctr2
mount /dev/md1p1 /ctr1
mount /dev/md1p2 /ctr2
(copy some random data over to each)
blkid | grep md1p >> /etc/fstab (then edit to use UUID, mount to ctr1 or 2, respective filesytem)
mdadm --examine --scan >> /etc/mdadm.conf (and edited out the duplicates)
mkinitcpio -p linux
systemctl reboot
Starts up, times out
Maintenance mode
systemctl daemon-reload
mount /crt1
mount /ctr2
ctrl-D
Login
ls /ctr1
(Never displays)
Ctrl-C
systemctl daemon-reload
ls /ctr1
*poof* lists contents
ls /ctr2
*poof* lists contents
We are good to go until we switch targets.
It looks like we just have a problem with RAIDs in a container.
cat /proc/mdstat looks the same before and after
blkid looks the same before and after
ls /dev looks the same before and after
lsmod looks the same before and after
udevadm looks the same before and after
Control group of a standalone disk sdc1 GPT ext4 using UUID in fstab identical to md1p2 always works, no timeouts, noproblems.
Hope this helps get closer to the resolution.
Not sure if it matters, but I am running in pure UEFI mode. (no BIOS or compatibility mode)
Please let me know if I can provide any additional information.
Thank you
The difference is that the boot process never times out, it's stuck after "Assembling FakeRAID devices" and cannot reboot (reboot process starts and hangs as well), so I don't know how to debug this. Is any error written to some log file?
I removed dmraid from mkinitcpio.conf with no effect. Only commenting the dmraid devices from /etc/fstab goes to the emergency shell.
Basically, from what I understood, in order to have the kernel boot I have to build my initramfs with BINARIES="/sbin/mdmon". When systemd tries to shut down, it kills this process and then tries to remount / read-only, failing miserably as mdmon is not running anymore. The bug is solved on Fedora, but they use dracut so I couldn't understand how they solved it. Can anyone help?
Adding mdmon to BINARIES by itself won't do anything (unless I'm missing something). It needs to be started in the initrd, and be passed the --offroot option so that it will not be killed by systemd. Did starting mdmon ever work with initscripts for you?
If I don't put mdmon in the BINARIES array the kernel isn't able to mount the root partition correctly (it hangs before the init, complaining about unknown filesystem or lack of an helper program). If I put it in mkinitcpio.conf and run mkinitcpio -p the kernel boots just fine. Both with initscripts and systemd the shutdown process hangs. With the initscripts it hangs at "unmounting non-API filesystems", with systemd while trying to remount / as read-only (the issue is exactly as in the fedora bug I linked). Is there anything I can do in order to avoid mdmon being killed?
I see in the code where mdadm/mdassemble can try to start mdmon, but mdadm needs to be called with --offroot in order for mdmon to be started with --offroot. Passing this option will prevent mdmon from being killed. Would be nice if mdadm was smart enough to look for /etc/initrd-release to autodetect being run from early userspace and add this flag automatically.
My problem is that I want to use the fake hardware RAID to be able to access it from Windows too.
Does mdadm provide this functionality?
Boot - emergency shell (/home on dmraid not mounted - time out) - manual mount of /home - systemctl default - boot continues and online
So, I cannot boot without manual intervention!!
Is there a plan to fix this also for dmraid?
@gondsman
How is the switch from dmraid to mdadm done?
The raid is also defined in the bios, must this be removed?
- Modify mkinitcpio.conf so that mdadm_udev is in the HOOKS line and /sbin/mdmon is in the BINARIES line (also, remove dmraid if any)
- Rebuild the initramfs with 'mkinitcpio -p linux'
- Update fstab and the bootloader config with the new device names (mdadm assembles partitions as /dev/md126pX)
Once the shutdown issue is solved I'm going to update the fakeRAID wiki page, as it's currently badly out of date.
I have the HW, I'm willing to search for a solution but ... I need help.
@gondsman
The raid is my /home partition, so no need to change or add mkinitcpio
Also the switch from dmraid to mdadm is not that strait forward because I expect the metadata to be different and the disk data will not be accessible anymore.
Suspecting the issue with the shutdown was in the initramfs (thanks to Tom's and Dave's suggestions) I compiled and installed dracut and created a initramfs with it. This way the system boots correctly and it shuts down without errors. Unfortunately, after shutdown the array is flagged as "dirty" and booting into windows (since Intel utility to check the array is for win only) forces a rebuild, which takes several hours. I experienced the same behavior when I gave fedora a try with the same setup.
I don't know if this can help troubleshooting the problem, let me know if I can do more tests.
EDIT: also, this may be relevant (although I'm really not smart enough to figure out exactly how the shutdown process works...): http://board.issociate.de/thread/509251/mdadm-forces-resync-every-boot.html
Thanks again.
EDIT: This may be relevant as well: https://bugzilla.redhat.com/show_bug.cgi?id=753335 . I'm going to try this solution, I'll report back.
Please, note that this problem is not Arch-specific, it's basically due to systemd not supporting initramfs mounting of root partition with dmraid. I have the same bug with Fedora 17. However, given the advertised "simplicity" of Arch (one of the main reasons I chose it), I would expect a "simple" way of checking what my system is doing at boot and during shutdown and an equally "simple" way of modifying it, not something that automagically works most of the times and I have no control over it if it doesn't.
I'm sorry for this post as it doesn't add anything to the discussion, but I really don't want to leave Arch for another distro because of issues like this.
I have:
- not the hardware to test this
- not the interest to fix this
And most of all, I don't have the time to read inane rants about how there's nothing happening with this bug.
@Tom: thanks, I don't know if I'm knowledgeable enough, but I should at least have an idea of what should be run in order to make it work, I just can't figure out how to do it. I'll drop by #archlinux-projects and see if I can help.
@Tom: I'm also willing to help but have not a clue where to start.
Question: where and how can I join #archlinux-projects
However, mdadm does support SOME fakeraid implementation. You should check if your raid can be assembled with mdadm and (if this is the case) use it instead of dmraid. If you want to do it without modifying your installation you can try it with a recent Arch livecd, as it contains mdadm. In my case mdadm automatically assembled my device after booting from the livecd, but doing it manually isn't that much more difficult, just check mdadm manual. At this point check /proc/mdstat to see if mdadm is working properly (you can also mount partitions in the array, they should be assembled as /dev/md126p*).
I've recently installed ArchLinux this week, coming from Gentoo. The hardware configuration I have installed on has not changed in any way from the Gentoo config. I've installed ArchLinux on my root drive and attempted to reconfigure in the storage systems I had running prior to the Arch install.
I am running a 6 drive mdadm RAID10 array mounted on /raid and my /home has been running on a dmraid RAID1 mirror using the "Fakeraid" hardware on my nvidia MB.
I can confirm all of the problems reported in this bug report. My fresh system has no trouble mounting the mdadm raid, but hangs attempting to complete the boot process and eventually drops me into the emergency shell. mount -a or mount /home mounts the dmraid without any issues and a systemctl default continues the boot process.
Seems that mdadm does not support all "fakeraid" chipsets, one of those unsupported being the nvidia chip. Seems a shame as I preferred using the two different code paths to support these two storage devices.
I've decided to stop beating my head on the wall hear (it is getting soft...) and convert the /home to mdadm software raid.
Adding my .02 here so that the higher powers recognize that this is not an unusual scenario.
Thanks. Loving Arch so far...
My guess (I have no idea how to check) is that now mdmon gets launched with the --offroot, so systemd doesn't kill it during shutdown. As a result, the system unmounts the partitions correctly. When root is mounted again read-only, however, the array isn't marked as clean with a
mdadm --wait-clean --scan
and so at every reboot a check is enforced.
@tomegun: I tried dropping by #archlinux-projects but I couldn't find you. I'll try again shortly, thanks for your help.
I have found the reason why /home on dmraid is not booting without manual intervention.
No symlink is made for /dev/mapper/nvidia_cjjcaiiep1 because the udev rules 10-dm.rules are not executed because of ENV{DM_UDEV_DISABLE_DM_RULES_FLAG}!="1", ENV{DM_NAME}=="?*", SYMLINK+="mapper/$env{DM_NAME}" which is true.
When I make a own 10-dm.rules in /etc/udev/rules with
ENV{DM_UDEV_DISABLE_DM_RULES_FLAG}!="0", ENV{DM_NAME}=="?*", SYMLINK+="mapper/$env{DM_NAME}" which is false a symlink is made and the /home is mounted and the pc is booting without any manual intervention.
So, the question is: where is ENV{DM_UDEV_DISABLE_DM_RULES_FLAG} set to '1' and is this correct
I need some help to dig further.
Looking at the source code of libdm and dmraid, my guess is that a missing call to dm_task_set_cookie() in dmraid is the problem.
Please change your dmraid.service file to
"ExecStart=/sbin/dmraid --ignorelocking --activate y -Z -dddd"
Hopefully that should spit out the right debug info.
-dddd added
pc rebooted
Partial output of journalctl -b see file
Thanks to Dave and the other developers on IRC the problem is now solved for people who can use mdadm to mount the array. This means (as far as I know) that fakeRAID with an intel storage matrix chipset is now supported. There are still a few error messages thrown out during shutdown, but everything works. I'm going to update the fakeRAID wiki page with info on how to set up mdadm to work with my chipset.
For users of other chipsets (nvidia for example), the bug is still there, I can't really help.
Concerning my problem, I suppose this is still existing but I cannot test it because my processor went to the cpu walhalla.
I have upgraded with a new mobo and new processor and new ram.
As a concequence the dmraid was not working anymore with the new HW and I have switched to mdadm raid, and this went well without data loss.
Now I can boot without manual intervention.
Thanks for the help.