FS#57860 - [mdadm] mdadm_udev does not assemble degraded array during boot

Attached to Project: Arch Linux
Opened by Daan van Rossum (drrossum) - Friday, 16 March 2018, 15:03 GMT
Last edited by Buggy McBugFace (bugbot) - Saturday, 25 November 2023, 20:13 GMT
Task Type Bug Report
Category Arch Projects
Status Closed
Assigned To Tobias Powalowski (tpowa)
Felix Yan (felixonmars)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 7
Private No


mkinitcpio's mdadm_udev hook uses mdadm upstream's udev rules to incrementally assemble raid arrays. 64-md-raid-assembly.rules uses systemd.timer to delay starting of partially assembled arrays to fix issues with slow array members (e.g. usb devices). But systemd.timer is not functional in initcpio. In case the array holds the root filesystem this has severe consequences:
- partially assembled array remain "inactive"
- the root FS is not available and does not get mounted
- boot process is interrupted with an emergency shell

From here, the following steps resume the boot process with a degraded array:
# mdadm --incremental --run --scan
# mount /dev/md0 /new_root
# exit

This can be fixed using custom install and hook files for the mdadm_udev hook, see attached.

I think it should be mdadm_udev's default behavior to auto-assemble arrays (that are configured for auto-assembly in mdadm.conf) in degraded mode if necessary, instead of interrupting the boot process.

Additional info:
mkinitcpio 24-2
mdadm 4.0-1

Steps to reproduce:
- Create a md raid-1 array
- move a system's root FS on the raid-1 array
- poweroff, remove one of the raid members
- poweron
- boot fails due to root FS not mounted
This task depends upon

Closed by  Buggy McBugFace (bugbot)
Saturday, 25 November 2023, 20:13 GMT
Reason for closing:  Moved
Additional comments about closing:  https://gitlab.archlinux.org/archlinux/p ackaging/packages/mdadm/issues/1
Comment by Daan van Rossum (drrossum) - Friday, 16 March 2018, 15:06 GMT
I believe this fix may be related to  FS#49071 , as the mdadm_udev hook will then only return once the arrays are assembled, in time for the lvm2 hook to start using the lv metadata.
Comment by eaut (eaut) - Sunday, 01 September 2019, 22:07 GMT
The bug is still present and therefore booting from an degraded md-raid device fails.

The way I understand it, the package build file "mdadm_udev_install" misses the following build rules:

add_systemd_unit "mdadm-last-resort@.service"
add_systemd_unit "mdadm-last-resort@.timer"

This makes sure that the required systemd units are copied to the initramfs. They are triggered by 64-md-raid-assembly.rules as described above.
No additional fixes should be required.
Comment by Daan van Rossum (drrossum) - Friday, 18 October 2019, 10:46 GMT
eaut, the fix you suggest works in a systemd init but not in a busybox init because systemd timer functionality is not available.

The fix I suggested is independent of the init system. It is less complicated than it may seem from the attached files. In fact, there is a one-line install patch and a few-line hook. The hook sleeps for 10 seconds if degraded arrays are detected and then retries to assemble the arrays and starts them even if some are still degraded.

--- /usr/lib/initcpio/install/mdadm_udev
+++ /etc/initcpio/install/mdadm_udev
@@ -12,8 +12,6 @@
add_binary "/usr/bin/mdadm"
add_file "/usr/lib/udev/rules.d/63-md-raid-arrays.rules"
add_file "/usr/lib/udev/rules.d/64-md-raid-assembly.rules"
+ add_runscript

help() {
Comment by eaut (eaut) - Sunday, 20 October 2019, 13:23 GMT
@Daan: I did not have busybox init in mind - why are you preferring it over systemd init?
Hopefully all the currently open initrd bugs with LVM, MD, etc. will be history after the anticipated Dracut switch.
Comment by eaut (eaut) - Sunday, 20 October 2019, 16:15 GMT
@Daan: I did not have busybox init in mind - why are you preferring it over systemd init?
Hopefully all the currently open initrd bugs with LVM, MD, etc. will be history after the anticipated Dracut switch.
Comment by Alexander Chumakov (delfer) - Sunday, 15 March 2020, 14:10 GMT
@drrossum thank you very much!

Here is my test case:

==0==) Build raid1 array

==1==) Add 1 line to /usr/lib/initcpio/install/mdadm_udev
--- /usr/lib/initcpio/install/mdadm_udev.orig 2020-03-15 16:37:07.967859694 +0300
+++ /usr/lib/initcpio/install/mdadm_udev 2020-03-15 16:38:18.824530798 +0300
@@ -12,6 +12,8 @@
add_binary "/usr/bin/mdadm"
add_file "/usr/lib/udev/rules.d/63-md-raid-arrays.rules"
add_file "/usr/lib/udev/rules.d/64-md-raid-assembly.rules"
+ add_runscript

help() {

==2==) Add new file /usr/lib/initcpio/hooks/mdadm_udev (mdadm_udev.hook from bug report)

==3==) mkinitcpio -p linux

==4==) shutdown -h now

==5==) Remove one hard drive and turn on the computer

It's started!

==6==) Shutdown computer, attach disk back and start the computer again

==7==) Attach disk back to raid
mdadm --manage --add /dev/md126 /dev/sdb2

==8==) Wait for sync to be complete
watch 'cat /proc/mdstat'
Comment by Sébastien Luttringer (seblu) - Tuesday, 07 April 2020, 13:07 GMT
Since mdadm 4.1-1 and mdadm hook has been replaced by mdadm_udev hook, I didn't yet faced a situation where I needed to reboot a server with a degraded root array.

Tobias, did you check if booting on a degraded array is now possible with mdadm_udev? This could close this one, and, if not it's a real issue.
Comment by eaut (eaut) - Wednesday, 08 April 2020, 06:23 GMT
Sébastien, "add_systemd_unit" is still missing in mdadm_udev_install.
See my comment from Sunday, 01 September 2019.
Comment by Nigel Kukard (nkukard) - Monday, 13 July 2020, 11:40 GMT
I've just run into the same issue when using qemu and -hda test1.qcow2 and -hdb test2.qcow2 .

After install, removing -hda test1.qcow2 causes the system to fail booting.
Comment by eaut (eaut) - Thursday, 10 September 2020, 16:55 GMT
-- bug wrangling day --
bug is still present, easy fix is described in comment from Sunday, 01 September 2019
Comment by Daan van Rossum (drrossum) - Thursday, 10 September 2020, 22:38 GMT
Please note the perceived shortcomings of that suggestion, as described in the comment from 2019-10-18.

In short: the proposed workaround seems to depend on optional hooks in the initramfs, so it may not work for all users.
Comment by eaut (eaut) - Friday, 11 September 2020, 05:31 GMT
Why are the hooks optional? They are already provided by the package!
See pacman -Ql mdadm | grep mdadm-last-resort
The only dependency for this fix is systemd - I would guess a solution for 99% of all systems.
Comment by Daan van Rossum (drrossum) - Tuesday, 15 September 2020, 20:05 GMT
Well, the user has the option to choose between systemd and busybox as a basis for their initcpio. If the user opts for busybox then the added systemd timer will never by executed.
Comment by eaut (eaut) - Tuesday, 15 September 2020, 20:22 GMT
A direct quote from the ArchWiki documentation (https://wiki.archlinux.org/index.php/Mkinitcpio#Using_RAID)
"Note: mdadm is deprecated. If using it you will see ==> WARNING: Hook 'mdadm' is deprecated. Replace it with 'mdadm_udev' in your config when doing an upgrade."
Comment by Daan van Rossum (drrossum) - Tuesday, 15 September 2020, 20:29 GMT
No, this is about the init system: base+udev vs base+systemd hooks
Comment by eaut (eaut) - Tuesday, 15 September 2020, 21:09 GMT
Anyway, the real problem is that we have bug fixes for both systemd and busybox for many months now and nobody integrates them.
In case of systemd init it is a simple two line oversight that needs to be fixed.
Comment by Tobias Powalowski (tpowa) - Wednesday, 30 March 2022, 08:03 GMT
Is this still an issue with 4.2?
Comment by D (skull133) - Sunday, 24 April 2022, 14:06 GMT
Guys, Im sorry, but I am very noob, but I really need this issue to be fixed. "This can be fixed using custom install and hook files for the mdadm_udev hook, see attached." - please tell how can this may be done. Googled, but not found. Thanks in advance!
Comment by Alexander E. Patrakov (patrakov) - Sunday, 16 October 2022, 18:15 GMT
It is still an issue with mdadm 4.2-2.
Comment by Alexander Blinne (Sunday) - Monday, 17 October 2022, 16:21 GMT
What happened here, was a classical case of trying to fix too much in one go.

The mdadm udev rules themselves are already targeted at systemd, which is obvious since they rely on the timer. The fix proposed by the op using an additional hook may work in the generic case, but the one by eaut is actually correct for a system using systemd. The more generic fix replaces a broken part of the systemd way with a workaround, but leaves the broken part in place, making for an awkward solution.

The correct solution for a system using busybox would be to change the udev rules themselves such that they do not rely on systemd. That would of course be a little more complicated than anything already proposed here.

I suggest doing two things. First, fix the obvious bug. Then deal with the more generic problem separately.
* apply eaut's fix in this context to solve the problem for systemd users
* open a new ticket to fix the rules for systems not running systemd but busybox init in the mkinitcpio. This can be done either by any of the following:
1. enhancing the udev rules that they are agnostic to sysdemd/busybox
2. creating a separate set of udev rules for this case
3. creating a separate mkinitcpio hook (mdadm_busybox) which fixes the udev rules by running the hook script provided by the op, but does not make the regular mdadm+systemd case awkward.

Opening a new ticket also should increase the chance to get anything done quickly for the busybox side, because all relevant information can be presented in a well sorted and concise manner.

The original description of this ticket does not mention busybox and therefore the fix applied for this ticket should be the obvious one, adding the missing timer to the initcpio.
Comment by Daan van Rossum (drrossum) - Tuesday, 18 October 2022, 14:27 GMT
Thank you, Alexander, for this valuable input. I agree with your assessment.
Comment by Alexander E. Patrakov (patrakov) - Tuesday, 18 October 2022, 14:40 GMT
For the busybox case, I would vote for solution 3: a separate mkinitcpio hook that starts the arrays if they are still degraded after the timeout.
Comment by Vecíno (vecino) - Monday, 27 February 2023, 07:50 GMT
Hi, I spent several hours about solving this problem before I found this thread. Please can you include this fix directly in the mdadm package? I think it will make life easier for more users. Thanks!
Comment by Benjamin Flesch (bflesch) - Wednesday, 21 June 2023, 14:30 GMT
I'm also affected by this issue. I have /dev/md0 is boot and /dev/md1 is cryptdevice. It drops to rescue shell and there I see that md0 is started but md1 is not started.
Comment by Buggy McBugFace (bugbot) - Tuesday, 08 August 2023, 19:11 GMT
This is an automated comment as this bug is open for more then 2 years. Please reply if you still experience this bug otherwise this issue will be closed after 1 month.