Arch Linux

Please read this before reporting a bug:

Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!

FS#60272 - [mkinitcpio] initramfs created without modules needed for booting with root fs on USB hard drive

Attached to Project: Arch Linux
Opened by Ferdinand Bachmann (Ferdi265) - Monday, 01 October 2018, 20:17 GMT
Last edited by Dave Reisner (falconindy) - Tuesday, 09 October 2018, 14:18 GMT
Task Type Bug Report
Category Arch Projects
Status Closed
Assigned To Dave Reisner (falconindy)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 17
Private No


## Description:

After installing linux-4.18.11, the mkinitcpio pacman hook creates an unbootable initramfs (fallback works) when
booting with the root filesystem on a USB hard drive because the usb_storage and uas modules are not added to the initramfs.

## Additional info:

=> package versions:

linux: 4.18.11 (problem doesn't appear when downgrading to 4.18.10)
mkinitcpio: 24-2

=> USB hard drive details:

UGreen USB 3.0 2.5inch SATA hard drive enclosure

## Steps to reproduce:

1.) setup a root filesystem on a USB hard drive
2.) boot correctly on stable kernel
3.) update to testing kernel
4.) reboot
5.) see linux not find the root filesystem

## Workaround

Add "usb_storage" and "uas" to the MODULES list in /etc/mkinitcpio.conf

## Discussion

I'm not sure what is causing this problem. Interestingly, mkinitcpio wasn't updated in the update that broke,
and downgrading linux fixes the initramfs again, so I'm guessing some change in the linux package was responsible.
(building the initramfs for linux-4.18.11 on a running linux-4.18.11 (via fallback) creates a broken initramfs as well)

The file list diff between the two linux packages shows some usb-related kernel module files were removed
(maybe they were instead linked into the kernel?), but I don't know much about the detailed changes between the two versions.

I can add further tests if needed and hope to get this fixed soon.
This task depends upon

Closed by  Dave Reisner (falconindy)
Tuesday, 09 October 2018, 14:18 GMT
Reason for closing:  Fixed
Additional comments about closing:  testing/mkinitcpio-24-3
Comment by mark (qinohe) - Tuesday, 02 October 2018, 22:01 GMT
Hi, I am also affected by this change in the config but in linux-hardened, and will be affected when linux-4.18.11 hits core.

for details see this thread

Everette88 has provided a link with the change for linux-hardened

That shows USB modules moved to 'buitlt in' so anyone booting from certain setups fail.

Thanks, qinohe

Comment by Dave Reisner (falconindy) - Thursday, 04 October 2018, 17:23 GMT
Well, the thread is wrong. Attempting to load a builtin module will *succeed*, not fail. For example, take a random builtin module and try to load it:

$ shuf -n1 /lib/modules/$(uname -r)/modules.builtin
$ modprobe regmap-mmio; echo $?

No problems there... mkinitcpio knows very well how to deal with builtins. It ignores them, because there's nothing to be done, nothing to load. The functionality is initialized with the kernel and always available.
Comment by rainer (raneon) - Thursday, 04 October 2018, 17:46 GMT
I'm affected by this as well and as there was no real error messages besides root device not found, I thought my USB stick didn't work anymore. So I ordered a SSD already as a replacement, but when I had the time to look closer at this issue I've seen that linux-hardened-fallback still worked. Then I moved the hook block before autodetect in mkinitcpio.conf and now my system works again. It would be nice to know the actual root cause of this issue, I have several other systems running with a USB stick as the main drive and not being able to boot is something that could be easily avoided by communicating a change properly (not implying that this is the fault of mkinitcpio!).
Comment by loqs (loqs) - Thursday, 04 October 2018, 17:57 GMT
Please post the working and none working mkinitcpio.conf and the output of `mkinicpio -v -p preset` (where preset is linux / linux-hardened etc)
Comment by mark (qinohe) - Thursday, 04 October 2018, 18:28 GMT
@raneon, Thanks, I don't know if that is the solution since 'autodetect' should run before other 'subsystem hooks' like 'block' I think ;)

@Dave, Thanks, my module is now in mkinitcpio and system is booting.

my output, slightly different of course:
$ shuf -n1 /lib/modules/$(uname -r)/modules.builtin
$ modprobe regmap-mmio; echo $?

Do I understand correct, module is loading but system is not able to use it, you write it quite cryptic ;-)
Thanks, mark
Comment by Tommy Schmitt (spinka) - Thursday, 04 October 2018, 19:00 GMT
@falconindy recently there were issues[0] with explicitly loading sha512 when it switched from being module to built-in. Now with USB drivers after similar change. Something clearly breaks when this happens.

Comment by mark (qinohe) - Thursday, 04 October 2018, 19:06 GMT
Hi @loqs,Thanks there is not much difference;)

MODULES=(i2c_hid i2c_designware_core i2c_designware_platform usb_storage uas)
HOOKS=(base udev autodetect modconf block filesystems keyboard fsck)

not working:
MODULES=(i2c_hid i2c_designware_core i2c_designware_platform)
Thanks, mark
   mkinit (40 KiB)
Comment by Ferdinand Bachmann (Ferdi265) - Thursday, 04 October 2018, 20:32 GMT
It seems the problem is not that the builtins don't work -- they do -- it's that mkinitcpio then seems to think that usb_storage isn't needed.

Upon inspection of the mkinitcpio code, the block hook does the following:

# usb
if add_checked_modules -f '(_cs|sl811_hcd|isp116x_hcd)' '/usb/host'; then
blockdevs+=(['usb_storage?']=1 ['sd_mod?']=1 ['sr_mod?']=1)
add_checked_modules '/drivers/usb/storage/'.

It only adds usb_storage if at least one module matching this regex is found by add_checked_modules.
On my system, these are "ehci_pci" and "xhci_pci", both modules which are now builtins.

What does add_checked_modules do?
It adds modules using add_module, but only if a matching module exists in all_modules and the autodetected module list (skipping the autodetect list on fallback)

The problem is now that all_modules just lists modules using "find" on the kernel module directory -- ignoring builtin modules.
This means that the block hook now thinks that no usb support is needed and proceeds without adding the usb_storage modules.

I think this could be fixed by also checking the list of builtins, but I don't know how I would patch that.
Definitely a bug in mkinitcpio proper, and not just a packaging issue.
Comment by Jeremy M. (jskier) - Friday, 05 October 2018, 18:24 GMT
I have a Chromebook booting to a root USB3 device. Confirming going back to 4.18.1 (or linux-lts) for example, it boots fine without pumping in the modules to mkinitcpio.conf.

Also the patchset kernels seem to be impacted (ie. ck). Makes sense, they're usually based off of the linux package in core.
Confirming also fallback does work. Lastly, confirming same results when specifying usb_storage in modules for mkinitcpio.conf, and it boots up just fine.
Comment by Maxim (mxfm) - Saturday, 06 October 2018, 08:51 GMT
I am also affected by the issue. In my case adding just 'usb_storage' (without other modules mentioned in this issue) module to mkinitcpio.conf is sufficient.
Comment by salam alekum (alu1) - Saturday, 06 October 2018, 23:09 GMT
that's a saturday never getting back - bug killed all my machine's that use usb during boot - in the end fixed with adding uas module but had to plant random sleeps in my custom initcpio hooks that have served me well for years - fair to say though this is the first major arch headache i've had in all those years!
Comment by Ave Ozkal (aoz) - Monday, 08 October 2018, 08:36 GMT
In case anyone is wondering, this issue is still present on linux 4.18.12. I fell back to linux-lts for the time being.
Comment by Maxim (mxfm) - Monday, 08 October 2018, 08:44 GMT
It looks like -fallback image works, so it can be used instead of linux downgrading.
Comment by Dave Reisner (falconindy) - Monday, 08 October 2018, 13:34 GMT
The +1 are unnecessary, Ferdi265 has correctly pointed out the issue (thank you for debugging that). This needs to be fixed in mkinitcpio.
Comment by Dave Reisner (falconindy) - Monday, 08 October 2018, 23:50 GMT
Could someone affected try this patch with autodetect?

Please include the output of 'mkinitcpio -v', particularly if the patch works for you.
Comment by salam alekum (alu1) - Tuesday, 09 October 2018, 12:48 GMT
patched /usr/lib/initcpio/install/block and it works for me - now boots root filesystem from usb without including uas or usb_storage modules in mkinitcpio.conf, standard hooks (base udev autodetect modconf block filesystems keyboard fsck). mkinitcpio -v output attached. not tried with my custom hooks yet but thks dave!
Comment by mark (qinohe) - Tuesday, 09 October 2018, 13:22 GMT
Thanks for the patch Dave, but for me it's not working ;(
patched /usr/lib/initcpio/install/block - removed 'uas - usb_storage' and ran 'mkinitcpio -p linux' reboot, emergency shell.
Comment by Dave Reisner (falconindy) - Tuesday, 09 October 2018, 13:31 GMT
No but really, I wanted the 'mkinitcpio -v' output.
Comment by mark (qinohe) - Tuesday, 09 October 2018, 13:42 GMT
Sorry really, here you go ;)
   mkinit (40 KiB)
Comment by Ferdinand Bachmann (Ferdi265) - Tuesday, 09 October 2018, 13:46 GMT
I tested the patch and made logs of `mkinitcpio -p linux -v` and `lsinitcpio /boot/initramfs-linux{,-fallback}.img`.
Here is the gist with the logs (patched and unpatched version without workaround):


unpatched: default doesn't include usb-storage.ko or uas.ko, fallback includes both
pathced: default includes usb-storage.ko, but not uas.ko, fallback includes usb-storage.ko, but not uas.ko

So autodetect was fixed, but it broke non-autodetect instead.

Comment by Dave Reisner (falconindy) - Tuesday, 09 October 2018, 13:46 GMT
@qinohe: Looks like you ran mkinitcpio with uas and usb-storage in your MODULES array...
Comment by Dave Reisner (falconindy) - Tuesday, 09 October 2018, 13:48 GMT
Could someone who depends on 'uas' (and *actually* depends on uas, because not all of you really do), post the output of 'mkinitcpio -M' ?
Comment by Ferdinand Bachmann (Ferdi265) - Tuesday, 09 October 2018, 13:52 GMT
I depend on uas, here's the output of mkinitcpio -M (unpatched and patched outputs are exactly the same):

as you can see, uas is detected
Comment by soloturn (soloturn) - Tuesday, 09 October 2018, 13:52 GMT
thanks mark for the patch it worked.

$ sudo mkinitcpio -v -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-3.img > patched
$ ls -l /etc/mkinitcpio.conf
-rw-r--r-- 1 root root 2490 6. Okt 2017 /etc/mkinitcpio.conf
$ ls -l /etc/mkinitcpio.d/linux.preset
-rw-r--r-- 1 root root 375 4. Okt 03:00 /etc/mkinitcpio.d/linux.preset
$ grep usb-sto patched
adding module: usb-storage
   patched (10.4 KiB)
Comment by Ferdinand Bachmann (Ferdi265) - Tuesday, 09 October 2018, 13:54 GMT
it seems your patch doesn't contain the line

add_checked_modules '/drivers/usb/storage/'
Comment by Dave Reisner (falconindy) - Tuesday, 09 October 2018, 14:06 GMT
@Ferdi265: Yep, that was wrongly dropped. Does re-adding it fix the autodetect case?
Comment by Ferdinand Bachmann (Ferdi265) - Tuesday, 09 October 2018, 14:11 GMT
@falconindy: adding that line again fixes both cases (autodetect and fallback)
Comment by Dave Reisner (falconindy) - Tuesday, 09 October 2018, 14:15 GMT