Arch Linux

Please read this before reporting a bug:
https://wiki.archlinux.org/title/Bug_reporting_guidelines

Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!
Tasklist

FS#74412 - [linux] [linux-zen] root partition not found

Attached to Project: Arch Linux
Opened by John (graysky) - Friday, 08 April 2022, 22:56 GMT
Last edited by Evangelos Foutras (foutrelis) - Saturday, 09 April 2022, 01:16 GMT
Task Type Bug Report
Category Packages: Testing
Status Assigned
Assigned To Jan Alexander Steffens (heftig)
Giancarlo Razzolini (grazzolini)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 3
Private No

Details

Booting into linux-5.17.2.arch2-1 drops me into an emergency shell. Downgrading to linux-5.17.1.arch1-1 fixes this issue. Since the root partition is not mounted, I have no journalctl output to post.

Output on screen:
Starting version 250.4-2-arch
ERROR: device '/dev/nvme0n1p5' not found. Skipping fsck.
mount /new_root: no filesystem type specified.
You are now being dropped into an emergency shell.
sh: can't access tty: job control turned off
This task depends upon

Comment by loqs (loqs) - Friday, 08 April 2022, 22:59 GMT
 FS#74411  Did whatever generated the initrd not add the NVME module?
Comment by John (graysky) - Friday, 08 April 2022, 23:12 GMT
@loqs - it was the hook that triggered it, but do not have anything custom in /etc/mkinitcpio.conf except for MODULES=(amdgpu).

Also, I believe nvme support is not modular in the kernel, no?
% zgrep -i nvme /proc/config.gz
# NVME Support
CONFIG_NVME_CORE=y
CONFIG_BLK_DEV_NVME=y
CONFIG_NVME_MULTIPATH=y
CONFIG_NVME_HWMON=y
CONFIG_NVME_FABRICS=m
CONFIG_NVME_RDMA=m
CONFIG_NVME_FC=m
CONFIG_NVME_TCP=m
CONFIG_NVME_TARGET=m
CONFIG_NVME_TARGET_PASSTHRU=y
CONFIG_NVME_TARGET_LOOP=m
CONFIG_NVME_TARGET_RDMA=m
CONFIG_NVME_TARGET_FC=m
CONFIG_NVME_TARGET_FCLOOP=m
CONFIG_NVME_TARGET_TCP=m
# end of NVME Support
CONFIG_RTC_NVMEM=y
CONFIG_NVMEM=y
CONFIG_NVMEM_SYSFS=y
CONFIG_NVMEM_RMEM=m

Perhaps the image is different?

From pacman.log:
[2022-04-08T18:47:49-0400] [ALPM] upgraded linux (5.17.1.arch1-1 -> 5.17.2.arch2-1)
[2022-04-08T18:47:49-0400] [ALPM] transaction completed
[2022-04-08T18:47:49-0400] [ALPM] running '30-systemd-update.hook'...
[2022-04-08T18:47:49-0400] [ALPM] running '60-depmod.hook'...
[2022-04-08T18:47:50-0400] [ALPM] running '90-mkinitcpio-install.hook'...
[2022-04-08T18:47:50-0400] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
[2022-04-08T18:47:50-0400] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
[2022-04-08T18:47:50-0400] [ALPM-SCRIPTLET] ==> Starting build: 5.17.2-arch2-1
[2022-04-08T18:47:50-0400] [ALPM-SCRIPTLET] -> Running build hook: [base]
[2022-04-08T18:47:50-0400] [ALPM-SCRIPTLET] -> Running build hook: [udev]
[2022-04-08T18:47:50-0400] [ALPM-SCRIPTLET] -> Running build hook: [autodetect]
[2022-04-08T18:47:50-0400] [ALPM-SCRIPTLET] -> Running build hook: [modconf]
[2022-04-08T18:47:50-0400] [ALPM-SCRIPTLET] -> Running build hook: [block]
[2022-04-08T18:47:50-0400] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: xhci_pci
[2022-04-08T18:47:51-0400] [ALPM-SCRIPTLET] -> Running build hook: [filesystems]
[2022-04-08T18:47:51-0400] [ALPM-SCRIPTLET] -> Running build hook: [keyboard]
[2022-04-08T18:47:51-0400] [ALPM-SCRIPTLET] -> Running build hook: [fsck]
[2022-04-08T18:47:51-0400] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2022-04-08T18:47:51-0400] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: /boot/initramfs-linux.img
[2022-04-08T18:47:51-0400] [ALPM-SCRIPTLET] ==> Image generation successful
Comment by loqs (loqs) - Friday, 08 April 2022, 23:21 GMT
NVME support was made modular see [1][2]

[1] https://github.com/archlinux/svntogit-packages/commit/0bbf02b31459152f9ca46984fb823537211031ca
[2] https://github.com/archlinux/svntogit-packages/blob/packages/linux/trunk/config#L2529
Comment by John (graysky) - Saturday, 09 April 2022, 07:20 GMT
As usual, thank you loqs. If I add nvme_core to the MODULES array, I can boot this kernel.

Perhaps a news item is needed before 5.17.2 goes into [core] or maybe something stronger like a check for nvme devices in the readme.install with a subsequent WARNING to stdout? Too much? I am also wondering which wiki page should be updated to reflect this... maybe https://wiki.archlinux.org/title/Installation_guide#Initramfs
Comment by nl6720 (nl6720) - Saturday, 09 April 2022, 07:56 GMT
Isn't this a mkinitcpio bug then?
Comment by Jan Alexander Steffens (heftig) - Saturday, 09 April 2022, 08:45 GMT
I had no problems booting from NVME. My initramfses contain nvme-core.ko without any special config.
Comment by John (graysky) - Saturday, 09 April 2022, 11:49 GMT
@heftig - I cannot reproduce this now; I too have that included in the image without modifying MODULES=
Comment by Jan Alexander Steffens (heftig) - Saturday, 09 April 2022, 12:02 GMT
Have you tried booting the kernel you installed 5.17.2.arch2-1 with and generating the initcpios again?
Comment by John (graysky) - Saturday, 09 April 2022, 12:38 GMT
@heftig - I changed my MODULES= array removing nvme_core, then downgraded to 5.17.1, booted into it, then updated with [testing] enabled to 5.17.2. I could not reproduce this bug.
Comment by Romain D (artefact) - Thursday, 14 April 2022, 08:51 GMT
I have the same issue on linux-5.17.2-arch3: my root partition does not show up during early boot. Downgrading to 5.17.1-arch1 and the system boots normally.

My root device is nvme, here are my mkinitcpio HOOKS: HOOKS="base systemd autodetect modconf block sd-tpm2-totp sd-encrypt sd-vconsole keyboard filesystems"

From pacman.log:
[2022-04-13T23:56:20+0200] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
[2022-04-13T23:56:20+0200] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
[2022-04-13T23:56:20+0200] [ALPM-SCRIPTLET] ==> Starting build: 5.17.2-arch3-1
[2022-04-13T23:56:20+0200] [ALPM-SCRIPTLET] -> Running build hook: [base]
[2022-04-13T23:56:20+0200] [ALPM-SCRIPTLET] -> Running build hook: [systemd]
[2022-04-13T23:56:21+0200] [ALPM-SCRIPTLET] -> Running build hook: [autodetect]
[2022-04-13T23:56:21+0200] [ALPM-SCRIPTLET] -> Running build hook: [modconf]
[2022-04-13T23:56:21+0200] [ALPM-SCRIPTLET] -> Running build hook: [block]
[2022-04-13T23:56:21+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: xhci_pci
[2022-04-13T23:56:21+0200] [ALPM-SCRIPTLET] -> Running build hook: [sd-tpm2-totp]
[2022-04-13T23:56:21+0200] [ALPM-SCRIPTLET] -> Running build hook: [sd-encrypt]
[2022-04-13T23:56:22+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qat_4xxx
[2022-04-13T23:56:22+0200] [ALPM-SCRIPTLET] -> Running build hook: [sd-vconsole]
[2022-04-13T23:56:22+0200] [ALPM-SCRIPTLET] -> Running build hook: [keyboard]
[2022-04-13T23:56:23+0200] [ALPM-SCRIPTLET] -> Running build hook: [filesystems]
[2022-04-13T23:56:23+0200] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2022-04-13T23:56:23+0200] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: /boot/initramfs-linux.img
[2022-04-13T23:56:23+0200] [ALPM-SCRIPTLET] ==> Image generation successful
[2022-04-13T23:56:23+0200] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
[2022-04-13T23:56:23+0200] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
[2022-04-13T23:56:23+0200] [ALPM-SCRIPTLET] ==> Starting build: 5.17.2-arch3-1
[2022-04-13T23:56:23+0200] [ALPM-SCRIPTLET] -> Running build hook: [base]
[2022-04-13T23:56:24+0200] [ALPM-SCRIPTLET] -> Running build hook: [systemd]
[2022-04-13T23:56:24+0200] [ALPM-SCRIPTLET] -> Running build hook: [modconf]
[2022-04-13T23:56:24+0200] [ALPM-SCRIPTLET] -> Running build hook: [block]
[2022-04-13T23:56:25+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: aic94xx
[2022-04-13T23:56:25+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: bfa
[2022-04-13T23:56:25+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qed
[2022-04-13T23:56:25+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qla1280
[2022-04-13T23:56:25+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qla2xxx
[2022-04-13T23:56:25+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: wd719x
[2022-04-13T23:56:26+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: xhci_pci
[2022-04-13T23:56:27+0200] [ALPM-SCRIPTLET] -> Running build hook: [sd-tpm2-totp]
[2022-04-13T23:56:27+0200] [ALPM-SCRIPTLET] -> Running build hook: [sd-encrypt]
[2022-04-13T23:56:27+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qat_4xxx
[2022-04-13T23:56:28+0200] [ALPM-SCRIPTLET] -> Running build hook: [sd-vconsole]
[2022-04-13T23:56:28+0200] [ALPM-SCRIPTLET] -> Running build hook: [keyboard]
[2022-04-13T23:56:29+0200] [ALPM-SCRIPTLET] -> Running build hook: [filesystems]
[2022-04-13T23:56:30+0200] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2022-04-13T23:56:30+0200] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: /boot/initramfs-linux-fallback.img
[2022-04-13T23:56:30+0200] [ALPM-SCRIPTLET] ==> Image generation successful
   IMG_20220414_005640.jpg (1.51 MiB)
Comment by Jan Alexander Steffens (heftig) - Thursday, 14 April 2022, 12:58 GMT
Doesn't the fallback image let you boot? What if you then regenerate the initramfs again?
Comment by Romain D (artefact) - Tuesday, 19 April 2022, 11:39 GMT
The fallback image has the same problem. I've attached a listing of the contents of my initramfs after upgrading to 5.17.3, which has the same issue.
   initramfs-linux.txt (53.7 KiB)
Comment by Jan Alexander Steffens (heftig) - Tuesday, 19 April 2022, 11:51 GMT
Well, the nvme modules are there.
Comment by Eric Blau (eblau) - Thursday, 21 April 2022, 22:27 GMT
I'm having the same problem. I added nvme_core to the MODULES list in /etc/mkinitcpio.conf and re-upgraded to 5.17.4 after downgrading to 5.17.1 to get a bootable system. That did not help.

When I get kicked to an emergency shell, I can see the nvme_core is loaded but my /dev/nvme* devices are all missing. Booting to the fallback image does not help either.
Comment by Eric Blau (eblau) - Friday, 22 April 2022, 16:53 GMT
I got past this issue now by upgrading my bootable USB drive to 5.17.4, booting 5.17.4, mounting my root filesystem and running "mkinitcpio -P" with nvme and nvme_core both added to the MODULES list in /etc/mkinitcpio.conf inside an arch-chroot environment for my root filesystem.

Thanks for the help here, folks.
Comment by Bas Bleeker (Bubbel) - Sunday, 08 May 2022, 23:07 GMT
@Erik, I had similar issues. Seems that I didn't have the block hook in my /etc/mkinitcpio.conf. When added (and without the nvme/vnme_core modules), all worked again.

Maybe something to check out.
Comment by Romain D (artefact) - Tuesday, 17 May 2022, 07:00 GMT
I still haven't figured out the issue with nvme as modules, for now I am running a self compiled kernel with "0bbf02b31459152f9ca46984fb823537211031ca:  FS#68021 ,  FS#74271 : Return nvme to a module" reverted. Adding them to the MODULES array doesn't work for me (they get embedded in the initramfs, and added to /etc/modules-load.d/MODULES.conf fine).

My mkinitcpio.conf (minus comments) is below.

MODULES="nvme nvme_core vfat nvidia nvidia_drm nvidia_modeset nvidia_uvm"
BINARIES=""
FILES="/boot/luks/nvme-Samsung_SSD_950_PRO_256GB_S2GLNCAGA04283T.hdr"
HOOKS="base systemd autodetect modconf block sd-tpm2-totp sd-encrypt sd-vconsole keyboard filesystems"
COMPRESSION="zstd"

Edit: I solved the problem, it was user error. Removing quiet from my cmdline hinted at an invalid parameter value in /etc/modprobe.d.

Loading...