FS#79837 - [kmod] 31-1 sets my system unbootable

Attached to Project: Arch Linux
Opened by Mihail Zaporozhets (kamheric) - Monday, 02 October 2023, 10:33 GMT
Last edited by Toolybird (Toolybird) - Sunday, 29 October 2023, 22:44 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To No-one
Architecture x86_64
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

Description:
after upgraded kmod (30-3 -> 31-1) my 11th Gen Intel(R) Core(TM) i5-1135G7 laptop can't boot to normal mode.
i recover system by use install media, arch-chroot to system, download old package kmod-30-3-x86_64.pkg.tar.zst, downgrade to it, and sucessful reboot to system.

in journal:
systemd[1]: Failed to start Load Kernel Modules.
...
кт 02 20:39:54 ux425 kernel: rcu: INFO: rcu_preempt self-detected stall on CPU
окт 02 20:39:54 ux425 kernel: rcu: 1-...!: (17419 ticks this GP) idle=f43c/1/0x4000000000000000 softirq=1639/1647 fqs=41
окт 02 20:39:54 ux425 kernel: rcu: (t=18001 jiffies g=-159 q=2706 ncpus=8)
окт 02 20:39:54 ux425 kernel: rcu: rcu_preempt kthread starved for 17876 jiffies! g-159 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
окт 02 20:39:54 ux425 kernel: rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
окт 02 20:39:54 ux425 kernel: rcu: RCU grace-period kthread stack dump:
..
kernel: watchdog: BUG: soft lockup - CPU#3 stuck for 74s! [(udev-worker):326]
...
окт 02 20:45:16 ux425 kernel: traps: Missing ENDBR: init_module+0x0/0xff0 [v4l2loopback_dc]
окт 02 20:45:16 ux425 kernel: ------------[ cut here ]------------
окт 02 20:45:16 ux425 kernel: kernel BUG at arch/x86/kernel/traps.c:255!

This task depends upon

Closed by  Toolybird (Toolybird)
Sunday, 29 October 2023, 22:44 GMT
Reason for closing:  Upstream
Additional comments about closing:  Please see comments
Comment by loqs (loqs) - Monday, 02 October 2023, 15:02 GMT
The error from output you posted is due to [1][2] not implementing a fix similar to [3]. Please post the full journal for a boot with the issue and one without for comparison.

[1] https://aur.archlinux.org/packages/v4l2loopback-dc-dkms
[2] https://github.com/dev47apps/droidcam
[3] https://github.com/umlaeute/v4l2loopback/commit/ddce39b5eb667d3a5db0009f96f6050496bd58ad
Comment by Dario (dario86) - Monday, 02 October 2023, 17:10 GMT
For the record, I am experiencing the same issue on Parabola GNU/Linux-libre.
Comment by loqs (loqs) - Monday, 02 October 2023, 18:12 GMT
Have you been able to bisect [1] which commit introduced the issue? Is there reason to believe this is a packaging issue? The release announcement for kmod 31 [2] from the same mailing list you can use to report the issue upstream.

[1] https://wiki.archlinux.org/title/Bisecting_bugs_with_Git
[2] https://lore.kernel.org/linux-modules/lnbkvt5hhvgksgjqko7t6niw6uzr5ewjp32wyy2s26rzwdgh2y%40l775iv6f6tkz/T/#u
Comment by Toolybird (Toolybird) - Friday, 06 October 2023, 07:22 GMT
Maybe related [1]? But not sure why only a small minority would be affected...

[1] https://lore.kernel.org/linux-modules/acf37f00-cea3-4f77-9c05-3c0301221d2a%40moonlit-rail.com/T/
Comment by loqs (loqs) - Friday, 06 October 2023, 09:21 GMT
> Maybe related [1]? But not sure why only a small minority would be affected...
Seems fairly easy to test by reverting [2]. See attached diff.

[1] https://lore.kernel.org/linux-modules/acf37f00-cea3-4f77-9c05-3c0301221d2a%40moonlit-rail.com/T/
[2] https://git.kernel.org/pub/scm/utils/kernel/kmod/kmod.git/patch/?id=09c9f8c5df0475d2627d9498e26250592dbfebd6
Comment by Dario (dario86) - Tuesday, 10 October 2023, 07:40 GMT
On Parabola GNU/Linux-libre I experience the issue with linux-libre kernel but it works with linux-xanmod. I don't know if it is relevant, but I use Booster for initramfs generation. I have yet to try the patch provided by loqs.
Comment by loqs (loqs) - Tuesday, 10 October 2023, 09:42 GMT
@dario does the initramfs contain any .xz files? Are there any such files in /usr/lib/modules `find /usr/lib/modules/ -name '*.ko.xz'`?
Comment by Dario (dario86) - Tuesday, 10 October 2023, 13:51 GMT
«find /usr/lib/modules/ -name '*.ko.xz'» yelds nothing with linux-libre kernel.
Comment by loqs (loqs) - Tuesday, 10 October 2023, 17:15 GMT
@dario would it help if I built the patched kmod for you to test? I could also help you with the bisection if needed.
Comment by Dario (dario86) - Tuesday, 10 October 2023, 22:18 GMT
Thank you, loqs, I can build it myself. I will get back to you in a few hours and let you know.
Comment by Dario (dario86) - Wednesday, 11 October 2023, 07:19 GMT
I confirm the patched version is working on Parabola GNU/Linux-libre with linux-libre kernel.
Comment by loqs (loqs) - Wednesday, 11 October 2023, 10:06 GMT
Does the output of `lsinitcpio /path/to/initrd` show any compressed kernel modules?
Does linux-libre enable CONFIG_MODULE_DECOMPRESS CONFIG_MODULE_COMPRESS_GZIP CONFIG_MODULE_COMPRESS_XZ CONFIG_MODULE_COMPRESS_ZSTD?
Comment by Dario (dario86) - Wednesday, 11 October 2023, 12:15 GMT
Of these modules, two are enabled in linux-libre: https://git.parabola.nu/abslibre.git/tree/libre/linux-libre/config.x86_64

# lsinitcpio /boot/booster-linux-libre.img
usr
usr/lib
lib
usr/local
usr/local/lib
usr/bin
usr/sbin
bin
sbin
usr/local/bin
usr/local/sbin
run
var
var/run
usr/lib64
lib64
init
usr/lib/modules
usr/lib/modules/libps2.ko
usr/lib/modules/crc16.ko
usr/lib/modules/vivaldi_fmap.ko
usr/lib/modules/xor.ko
usr/lib/modules/xhci_pci.ko
usr/lib/modules/dm_mod.ko
usr/lib/modules/mbcache.ko
usr/lib/modules/cdrom.ko
usr/lib/modules/i8042.ko
usr/lib/modules/crc32c_intel.ko
usr/lib/modules/xhci_pci_renesas.ko
usr/lib/modules/vfat.ko
usr/lib/modules/cryptd.ko
usr/lib/modules/ext4.ko
usr/lib/modules/isofs.ko
usr/lib/modules/fuse.ko
usr/lib/modules/btrfs.ko
usr/lib/modules/squashfs.ko
usr/lib/modules/dm_snapshot.ko
usr/lib/modules/crypto_user.ko
usr/lib/modules/uas.ko
usr/lib/modules/sr_mod.ko
usr/lib/modules/usb_storage.ko
usr/lib/modules/serio.ko
usr/lib/modules/usbhid.ko
usr/lib/modules/atkbd.ko
usr/lib/modules/crypto_simd.ko
usr/lib/modules/jbd2.ko
usr/lib/modules/crct10dif_pclmul.ko
usr/lib/modules/serio_raw.ko
usr/lib/modules/raid6_pq.ko
usr/lib/modules/libcrc32c.ko
usr/lib/modules/aesni_intel.ko
usr/lib/modules/blake2b_generic.ko
usr/lib/modules/crc32c_generic.ko
usr/lib/modules/ghash_clmulni_intel.ko
usr/lib/modules/fat.ko
usr/lib/modules/dm_bufio.ko
usr/lib/modules/crc32_pclmul.ko
usr/lib/modules/booster.alias
etc
etc/booster.init.yaml
etc/initrd-release
Comment by loqs (loqs) - Wednesday, 11 October 2023, 12:41 GMT
To confirm you are on linux-libre 6.3.3-1 or newer as that added CONFIG_MODULE_DECOMPRESS=y? If so I do not know why that kernel is having issues.

[1] https://git.parabola.nu/abslibre.git/commit/libre/linux-libre/config.x86_64?id=f530eea22924e327de176787a29854fabb8691fd
Comment by Dario (dario86) - Wednesday, 11 October 2023, 12:57 GMT
I am on linux-libre 6.3.3-1.
Comment by loqs (loqs) - Wednesday, 11 October 2023, 13:04 GMT
If you use the config from linux-libre 6.3.3-1 to build linux 6.3.3 does that kernel have the issue? If so does updating the kernel to 6.5.7 keeping the same config still produce the issue?
Comment by Dario (dario86) - Wednesday, 11 October 2023, 15:56 GMT
This kernel built from source works for me: https://git.parabola.nu/abslibre.git/tree/libre/linux-libre. Unfortunately at the moment there is no 6.5.7 version of linux-libre packaged for my distribution to test out.
Comment by loqs (loqs) - Wednesday, 11 October 2023, 17:34 GMT
As the issue appears to be limited to the linux-libre kernel have you raised the issue with the linux-libre maintainers? While there may be an issue with mkinitcpio and xz compressed modules due to the missing --lzma2=dict=1MiB option [2] you are using booster and there are no compressed modules in the initrd.

[1] https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio/-/blob/v36/mkinitcpio?ref_type=tags#L233
[2] https://lore.kernel.org/all/1bf10ec1-c154-52f3-79dd-7ec5380aa159%40tweek.dk/
Comment by Dario (dario86) - Wednesday, 11 October 2023, 18:38 GMT
I did not notice before but now there is an open ticket for Parabola GNU/Linux-libre: https://labs.parabola.nu/issues/3539. They are aware of the issue. If it turns out to be a problem with linux-libre, the maintainers of the distribution will eventually raise the issue to the Linux-libre maintainers.
Comment by Pavel Sayekat (pavlushka) - Saturday, 14 October 2023, 08:55 GMT
The OP didn't specify the kernel though.
Comment by Darrell (0strodamus) - Tuesday, 17 October 2023, 02:28 GMT
According to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582#30, it sounds like kmod should not be trying to use the kernel to decompress XZ modules until it can handle CRC64
and dictionaries larger than 1 MiB.
I was able to resolve this issue by compiling my custom kernel using either the patch in the debian link (needed to edit it for the 6.1 LTS kernel) or using gzip compression (tested both).
Comment by loqs (loqs) - Friday, 20 October 2023, 19:01 GMT
Has the issue been fixed for linux-libre [1]?

[1] https://labs.parabola.nu/issues/3539#note-9
Comment by Pavel Sayekat (pavlushka) - Monday, 23 October 2023, 10:26 GMT
Yeah. Now it is using linux-libre-6.5.8-1 vanilla variant as main kernel which is working fine.
Comment by Toolybird (Toolybird) - Sunday, 29 October 2023, 22:44 GMT
Seeing as Arch uses zstd compressed kernel modules by default, the workaround for anyone affected who is using xz compression is straightforward.

Loading...