FS#76468 - [systemd] garbage at the end of /proc/cmdline
Attached to Project:
Arch Linux
Opened by Mateusz Gozdek (invidian) - Monday, 07 November 2022, 11:31 GMT
Last edited by Christian Hesse (eworm) - Tuesday, 29 November 2022, 11:08 GMT
Opened by Mateusz Gozdek (invidian) - Monday, 07 November 2022, 11:31 GMT
Last edited by Christian Hesse (eworm) - Tuesday, 29 November 2022, 11:08 GMT
|
Details
Description:
Recently after rebooting I noticed that one of the cmdline setting has not been applied. Upon inspecting `/proc/cmdline`, it turned out that there is some garbage appended at the end of it, which causes last option to not be properly handled by the kernel (in this case it was "audit=0"). This is how it looks like at first sight: ``` $ cat /proc/cmdline rd.luks.name=6e027106-de26-4d6b-b6cd-20706536115e=cryptlvm rd.luks.options=discard root=/dev/mapper/vgroot-lvroot rw mem_sleep_default=deep nmi_watchdog=0 pcie_aspm=force quiet splash loglevel=3 rd.udev.log_priority=3 acpi_osi=Linux acpi_osi=! acpi_osi='Windows 2019' acpi_backlight=vendor pci=nocrs i915.enable_guc=2 i915.enable_psr=0 mitigations=off nvidia-drm.modeset=1 vt.global_cursor_default=0 audit=0 ڣ#, $ cat /proc/cmdline | xxd -d 00000000: 7264 2e6c 756b 732e 6e61 6d65 3d36 6530 rd.luks.name=6e0 00000016: 3237 3130 362d 6465 3236 2d34 6436 622d 27106-de26-4d6b- 00000032: 6236 6364 2d32 3037 3036 3533 3631 3135 b6cd-20706536115 00000048: 653d 6372 7970 746c 766d 2072 642e 6c75 e=cryptlvm rd.lu 00000064: 6b73 2e6f 7074 696f 6e73 3d64 6973 6361 ks.options=disca 00000080: 7264 2072 6f6f 743d 2f64 6576 2f6d 6170 rd root=/dev/map 00000096: 7065 722f 7667 726f 6f74 2d6c 7672 6f6f per/vgroot-lvroo 00000112: 7420 7277 206d 656d 5f73 6c65 6570 5f64 t rw mem_sleep_d 00000128: 6566 6175 6c74 3d64 6565 7020 6e6d 695f efault=deep nmi_ 00000144: 7761 7463 6864 6f67 3d30 2070 6369 655f watchdog=0 pcie_ 00000160: 6173 706d 3d66 6f72 6365 2071 7569 6574 aspm=force quiet 00000176: 2073 706c 6173 6820 6c6f 676c 6576 656c splash loglevel 00000192: 3d33 2072 642e 7564 6576 2e6c 6f67 5f70 =3 rd.udev.log_p 00000208: 7269 6f72 6974 793d 3320 6163 7069 5f6f riority=3 acpi_o 00000224: 7369 3d4c 696e 7578 2061 6370 695f 6f73 si=Linux acpi_os 00000240: 693d 2120 6163 7069 5f6f 7369 3d27 5769 i=! acpi_osi='Wi 00000256: 6e64 6f77 7320 3230 3139 2720 6163 7069 ndows 2019' acpi 00000272: 5f62 6163 6b6c 6967 6874 3d76 656e 646f _backlight=vendo 00000288: 7220 7063 693d 6e6f 6372 7320 6939 3135 r pci=nocrs i915 00000304: 2e65 6e61 626c 655f 6775 633d 3220 6939 .enable_guc=2 i9 00000320: 3135 2e65 6e61 626c 655f 7073 723d 3020 15.enable_psr=0 00000336: 6d69 7469 6761 7469 6f6e 733d 6f66 6620 mitigations=off 00000352: 6e76 6964 6961 2d64 726d 2e6d 6f64 6573 nvidia-drm.modes 00000368: 6574 3d31 2076 742e 676c 6f62 616c 5f63 et=1 vt.global_c 00000384: 7572 736f 725f 6465 6661 756c 743d 3020 ursor_default=0 00000400: 6175 6469 743d 3020 14da a314 231b 2221 audit=0 ....#."! 00000416: 6d1d 181e 2c0a m...,. ``` I'm using `mkinitcpio` to generate UEFI binary, which is then signed by the `sbsign`. Signing does not seem to affect the garbage. This is how EFI binary looks like: ``` $ objdump -h /efi/EFI/arch/linux-zen-signed.efi /efi/EFI/arch/linux-zen-signed.efi: file format pei-x86-64 Sections: Idx Name Size VMA LMA File off Algn 0 .text 00008df0 0000000000003000 0000000000003000 00000400 2**4 CONTENTS, ALLOC, LOAD, READONLY, CODE 1 .reloc 0000000c 000000000000c000 000000000000c000 00009200 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA 2 .data 00003240 000000000000d000 000000000000d000 00009400 2**4 CONTENTS, ALLOC, LOAD, DATA 3 .dynamic 00000100 0000000000011000 0000000000011000 0000c800 2**2 CONTENTS, ALLOC, LOAD, DATA 4 .rela 00000f30 0000000000012000 0000000000012000 0000ca00 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA 5 .dynsym 00000018 0000000000013000 0000000000013000 0000da00 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA 6 .sbat 000000e8 0000000000015000 0000000000015000 0000dc00 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA 7 .sdmagic 0000002e 0000000000015100 0000000000015100 0000de00 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA 8 .osrel 00000163 0000000000020000 0000000000020000 0000e000 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA 9 .cmdline 00000198 0000000000030000 0000000000030000 0000e200 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA 10 .linux 00b58620 0000000002000000 0000000002000000 0000e400 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA 11 .initrd 077613d4 0000000003000000 0000000003000000 00b66c00 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA $ objdump -s -j .cmdline /efi/EFI/arch/linux-zen-signed.efi /efi/EFI/arch/linux-zen-signed.efi: file format pei-x86-64 Contents of section .cmdline: 30000 72642e6c 756b732e 6e616d65 3d366530 rd.luks.name=6e0 30010 32373130 362d6465 32362d34 6436622d 27106-de26-4d6b- 30020 62366364 2d323037 30363533 36313135 b6cd-20706536115 30030 653d6372 7970746c 766d2072 642e6c75 e=cryptlvm rd.lu 30040 6b732e6f 7074696f 6e733d64 69736361 ks.options=disca 30050 72642072 6f6f743d 2f646576 2f6d6170 rd root=/dev/map 30060 7065722f 7667726f 6f742d6c 76726f6f per/vgroot-lvroo 30070 74207277 206d656d 5f736c65 65705f64 t rw mem_sleep_d 30080 65666175 6c743d64 65657020 6e6d695f efault=deep nmi_ 30090 77617463 68646f67 3d302070 6369655f watchdog=0 pcie_ 300a0 6173706d 3d666f72 63652071 75696574 aspm=force quiet 300b0 2073706c 61736820 6c6f676c 6576656c splash loglevel 300c0 3d332072 642e7564 65762e6c 6f675f70 =3 rd.udev.log_p 300d0 72696f72 6974793d 33206163 70695f6f riority=3 acpi_o 300e0 73693d4c 696e7578 20616370 695f6f73 si=Linux acpi_os 300f0 693d2120 61637069 5f6f7369 3d275769 i=! acpi_osi='Wi 30100 6e646f77 73203230 31392720 61637069 ndows 2019' acpi 30110 5f626163 6b6c6967 68743d76 656e646f _backlight=vendo 30120 72207063 693d6e6f 63727320 69393135 r pci=nocrs i915 30130 2e656e61 626c655f 6775633d 32206939 .enable_guc=2 i9 30140 31352e65 6e61626c 655f7073 723d3020 15.enable_psr=0 30150 6d697469 67617469 6f6e733d 6f666620 mitigations=off 30160 6e766964 69612d64 726d2e6d 6f646573 nvidia-drm.modes 30170 65743d31 2076742e 676c6f62 616c5f63 et=1 vt.global_c 30180 7572736f 725f6465 6661756c 743d3020 ursor_default=0 30190 61756469 743d3020 audit=0 $ objdump -s -j .linux /efi/EFI/arch/linux-zen-signed.efi | head /efi/EFI/arch/linux-zen-signed.efi: file format pei-x86-64 Contents of section .linux: 2000000 4d5aea07 00c0078c c88ed88e c08ed031 MZ.............1 2000010 e4fbfcbe 4000ac20 c07409b4 0ebb0700 ....@.. .t...... 2000020 cd10ebf2 31c0cd16 cd19eaf0 ff00f000 ....1........... 2000030 00000000 00000000 00000000 82000000 ................ 2000040 55736520 6120626f 6f74206c 6f616465 Use a boot loade 2000050 722e0d0a 0a52656d 6f766520 6469736b r....Remove disk ``` I also have sufficient space on `/efi` partition. The space after `audit=0` in `/etc/kernel/cmdline` is used as a workaround to avoid garbage affecting last specified option. ``` $ cat /etc/kernel/cmdline rd.luks.name=6e027106-de26-4d6b-b6cd-20706536115e=cryptlvm rd.luks.options=discard root=/dev/mapper/vgroot-lvroot rw mem_sleep_default=deep nmi_watchdog=0 pcie_aspm=force quiet splash loglevel=3 rd.udev.log_priority=3 acpi_osi=Linux acpi_osi=! acpi_osi='Windows 2019' acpi_backlight=vendor pci=nocrs i915.enable_guc=2 i915.enable_psr=0 mitigations=off nvidia-drm.modeset=1 vt.global_cursor_default=0 audit=0 $ cat /etc/mkinitcpio.conf | grep -v ^$ | grep -v ^# MODULES=(i915 nvidia nvidia_drm nvidia_modeset nvidia_uvm thunderbolt) BINARIES=() FILES=() HOOKS=(base systemd sd-plymouth autodetect modconf block sd-encrypt lvm2 filesystems keyboard sd-vconsole fsck) $ cat /etc/mkinitcpio.d/linux-zen.preset # mkinitcpio preset file for the 'linux-zen' package ALL_config="/etc/mkinitcpio.conf" ALL_kver="/boot/vmlinuz-linux-zen" ALL_microcode="/boot/*-ucode.img" PRESETS=('default') default_image="/boot/initramfs-linux-zen.img" default_efi_image="/efi/EFI/arch/linux-zen-signed.efi" $ cat /etc/pacman.d/hooks/sign-kernel.hook [Trigger] Operation = Install Operation = Upgrade Type = Package Target = linux-zen [Action] Description = Signing Kernel for SecureBoot When = PostTransaction Exec = /usr/bin/sbsign --key /etc/efi-keys/db.key --cert /etc/efi-keys/db.crt --output /efi/EFI/arch/linux-zen-signed.efi /efi/EFI/arch/linux-zen-signed.efi Depends = sbsigntools ``` I've also tried kernels `6.0.6-zen1-2-zen`, `6.0.3-zen2-1-zen`, `linux-zen-6.0.1.zen1-1` and `linux-zen-5.19.9.zen1-1`, all with the same result. Additional info: * package version(s): linux-zen 6.0.7.zen1-1 mkinitcpio 32-2 openssl 3.0.7-2 sbsigntools 0.9.4-2 Unfortunately I don't have steps to reproduce. |
This task depends upon
Closed by Christian Hesse (eworm)
Tuesday, 29 November 2022, 11:08 GMT
Reason for closing: Fixed
Additional comments about closing: systemd 252.2-2
Tuesday, 29 November 2022, 11:08 GMT
Reason for closing: Fixed
Additional comments about closing: systemd 252.2-2
Also, garbage remains the same over reboots and when printing /proc/cmdline multiple times or mounting proc in a different place. I've also tried generating EFI binary using sbsign and the result was the same, but then between kernel versions or regenerating the EFI binary the garbage data changed.
$ pacman -Q systemd
systemd 252-1
$ systemctl --version
systemd 252 (252-1-arch)
+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified
```
I'm fully updated with testing repositories enabled.
https://github.com/systemd/systemd/pull/25222
Can we make mkinitcpio add this null-byte instead?
as it uses fixed offsets.
Source: https://github.com/systemd/systemd/releases/tag/v252
Extract:
The PE section offsets that are used by tools that assemble unified
kernel images have historically been hard-coded. This may lead to
overlapping PE sections which may break on boot. The UKI will now try
to detect and warn about this.
Any tools that assemble UKIs must update to calculate these offsets
dynamically. Future sd-stub versions may use offsets that will not
work with the currently used set of hard-coded offsets!
FS#76641```
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -U /boot/archlinux-linux.efi -g /boot/initramfs-linux.img --splash /usr/share/systemd/bootctl/splash-arch.bmp --microcode /boot/*-ucode.img
==> Starting build: 6.0.9-arch1-1
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [resume]
-> Running build hook: [autodetect]
libkmod: kcmdline_parse_result: Ignoring bad option on kernel command line while parsing module name: 'BZ[OKKKKKK[CKK4}45%&7/'
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: xhci_pci
-> Running build hook: [filesystems]
-> Running build hook: [keyboard]
-> Running build hook: [keymap]
-> Running build hook: [fsck]
==> Generating module dependencies
libkmod: kcmdline_parse_result: Ignoring bad option on kernel command line while parsing module name: 'BZ[OKKKKKK[CKK4}45%&7/'
==> Creating zstd-compressed initcpio image: /boot/initramfs-linux.img
==> Image generation successful
==> Creating UEFI executable: /boot/archlinux-linux.efi
-> Using UEFI stub: /usr/lib/systemd/boot/efi/linuxx64.efi.stub
-> Using kernel image: /lib/modules/6.0.9-arch1-1/vmlinuz
-> Using cmdline file: /etc/kernel/cmdline
-> Using os-release file: /etc/os-release
-> Using splash image: /usr/share/systemd/bootctl/splash-arch.bmp
-> Using microcode image: /boot/intel-ucode.img
==> UEFI executable generation successful
```
Does that mean the UEFI executable gets corrupted somehow? Is it still bootable?
When UKI created with manual method as described in wiki https://wiki.archlinux.org/title/Unified_kernel_image#Manually
no processing on cmdline done.
In both cases .cmdline section size is equal
# objdump /boot/EFI/Linux/archlinux.efi -h -j .cmdline
/boot/EFI/Linux/archlinux.efi: file format pei-x86-64
Sections:
Idx Name Size VMA LMA File off Algn
9 .cmdline 0000005f 0000000000015293 0000000000015293 0000e200 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
Section dump for mkinitcpio created UKI
# objdump /boot/EFI/Linux/archlinux.efi -s -j .cmdline
/boot/EFI/Linux/archlinux.efi: file format pei-x86-64
Contents of section .cmdline:
15293 726f6f74 3d4c4142 454c3d41 52434846 root=LABEL=ARCHF
152a3 53207473 633d756e 73746162 6c652074 S tsc=unstable t
152b3 72616365 5f636c6f 636b3d6c 6f63616c race_clock=local
152c3 20727720 38323530 2e6e725f 75617274 rw 8250.nr_uart
152d3 733d3120 6c6f676c 6576656c 3d332066 s=1 loglevel=3 f
152e3 73636b2e 6d6f6465 3d666f72 636520 sck.mode=force
Section dump for manually created UKI
# objdump /boot/EFI/Linux/archlinux.efi -s -j .cmdline
/boot/EFI/Linux/archlinux.efi: file format pei-x86-64
Contents of section .cmdline:
15293 726f6f74 3d4c4142 454c3d41 52434846 root=LABEL=ARCHF
152a3 53207473 633d756e 73746162 6c652074 S tsc=unstable t
152b3 72616365 5f636c6f 636b3d6c 6f63616c race_clock=local
152c3 20727720 38323530 2e6e725f 75617274 rw 8250.nr_uart
152d3 733d3120 6c6f676c 6576656c 3d332066 s=1 loglevel=3 f
152e3 73636b2e 6d6f6465 3d666f72 63650a sck.mode=force.
The only difference is space/EOL byte at the end.
In file are this byte sequences followed by null characters.
When booted I get different outputs for
# cat /proc/cmdline
root=LABEL=ARCHFS tsc=unstable trace_clock=local rw 8250.nr_uarts=1 loglevel=3 fsck.mode=force ptalk
root=LABEL=ARCHFS tsc=unstable trace_clock=local rw 8250.nr_uarts=1 loglevel=3 fsck.mode=force
On other computers I get longer and nonsense garbage.
It looks like .cmdline section is put in memory from UKI as is and then during cat is processed beyond its size
until it randomly found null or eol character.
- printf('\0') = garbage
- printf('\n') = garbage
- printf('\n\0') = OK
- printf('\n\n') = OK
- printf('\0\0') = garbage
It looks, that problems are with space character at the end of string, produced by
(grep '^[^#]' "$cmdline" | tr -s '\n' ' ')
when i replace last space with EOL:
grep '^[^#]' "$cmdline" | tr -s '\n' ' ' | sed 's/.$/\n/')
all is OK (without garbage):
# objdump /boot/EFI/Linux/archlinux.efi -s -j .cmdline
/boot/EFI/Linux/archlinux.efi: file format pei-x86-64
Contents of section .cmdline:
30000 726f6f74 3d4c4142 454c3d41 52434846 root=LABEL=ARCHF
30010 53207473 633d756e 73746162 6c652074 S tsc=unstable t
30020 72616365 5f636c6f 636b3d6c 6f63616c race_clock=local
30030 20727720 38323530 2e6e725f 75617274 rw 8250.nr_uart
30040 733d3120 6c6f676c 6576656c 3d332066 s=1 loglevel=3 f
30050 73636b2e 6d6f6465 3d666f72 63650a sck.mode=force.
https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio/-/commit/655f3085fa1f12be765b4baa346c4f5cbe04c9b2
(sbctl puts /etc/kernel/cmdline direcly to UKI)
Maybe something else changed in systemd-252.2, I'll dig a bit further tonight.
systemd 252.2-1 doesn't have it
systemd 252.2-2 doesn't have it either, but not sure how relevant that last fix is then?
mkinitcpio inserts invalid .cmdline section to UKI (it's ended by space, not \n character) and
rely on correct parsing in systemd - which was broken.