Arch Linux

Please read this before reporting a bug:

Do NOT report bugs when a package is just outdated, or it is in Unsupported. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!

FS#63697 - systemd 243.0-1 poweroff/reboot umount errors

Attached to Project: Arch Linux
Opened by Scott K. (Pillgar) - Monday, 09 September 2019, 00:36 GMT
Last edited by freswa (frederik) - Saturday, 22 February 2020, 16:18 GMT
Task Type Bug Report
Category Packages: Core
Status Assigned
Assigned To Dave Reisner (falconindy)
Christian Hesse (eworm)
Architecture x86_64
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 6
Private No


Description: after upgrading from systemd 242.84-2 to 243.0-1 (lib32 included), I've noticed two errors during shutdown or reboot.

sd-umount[1636]: Failed to umount /oldroot: Device or resource busy
sd-umount[1637]: Failed to umount /oldroot/sys: Device or resource busy

Might be a regression from I vaguely remember seeing this in a previous version awhile ago.

Additional info:
* package version(s) systemd 243.0-1

Steps to reproduce:
upgrade to systemd 243.0-1 and reboot
This task depends upon

Comment by jari_45 (jari_45) - Wednesday, 11 September 2019, 08:59 GMT
I have noticed this bug too.
Comment by christian (damachine) - Thursday, 12 September 2019, 15:08 GMT
Comment by gamezelda (gamezelda) - Friday, 13 September 2019, 21:08 GMT
After upgrading to systemd 243.0-1 I started getting a new message when shutting down / rebooting:

printk: shutdown: 4 output lines suppressed due to ratelimiting

If I add the "debug" parameter to the kernel command line, I see the "Failed to umount /oldroot" message as well, as reported in this bug.

HOWEVER, If I downgrade to systemd 242.84-2, I no longer see the "lines suppressed due to ratelimiting" message, but I still see the "Failed to umount /oldroot" message. So, it appears that at least in my system, the "Failed to umount /oldroot" always appeared with the "debug" kernel parameter. But the "lines suppressed due to ratelimiting" message started appearing with this upgrade.

Now, if on 243.0-1 I run "sudo sysctl -w kernel.printk_devkmsg=on -w kernel.printk_ratelimit=0", the "lines suppressed due to ratelimiting" message disappears, with no apparent ill effect (I tried various debugging parameters like "debug", "printk_devkmsg=on", etc. and didn't find any further error).

My current hypothesis is:
* The "Failed to umount /oldroot" message is not a regression from 242.84-2 -> 243.0-1, it was there but I simply didn't see it because it only appears with the "debug" kernel parameter.
* The "lines suppressed due to ratelimiting" message is a regression in 243.0-1. But it's possible that it's simply that systemd is too chatty in the shutdown process and this triggers a ratelimiting message even though there is really no error condition.

Can someone else reproduce this behaviour as well?
Comment by Ales Katona (almindor) - Friday, 15 November 2019, 19:11 GMT
I'm getting same. One thing to note here is that I identified the "culprit" for failing the unmount in my case is RLS (rust language server, started by VS code) using lsof | grep ...
If I stop RLS myself and shutdown it works without errors.

It seems systemd is not following proper shutdown logic and isn't waiting for all processes to kill before trying the unmount.
Comment by Dennis Kadioglu (dkadioglu) - Wednesday, 11 December 2019, 14:45 GMT
I have the same problem - at least as far as I can see. Setting sudo sysctl -w kernel.printk_devkmsg=on -w kernel.printk_ratelimit=0 also made the ratelimiting message disappear. But still, I see the following messages:
sd-umount[1334]: Failed to unmount /oldroot: Device or resource busy
shutdown[1]: Failed to finalize file systems, ignoring.

I already asked upstream: and Lennart Poettering thinks, this must have something to do with Archs initrd.
I would like to investigate this further but, I don't know how. Debugging systemd during shutdown following lead to the following log:
At least I cannot identify anything problematic in the log.

As a sidenote, I don't have rust language server installed.
Comment by Leo P. (jpegxguy) - Friday, 13 December 2019, 11:07 GMT
No that you mention Arch's initrd, I think the message didn't happen for the short test time that I used dracut to generate the initramfs. Have you tried it?
Comment by Dennis Kadioglu (dkadioglu) - Friday, 13 December 2019, 16:13 GMT
I tried it with dracut now and the messages still appear. Thanks for that idea anyway.