FS#77789 - [systemd] updating resets keyboard layout / udev reload

Attached to Project: Arch Linux
Opened by Dawid Potocki (dawidpotocki) - Thursday, 09 March 2023, 09:46 GMT
Last edited by Christian Hesse (eworm) - Wednesday, 27 September 2023, 09:56 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Christian Hesse (eworm)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 27
Private No



I set my keyboard layout using `setxkbmap`, keyboard remaps using `xcape` and disable my trackpad using `xinput` in xinitrc; it all gets reset after a systemd update.

Got introduced by adding `/usr/bin/udevadm trigger` in this commit:

Steps to reproduce:

1. Update systemd
2. Cry while struggling to run the commands to fix your stuff
This task depends upon

Closed by  Christian Hesse (eworm)
Wednesday, 27 September 2023, 09:56 GMT
Reason for closing:  Fixed
Additional comments about closing:  systemd 254.4-3
Comment by Toolybird (Toolybird) - Thursday, 09 March 2023, 19:47 GMT
You don't reboot after updating systemd? There are some pkg updates where a reboot just seems naturally wise...kernel, glibc, systemd. The commit message doesn't explain the motivation for the change, but on face value it doesn't seem like much of a bug. Maybe you also need to "hook" your config changes somehow?
Comment by Christian Hesse (eworm) - Thursday, 09 March 2023, 20:20 GMT
Is there any reason you do not set the keyboard layout properly for the system, with `localectl`?
And the trackpad thing could be a udev rule...

Before the change we made udev reload the rules, but they were not applied. Now they are. That was important as udev from systemd 253 now creates device files that did not exist before, but are important for boot preparation and boot loader installation.
Comment by Dawid Potocki (dawidpotocki) - Thursday, 09 March 2023, 20:54 GMT
> Is there any reason you do not set the keyboard layout properly for the system, with `localectl`?
> And the trackpad thing could be a udev rule...

It allows me to dump it in my dotfiles without having to modify system files.

xcape still works
Comment by Michel Koss (MichelKoss1) - Saturday, 11 March 2023, 20:36 GMT
> You don't reboot after updating systemd? There are some pkg updates where a reboot just seems naturally wise...kernel, glibc, systemd

This hook runs not only when you update systemd but when anything that put files in /usr/lib/udev/rules.d/* updates.


There are dozens of such packages. See https://bugs.archlinux.org/task/77815 for another example.
Comment by Toolybird (Toolybird) - Saturday, 11 March 2023, 20:58 GMT
Dupe  FS#77815 
Comment by Tarqi Kazan (Tarqi) - Thursday, 23 March 2023, 20:46 GMT
Same here, all settings (keymap,key delay/rate, touchpad, autorandr, ...) are applied in .xinitrc which is used on multiple systems, so no explicit system settings are necessary and the config is portable. If there is another *portable* way, please let me know. Otherwise the whole xinit configuration using 'xset', 'setxkbmap', 'xinput set-prop ...' and so on would be broken and obsolete and the configuration much more complicated.
Comment by Gerry Kessler (renegat) - Tuesday, 04 April 2023, 14:57 GMT
I set the locale settings with localectl / xorg.d conf files and experience the same issue:
After everz rebuild of the initramfs the kezboard lazout is reset back to US

# localectl
System Locale: LANG=de_DE.UTF-8
VC Keymap: de
X11 Layout: de
X11 Model: pc105

# Written by systemd-localed(8), read by systemd-localed and Xorg. It's
# probably wise not to edit this file manually. Use localectl(1) to
# instruct systemd-localed to update it.
Section "InputClass"
Identifier "system-keyboard"
MatchIsKeyboard "on"
Option "XkbLayout" "de"
Option "XkbModel" "pc105"

Even 'loadkeys de-latin1' does not bring back german keyboard layout.
I|am not using .xinitrc to set locales!
Comment by Tarqi Kazan (Tarqi) - Tuesday, 04 April 2023, 15:19 GMT
@renegat: loadkeys is used for console only, try 'setxkbmap de'.
Comment by David Roth (V1del) - Wednesday, 05 April 2023, 11:30 GMT
There are quite a few weird behaviours possible from this in my case I always lose/reload audio devices for a split second and I've had a complete session crash once from kwin_wayland not being able to access the DRM device anymore/broken monitor setup from device names switching around.

@eworm do you have an example/commit ref where doing this is "necessary" for boot prep/bootloader config (... and by extension this will presumably only be relevant for systemd-boot?) It seems weird to me to trigger a logical reload of every single piece of HW just because something "might" have changed (... from presumably previously already working hardware). Leave that to - well - the actual boot or when the hardware actually changes due to being triggered/reconnected
Comment by Christian Hesse (eworm) - Wednesday, 05 April 2023, 12:02 GMT
IIRC this one required a rule reload (which we already did before), followed by a trigger:

The symlink was used and thus required in a userland tool shipped by systemd... Would have to look up the details myself.
Comment by Christian Hesse (eworm) - Wednesday, 05 April 2023, 12:27 GMT
I guess the complete session crash was caused by  FS#78097 ...
Comment by David Roth (V1del) - Wednesday, 05 April 2023, 13:13 GMT
That was a one time thing/potentially freak accident so I'd also not consider investigating too much there and I'm also hoping that got fixed by that (... though from looking at the logs there were definitely some issues with the DRM device/amdgpu), but just from a logic perspective it seems weird to me to retrigger everything (... FWIW another one I definitely had just yesterday is the following mkinitcpio initramfs rebuild not finding a certain device node -- which could lead to failure to boot from the non-fallback initramfs)

That last one might be some racyness that could be worked around with a bit of a sleep or so (or the --settle argument), though IMO the amount of potential side effects and breakage seems much higher than some userland tool crashing once for one specific systemd version/update/command.

Some more refs e.g. https://bbs.archlinux.org/viewtopic.php?id=284249 - this might be a plasma bug at the end of the day but could be avoided by not triggering udev reloads.
Comment by David Roth (V1del) - Thursday, 06 April 2023, 08:28 GMT
This definitely is also related to https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio/-/issues/177 and this currently can actively break initramfs generation for people. If you want to keep the udevadm trigger, then please add --settle as an argument so that the devices are populated when mkinitcpio runs. There are a bunch of threads on the boards already where this actively breaks the boot for people.

Due to all of the potential side effects I'm personally still voting for not doing the explicit trigger unconditionally anyway, but is technically something that can be up to debate. Ensuring the devices are ready for mkinitcpio to work isn't.

Comment by Michel Koss (MichelKoss1) - Thursday, 06 April 2023, 16:21 GMT
Looks like the '--settle' option breaks something else: https://bugs.archlinux.org/task/78134

I think the '/usr/bin/udevadm trigger' should be backed out. Requiring one time reboot after systemd 252 -> 253 update is minor issue in comparison with constant problem with updates and each day passed makes it even less relevant.
Comment by David Roth (V1del) - Wednesday, 03 May 2023, 17:25 GMT
Another case https://bbs.archlinux.org/viewtopic.php?id=285495 ... would we need to do a hard sleep due to udevadm settle being unreliable?
Comment by David Roth (V1del) - Wednesday, 10 May 2023, 16:02 GMT
For posterity's sake from the mkinitcpio gitlab:

I considered https://bugs.archlinux.org/task/77789 my personal go to for checking regarding this, but this really isn't just limited to session hangs or problems in mkinitcpio itself. While this has inadvertently uncovered that a bunch of people have old broken xorg evdev configs around that can get fixed by removing them, there are a bunch of other people where this leads to a variety of issues still. E.g.
to name a few.
Comment by Tom Englund (gulafaran) - Wednesday, 31 May 2023, 11:50 GMT
doesnt this technically break tlp/laptop mode tools aswell? you pull the cord on your laptop trigger an event and the scripts/tools changes the various device states and changes. while abroad you figure right i want neofetch. pacman -Syu neofetch , only this now brought in a package update that triggers the udev reload. and poof all your devices are now back in default state until another acpi event is triggered that you pulled the cord? . have not tested it myself but worth considering.
Comment by Alex (bidski) - Tuesday, 20 June 2023, 23:37 GMT
I also seem to be plagued by this issue. For me it manifests as a session logout (I get kicked out of my gnome session and returned to the gnome login screen). Depending on what packages are being installed this can leave me with no initramfs images and no vmlinuz-linux. I made a forum post here (https://bbs.archlinux.org/viewtopic.php?id=286657) about my issues.
Comment by Pavel Kolesnikov (pkolesnikov) - Friday, 14 July 2023, 05:02 GMT
Any news on this? Symptoms gets quite annoying and the fix seems straightforward.
Comment by Toolybird (Toolybird) - Thursday, 27 July 2023, 20:08 GMT
Dupe  FS#79219 
Comment by Tarqi Kazan (Tarqi) - Saturday, 09 September 2023, 22:19 GMT
This gets more and more annoying, while it _might_ be ok for keyboard/mouse settings, there are cases where it is not, see previous comments. I also noticed that a pacman update that triggers the hook always wakes up the disks on my *headless* server, which should not happen. Please: Find a solution. Thanks.
Comment by Michel Koss (MichelKoss1) - Sunday, 17 September 2023, 12:53 GMT
The solution is known from the beginning: revert the change that causes this issue, linked in bug description. It was needed once on systemd 252->253 update and now it's perfectly safe to do drop it.
Comment by T.J. Townsend (blakkheim) - Sunday, 24 September 2023, 20:40 GMT
This resetting my mouse button settings and cursor speed has been driving me nuts forever.
Comment by Christian Hesse (eworm) - Tuesday, 26 September 2023, 20:32 GMT
The systemd upstream expects us (and any other distribution) to reload udev rules on package updates by running `udevadm trigger`. Note adding this was not just a one-time fix, but could be required again any time in the future. Thus I do not intend to just revert the change.
(The issue is not systemd / udevd handling this wrong. This issue is device drivers not handling the CHANGE event properly. So if you are effected - please report upstream!)

However I do understand that the behavior is annoying for those being hit. On the other hand I - as expressed before - do not intend to revert the default behavior. How about an option to opt-out from reloading udev rules on updates? This is what I came up with:


It introduces a new configuration file (/etc/systemd/pacman.conf) that is read by the pacman / libalpm hook. Adding `UdevadmTriggerAndSettle = false` will make the hook skip the step reloading the rules.

Any comments? Concerns? Suggestions?
Comment by Toolybird (Toolybird) - Tuesday, 26 September 2023, 23:08 GMT
> The systemd upstream expects us

For clarity, it seems to be documented here [1]

[1] https://github.com/systemd/systemd-stable/blob/v254.4/NEWS#L4631-L4656
Comment by Christian Hesse (eworm) - Wednesday, 27 September 2023, 06:56 GMT
BTW, this has been in NEWS for v247 back in November 2020:

Comment by Lone_Wolf (Lone_Wolf) - Wednesday, 27 September 2023, 09:03 GMT
So it's an upstream decision.

I'm one of those users that edited the service file and put it in noupgrade. The opt-out solution is an improvement over that method.

The best solution in my opinion though would be if pacman warned the user of the intrusive nature of the update and gave the option to abort it.

Comment by Christian Hesse (eworm) - Wednesday, 27 September 2023, 09:56 GMT
Ok, some recap...

The upstream news had an issue, where the wrong option was mentioned for the command. Instead of `udevadm trigger -a change` this should have read `udevadm trigger -c change` to match the action. I have just fixed that:


... and also put the correct and complete command into the hook:


Wondering if that fixes it, at least for some of the cases. So please try without workaround first!

For sure there are still cases where this does cause issues, see the link from previous comment for details. So I added a workaround to opt-out from reloading udev rules. Use at your own risk!