Arch Linux

You don't reboot after updating systemd? There are some pkg updates where a reboot just seems naturally wise...kernel, glibc, systemd. The commit message doesn't explain the motivation for the change, but on face value it doesn't seem like much of a bug. Maybe you also need to "hook" your config changes somehow?

Is there any reason you do not set the keyboard layout properly for the system, with `localectl`?
And the trackpad thing could be a udev rule...

Before the change we made udev reload the rules, but they were not applied. Now they are. That was important as udev from systemd 253 now creates device files that did not exist before, but are important for boot preparation and boot loader installation.

> Is there any reason you do not set the keyboard layout properly for the system, with `localectl`?
> And the trackpad thing could be a udev rule...

It allows me to dump it in my dotfiles without having to modify system files.

Correction:
xcape still works

> You don't reboot after updating systemd? There are some pkg updates where a reboot just seems naturally wise...kernel, glibc, systemd

This hook runs not only when you update systemd but when anything that put files in /usr/lib/udev/rules.d/* updates.

https://github.com/archlinux/svntogit-packages/blob/packages/systemd/trunk/30-systemd-udev-reload.hook#L6

There are dozens of such packages. See https://bugs.archlinux.org/task/77815 for another example.

Dupe  FS#77815 

Same here, all settings (keymap,key delay/rate, touchpad, autorandr, ...) are applied in .xinitrc which is used on multiple systems, so no explicit system settings are necessary and the config is portable. If there is another *portable* way, please let me know. Otherwise the whole xinit configuration using 'xset', 'setxkbmap', 'xinput set-prop ...' and so on would be broken and obsolete and the configuration much more complicated.

I set the locale settings with localectl / xorg.d conf files and experience the same issue:
After everz rebuild of the initramfs the kezboard lazout is reset back to US

# localectl
System Locale: LANG=de_DE.UTF-8
VC Keymap: de
X11 Layout: de
X11 Model: pc105

00-keyboard.conf
# Written by systemd-localed(8), read by systemd-localed and Xorg. It's
# probably wise not to edit this file manually. Use localectl(1) to
# instruct systemd-localed to update it.
Section "InputClass"
Identifier "system-keyboard"
MatchIsKeyboard "on"
Option "XkbLayout" "de"
Option "XkbModel" "pc105"
EndSection


Even 'loadkeys de-latin1' does not bring back german keyboard layout.
I|am not using .xinitrc to set locales!

@renegat: loadkeys is used for console only, try 'setxkbmap de'.

There are quite a few weird behaviours possible from this in my case I always lose/reload audio devices for a split second and I've had a complete session crash once from kwin_wayland not being able to access the DRM device anymore/broken monitor setup from device names switching around.

@eworm do you have an example/commit ref where doing this is "necessary" for boot prep/bootloader config (... and by extension this will presumably only be relevant for systemd-boot?) It seems weird to me to trigger a logical reload of every single piece of HW just because something "might" have changed (... from presumably previously already working hardware). Leave that to - well - the actual boot or when the hardware actually changes due to being triggered/reconnected

IIRC this one required a rule reload (which we already did before), followed by a trigger:
https://github.com/systemd/systemd/commit/583dc6d933d8eb84f338c28cbe2f4c86ee1ff069

The symlink was used and thus required in a userland tool shipped by systemd... Would have to look up the details myself.

I guess the complete session crash was caused by  FS#78097 ...

That was a one time thing/potentially freak accident so I'd also not consider investigating too much there and I'm also hoping that got fixed by that (... though from looking at the logs there were definitely some issues with the DRM device/amdgpu), but just from a logic perspective it seems weird to me to retrigger everything (... FWIW another one I definitely had just yesterday is the following mkinitcpio initramfs rebuild not finding a certain device node -- which could lead to failure to boot from the non-fallback initramfs)

That last one might be some racyness that could be worked around with a bit of a sleep or so (or the --settle argument), though IMO the amount of potential side effects and breakage seems much higher than some userland tool crashing once for one specific systemd version/update/command.

Some more refs e.g. https://bbs.archlinux.org/viewtopic.php?id=284249 - this might be a plasma bug at the end of the day but could be avoided by not triggering udev reloads.

This definitely is also related to https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio/-/issues/177 and this currently can actively break initramfs generation for people. If you want to keep the udevadm trigger, then please add --settle as an argument so that the devices are populated when mkinitcpio runs. There are a bunch of threads on the boards already where this actively breaks the boot for people.

Due to all of the potential side effects I'm personally still voting for not doing the explicit trigger unconditionally anyway, but is technically something that can be up to debate. Ensuring the devices are ready for mkinitcpio to work isn't.

Looks like the '--settle' option breaks something else: https://bugs.archlinux.org/task/78134

I think the '/usr/bin/udevadm trigger' should be backed out. Requiring one time reboot after systemd 252 -> 253 update is minor issue in comparison with constant problem with updates and each day passed makes it even less relevant.

Another case https://bbs.archlinux.org/viewtopic.php?id=285495 ... would we need to do a hard sleep due to udevadm settle being unreliable?

For posterity's sake from the mkinitcpio gitlab:

I considered https://bugs.archlinux.org/task/77789 my personal go to for checking regarding this, but this really isn't just limited to session hangs or problems in mkinitcpio itself. While this has inadvertently uncovered that a bunch of people have old broken xorg evdev configs around that can get fixed by removing them, there are a bunch of other people where this leads to a variety of issues still. E.g.
https://bbs.archlinux.org/viewtopic.php?pid=2098380#p2098380
https://bbs.archlinux.org/viewtopic.php?id=285369
https://bbs.archlinux.org/viewtopic.php?id=285241
to name a few.

doesnt this technically break tlp/laptop mode tools aswell? you pull the cord on your laptop trigger an event and the scripts/tools changes the various device states and changes. while abroad you figure right i want neofetch. pacman -Syu neofetch , only this now brought in a package update that triggers the udev reload. and poof all your devices are now back in default state until another acpi event is triggered that you pulled the cord? . have not tested it myself but worth considering.

I also seem to be plagued by this issue. For me it manifests as a session logout (I get kicked out of my gnome session and returned to the gnome login screen). Depending on what packages are being installed this can leave me with no initramfs images and no vmlinuz-linux. I made a forum post here (https://bbs.archlinux.org/viewtopic.php?id=286657) about my issues.

@eworm
Any news on this? Symptoms gets quite annoying and the fix seems straightforward.

Dupe  FS#79219 

This gets more and more annoying, while it _might_ be ok for keyboard/mouse settings, there are cases where it is not, see previous comments. I also noticed that a pacman update that triggers the hook always wakes up the disks on my *headless* server, which should not happen. Please: Find a solution. Thanks.

The solution is known from the beginning: revert the change that causes this issue, linked in bug description. It was needed once on systemd 252->253 update and now it's perfectly safe to do drop it.

This resetting my mouse button settings and cursor speed has been driving me nuts forever.

The systemd upstream expects us (and any other distribution) to reload udev rules on package updates by running `udevadm trigger`. Note adding this was not just a one-time fix, but could be required again any time in the future. Thus I do not intend to just revert the change.
(The issue is not systemd / udevd handling this wrong. This issue is device drivers not handling the CHANGE event properly. So if you are effected - please report upstream!)

However I do understand that the behavior is annoying for those being hit. On the other hand I - as expressed before - do not intend to revert the default behavior. How about an option to opt-out from reloading udev rules on updates? This is what I came up with:

https://gitlab.archlinux.org/archlinux/packaging/packages/systemd/-/commits/udev-rules/

It introduces a new configuration file (/etc/systemd/pacman.conf) that is read by the pacman / libalpm hook. Adding `UdevadmTriggerAndSettle = false` will make the hook skip the step reloading the rules.

Any comments? Concerns? Suggestions?

> The systemd upstream expects us

For clarity, it seems to be documented here [1]

[1] https://github.com/systemd/systemd-stable/blob/v254.4/NEWS#L4631-L4656

BTW, this has been in NEWS for v247 back in November 2020:

https://github.com/systemd/systemd-stable/blob/v247/NEWS#L89-L114

So it's an upstream decision.

I'm one of those users that edited the service file and put it in noupgrade. The opt-out solution is an improvement over that method.

The best solution in my opinion though would be if pacman warned the user of the intrusive nature of the update and gave the option to abort it.

Ok, some recap...

The upstream news had an issue, where the wrong option was mentioned for the command. Instead of `udevadm trigger -a change` this should have read `udevadm trigger -c change` to match the action. I have just fixed that:

https://github.com/systemd/systemd/commit/75271582bead7e65c20fa3ca4fb729a5c9e261d0

... and also put the correct and complete command into the hook:

https://gitlab.archlinux.org/archlinux/packaging/packages/systemd/-/commit/368be7309b1102481f8302f59cccb7022f2ec322

Wondering if that fixes it, at least for some of the cases. So please try without workaround first!

For sure there are still cases where this does cause issues, see the link from previous comment for details. So I added a workaround to opt-out from reloading udev rules. Use at your own risk!

https://gitlab.archlinux.org/archlinux/packaging/packages/systemd/-/commit/17aae5e870a770426ab303b6e4d38e97e8b3baff