FS#44572 - [systemd / initcpio] udevadm trigger is not guarantee to have finished before udevadm control --exit

Attached to Project: Arch Linux
Opened by Tom Yan (tom.ty89) - Monday, 13 April 2015, 16:07 GMT
Last edited by Dave Reisner (falconindy) - Friday, 24 April 2015, 00:08 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Thomas Bächler (brain0)
Dave Reisner (falconindy)
Tom Gundersen (tomegun)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:

With certain hardware or default attribute, we can see that `udevadm trigger` in the udev runtime hook (or installed service of systemd hook) is not guarantee to have finished its job before `udevadm control --exit`.

A udev rule is therefore not certain to get applied. See the attachments for test cases:
1. norule -- default attribute without any udev rule set
2. without_sleep -- a udev rule set but required a manual udevadm trigger to get it actually apply (which shows that the rule itself is correct)
3. with_sleep -- after adding `sleep 1` at any place between `udevadm trigger --add=action --type devices` in run_hook() and `udevadm control --exit` in run_cleanuphook() in /lib/initcpio/hook/udev (or at the end of ExecStart in systemd-udev-trigger.service), rebuild initramfs and reboot, the rule get apply as expected automatically.

Additional info:
* package version(s): systemd 218 (systemd-git is also tested not long ago before this report)
This task depends upon

Closed by  Dave Reisner (falconindy)
Friday, 24 April 2015, 00:08 GMT
Reason for closing:  Upstream
Comment by Tom Yan (tom.ty89) - Monday, 13 April 2015, 20:09 GMT
According to the man page of udevadm, `udevadm control --exit` should "signal and wait for systemd-udevd to exit", but with the two scripts attached, I can confirm that `udevadm control --exit` will kill systemd-udevd before the udev rule get applied. So should I file a bug report upstream?

Yet there is another problem, what about the systemd hook? How does the "early" systemd-udevd exit? Does it count on `udevadm control --exit` anyway?

P.S. `udevadm settle` is basically a no-op in this issue, though in my case REMOVING it from the udev hook somehow introduce some extra time for the trigger command to finish its job, which confused me at the beginning
Comment by Tom Yan (tom.ty89) - Tuesday, 14 April 2015, 10:05 GMT
So I try to "simulate" the systemd hook case, see the attachments for details (it's probably not exactly the same as what happen in the hook, but at least it shows that there can be a "race condition" or similar)

P.S. The settle service is neither installed by the systemd hook nor used in default systemd environment, though it is not a fail-safe solution anyway.
Comment by Dave Reisner (falconindy) - Tuesday, 14 April 2015, 12:56 GMT
> Certain hardware
What hardware?

> A udev rule
What rule?

> not guarantee to have finished its job before `udevadm control --exit`.
Why is it important that this finishes before exiting? What is lost? Is this corrected when uevents are replayed on the real root?

You haven't actually explained what the failure mode is here...
Comment by Tom Yan (tom.ty89) - Tuesday, 14 April 2015, 14:22 GMT
> What hardware?
An arbitrary mouse/keyboard USB receiver. My point isn't about the hardware, it's about "my hardware made me discovered this tricky race-condition-ish problem"

> What rule?
If you have read my test cases, you would have seen. But it might not matter anyway.

> Why is it important that this finishes before exiting?
I don't know, should things exit at random points with their jobs half finished?

> What is lost?
The udev rule I written does not get applied, not if I run `udevadm trigger` manually after boot.

> Is this corrected when uevents are replayed on the real root?
It doesn't seem to be corrected in the real root. Otherwise I don't need the run the command myself.

> You haven't actually explained what the failure mode is here...
I think the "specific failure mode here is:
Without a command like `sleep 1` after `udevadm trigger --add=action --type=devices` in the udev hook or service installed by the systemd hook, a udev rule might not get apply in boot.

But in general my question/concern:
Should
1. `udevadm control --exit`
2. `systemd stop systemd-udevd`
3. whatever kills systemd-udevd when the systemd hook is used
get systemd-udevd killed/exited before it finish its jobs (assuming applying a udev rule is one of them)?

Anyway I think I can tell now it's an upstream matter anyway. Though it would be nice if you can confirm this.
Comment by Tom Yan (tom.ty89) - Wednesday, 15 April 2015, 08:09 GMT
EDIT: Okay this is so weird now. Here's what I got from a fresh boot:

[tom@localhost ~]$ sudo journalctl -b | grep wakeup
Apr 15 16:54:03 localhost kernel: pci 0000:00:01.0: System wakeup disabled by ACPI
Apr 15 16:54:03 localhost kernel: pci 0000:00:14.0: System wakeup disabled by ACPI
Apr 15 16:54:03 localhost kernel: pci 0000:00:1a.0: System wakeup disabled by ACPI
Apr 15 16:54:03 localhost kernel: pci 0000:00:1c.0: System wakeup disabled by ACPI
Apr 15 16:54:03 localhost kernel: pci 0000:00:1c.2: System wakeup disabled by ACPI
Apr 15 16:54:03 localhost kernel: pci 0000:00:1c.3: System wakeup disabled by ACPI
Apr 15 16:54:03 localhost kernel: pci 0000:00:1c.4: System wakeup disabled by ACPI
Apr 15 16:54:03 localhost kernel: pci 0000:00:1d.0: System wakeup disabled by ACPI
Apr 15 16:54:03 localhost kernel: pci 0000:01:00.0: System wakeup disabled by ACPI
Apr 15 16:54:03 localhost kernel: pci 0000:03:00.0: System wakeup disabled by ACPI
Apr 15 16:54:03 localhost kernel: pci 0000:04:00.0: System wakeup disabled by ACPI
Apr 15 16:54:03 localhost kernel: pci 0000:06:00.0: System wakeup disabled by ACPI
Apr 15 16:54:03 localhost systemd-udevd[205]: Reading rules file: /etc/udev/rules.d/wakeup.rules
Apr 15 16:54:04 localhost systemd-udevd[214]: ATTR '/sys/devices/pci0000:00/0000:00:14.0/usb3/3-13/power/wakeup' writing 'enabled' /etc/udev/rules.d/wakeup.rules:1
Apr 15 16:54:04 localhost systemd-udevd[215]: ATTR '/sys/devices/pci0000:00/0000:00:14.0/usb3/3-14/power/wakeup' writing 'disabled' /etc/udev/rules.d/wakeup.rules:2
[tom@localhost ~]$ cat /sys/bus/usb/devices/3-13/power/wakeup
enabled
[tom@localhost ~]$ cat /sys/bus/usb/devices/3-14/power/wakeup
enabled
(The orignal/default attribute (without any related udev rules) of 3-13 is disabled, 3-14 is enabled.)

But the thing is, what happened in the early user space or early udevd (e.g. the `sleep 1` in udev hook) did have actual impact on the final results.

P.S. I have to change the order of the two commands in systemd-udev-trigger.service in order to get the ATTR lines logged.
Comment by Tom Yan (tom.ty89) - Wednesday, 15 April 2015, 11:01 GMT
With the attached modified systemd-udev-trigger.service in the real root, I can capture something interesting about the effect of the early user space or early udevd on what happen later.

Case 1: Without `sleep 1` in the udev hook (original)
Pre-trigger:
3-13: disabled
3-14: disabled
Post-trigger:
3-13: enabled
3-14: enabled

Case 2: With `sleep 1` in the udev hook
Pre-trigger:
3-13: disabled
3-14: enabled
(as what should be without the udev rules)
Post-trigger:
3-13: enabled
3-14: disabled
(as what should be with the udev rules applied)

So it seems to me that if the device isn't "initialized" in early user space properly, the device might got "re-initialized" later, which override the applied udev rules (or it stop the rules from actually applied despite the ATTR message in the journal?)
Comment by Tom Yan (tom.ty89) - Tuesday, 21 April 2015, 10:27 GMT
The issue is basically a side effect of two upstream issue. First udevd applies a udev rule after a kernel module which has a hard-coded default is loaded: https://bugs.freedesktop.org/show_bug.cgi?id=90041; Second udevd has no measure to avoid exits before jobs are done: https://bugs.freedesktop.org/show_bug.cgi?id=90051. So this bug report can be closed.

Loading...