FS#16001 - [udev] udevd uses excessive cpu and spawns too many process on boot-up

Attached to Project: Arch Linux
Opened by Jay Tanzman (jt512) - Saturday, 05 September 2009, 00:47 GMT
Last edited by Angel Velasquez (angvp) - Monday, 05 July 2010, 21:46 GMT
Task Type Bug Report
Category Upstream Bugs
Status Closed
Assigned To Tobias Powalowski (tpowa)
Thomas Bächler (brain0)
Architecture x86_64
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 16
Private No

Details

Description:
Upon upgrading to 145-1, udevd spawns a second process and cpu usages goes up to around 25% (normal is about 1% on my laptop). After reboot, udevd spawns about 20 processes, and cpu usage is excessive.

Additional info:
* udev-145-1

Steps to reproduce:
1. Upgrade udev to 145-1
2. `ps -ef | grep udev` will show 2 udevd process, and cpu usage will be excessive
3. Reboot
4. `ps -ef | grep udev will show many udevd processes, and cpu usage will be excessive
This task depends upon

Closed by  Angel Velasquez (angvp)
Monday, 05 July 2010, 21:46 GMT
Reason for closing:  Fixed
Additional comments about closing:  re-open in case it happens again
Comment by Roman Kyrylych (Romashka) - Saturday, 05 September 2009, 09:07 GMT
Do you have the latest initscripts (2009.08-1)?
Status with udev 146 from testing?
Comment by Roman Kyrylych (Romashka) - Saturday, 05 September 2009, 09:09 GMT
Hm, I have only 3 udevd processes running and CPU usage is low.
Could you please list the contents of /etc/udev/rules.d ?
Maybe this is caused by some broken rules processing :-/
Comment by Jay Tanzman (jt512) - Saturday, 05 September 2009, 16:12 GMT
I have the latest initscripts-2009.08-1, and the problem persists in udev-146-1 from testing.
ls /etc/udev/rules.d:
10-usb.rules
10-usb.rules~
53-sane.rules~
54-gphoto.rules
60-pcmcia.rules
75-cd-aliases-generator.rules.optional
75-persistent-net-generator.rules.optional
90-hal.rules
97-bluetooth-serial.rules
99-fuse.rules
device-mapper.rules

Do you want the contents of these files, too?
Comment by Roman Kyrylych (Romashka) - Saturday, 05 September 2009, 17:52 GMT
@Jay: from the logfile that you sent me it looks like it has something to do with your CD burner.

@Tobias: see http://nopaste.com/p/a9h826y0E - the logfile is flooded with this.
Comment by Andrew (abrouwers) - Sunday, 06 September 2009, 03:13 GMT
On a fresh install, udev is spawning 3 processes here as well; I am pretty sure it should only be spawning 1. This is a fresh install of arch, with no outdated configuration files / rules.
Comment by Roman Kyrylych (Romashka) - Sunday, 06 September 2009, 11:32 GMT
I don't know how many processes it should spawn, but anyway CPU usage is fine here,
while on your system udev is flooded with events from sr0, don't know why
Comment by Tobias Powalowski (tpowa) - Sunday, 06 September 2009, 12:20 GMT
Hrm i have also 3 processes here but no spamming of the cpu at all.
Comment by Tobias Powalowski (tpowa) - Sunday, 06 September 2009, 12:21 GMT
Have you tried udev-146 from testing?
Comment by thibault (inflames) - Sunday, 06 September 2009, 15:48 GMT
Hi, similar problem here, I'm on x64. Two udevd instance eating a lot of cpu after boot. I've no manually aded udev rules :

ls /etc/udev/rules.d/
10-vboxdrv.rules 54-gphoto.rules 75-cd-aliases-generator.rules.optional 90-hal.rules 99-fuse.rules
45-libnjb.rules 60-pcmcia.rules 75-persistent-net-generator.rules.optional 97-bluetooth-serial.rules device-mapper.rules

but at boot, I have to wait more than 3min to get uevent processed...
udevadm monitor is flooded with tons of :
KERNEL[1252250981.606281] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0/block/sr0 (block)
UDEV [1252250981.606560] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0 (scsi)

The problem still exactly the same with udev 146 from testing.
Everything else is up to date on my system.
Comment by Tobias Powalowski (tpowa) - Sunday, 06 September 2009, 15:51 GMT
do you have a cdrom in your optical drive?
Comment by thibault (inflames) - Sunday, 06 September 2009, 15:53 GMT
No, nothing inside my optical drive.
Comment by Tobias Powalowski (tpowa) - Sunday, 06 September 2009, 16:00 GMT
dbus and hal are running?
Comment by thibault (inflames) - Sunday, 06 September 2009, 17:39 GMT
Yes everything is working as usual, except for those annoying bugs.
Comment by thibault (inflames) - Sunday, 06 September 2009, 19:09 GMT
This could be interesting, with the working version of udev (143) udevadm monitor return me :
KERNEL[1252263519.016398] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0 (scsi)
KERNEL[1252263519.017157] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0/block/sr0 (block)
UDEV [1252263519.017403] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0 (scsi)
UDEV [1252263519.119416] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0/block/sr0 (block)

BUT one packet of this size by second, no more. With udev 145 and 146 I'm getting a lot of them by second.
Comment by Jay Tanzman (jt512) - Wednesday, 09 September 2009, 17:32 GMT
This bug persists in udev-146-2 (on x86_64).
Comment by Vladimir (elstop) - Monday, 21 September 2009, 05:05 GMT
Confirm! udev versions 145-1,146-2 on my laptop (i686) heavily loads the CPU.
In the version of udev-141 is no problem about this.
"udevadm monitor" gives the following:

KERNEL[1253500790.024037] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0 (scsi)
KERNEL[1253500790.025426] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0/block/sr0 (block)
UDEV [1253500790.025977] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0 (scsi)
UDEV [1253500790.220099] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0/block/sr0 (block)
KERNEL[1253500792.045671] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0 (scsi)
KERNEL[1253500792.046727] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0/block/sr0 (block)
UDEV [1253500792.047275] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0 (scsi)
UDEV [1253500792.203610] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0/block/sr0 (block)

In version 141-1 aproximetlly 1-2 times per second and stops after "hal-disable-polling - device / dev / cdrom",
but versions of the 145-1 and 146-2 are not ceasing and "hal-disable-polling -- device / dev / cdrom "does not stop it.
Comment by Thomas Bächler (brain0) - Monday, 21 September 2009, 22:28 GMT
Can you provide the output of udevadm with --property (--env on older udev versions)?
Comment by Vladimir (elstop) - Monday, 21 September 2009, 23:19 GMT
For the purity of the experiment, again I installed version udev-146-2.
The file output
udevadm monitor --property

What you need more?
thanks
Comment by thibault (inflames) - Monday, 21 September 2009, 23:26 GMT
My optical drive is exaxctly the same!
I'm getting very similar msg with the working version of udev :
UDEV [1253572423.115047] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0/block/sr0 (block)
UDEV_LOG=2
ACTION=change
DEVPATH=/devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0/block/sr0
SUBSYSTEM=block
DEVTYPE=disk
SEQNUM=96906
ID_CDROM=1
ID_CDROM_CD_R=1
ID_CDROM_CD_RW=1
ID_CDROM_DVD=1
ID_CDROM_DVD_R=1
ID_CDROM_DVD_RW=1
ID_CDROM_DVD_RAM=1
ID_CDROM_DVD_PLUS_R=1
ID_CDROM_DVD_PLUS_RW=1
ID_CDROM_DVD_PLUS_R_DL=1
ID_CDROM_MRW=1
ID_CDROM_MRW_W=1
ID_VENDOR=Optiarc
ID_VENDOR_ENC=Optiarc\x20
ID_MODEL=DVD_RW_AD-7560S
ID_MODEL_ENC=DVD\x20RW\x20AD-7560S\x20
ID_REVISION=S801
ID_TYPE=cd
ID_BUS=scsi
ID_PATH=pci-0000:00:1f.2-scsi-1:0:0:0
DEVNAME=/dev/sr0
MAJOR=11
MINOR=0
DEVLINKS=/dev/block/11:0 /dev/scd0 /dev/disk/by-path/pci-0000:00:1f.2-scsi-1:0:0:0 /dev/cd/cdrom-1:0:0:0 /dev/cd/cdrw-1:0:0:0 /dev/cd/dvd-1:0:0:0 /dev/cd/dvdrw-1:0:0:0

KERNEL[1253572425.014430] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0 (scsi)
UDEV_LOG=2
ACTION=change
DEVPATH=/devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0
SUBSYSTEM=scsi
SDEV_MEDIA_CHANGE=1
DEVTYPE=scsi_device
DRIVER=sr
MODALIAS=scsi:t-0x05
SEQNUM=96907

KERNEL[1253572425.014861] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0/block/sr0 (block)
UDEV_LOG=2
ACTION=change
DEVPATH=/devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0/block/sr0
SUBSYSTEM=block
DEVTYPE=disk
SEQNUM=96908
MAJOR=11
MINOR=0

UDEV [1253572425.015228] change /devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0 (scsi)
UDEV_LOG=2
ACTION=change
DEVPATH=/devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0
SUBSYSTEM=scsi
SDEV_MEDIA_CHANGE=1
DEVTYPE=scsi_device
DRIVER=sr
MODALIAS=scsi:t-0x05
SEQNUM=96907
Comment by Arvid Picciani (aep) - Thursday, 24 September 2009, 23:03 GMT
problem ocures for me with no rules whatsoever in rules.d with udev 146-2
udevadm monitor doesnt show anything at all.
Comment by Arvid Picciani (aep) - Thursday, 24 September 2009, 23:08 GMT
btw this is a xen dom. it shouldnt have anything udev cares about anyway. i killed it in rc.local for now
Comment by Vladimir (elstop) - Friday, 25 September 2009, 00:13 GMT
It's more as old crutches than on new legs :)
Comment by Kai Kaminski (kok) - Friday, 02 October 2009, 20:55 GMT
I seem to have the same problem. There are three udevd instances, one of which is using about 15-30% CPU. I've attached 'dmesg' and 'udevadm monitor' output.
Comment by runiq (runiq) - Sunday, 04 October 2009, 09:40 GMT
I also have this problem, my udevadm monitor --property output is the same as Vladimir's and thibault's (ID_MODEL says DVD_RW_AD-7640S instead of DVD_RW_AD-7560S, though this shouldn't matter). Udev version 146-1, system's updated.

I've found an entry on the kernel bug tracker about this issue:

http://bugzilla.kernel.org/show_bug.cgi?id=13783
Comment by runiq (runiq) - Sunday, 04 October 2009, 22:36 GMT
(double post)
Comment by Mukul (mukul_s) - Sunday, 25 October 2009, 09:19 GMT
facing the same problem in udev 146-2 and Kernel 2.6.31-5. Both are installed via full system upgrade.
Comment by Philipp B. (TamCore) - Sunday, 25 October 2009, 16:25 GMT
I think its fixed in udev 146-2 and Kernel 2.6.31.5-1. I have 2-5 running udevd instances with a total of 0.2% cpu usage.
Comment by Kamil Sałaś (Kamil) - Sunday, 25 October 2009, 17:02 GMT
I tried udev 146-2 and Kernel 2.6.31.5-1 and the problem is not fixed on my computer.
Comment by Vladimir (elstop) - Sunday, 25 October 2009, 17:45 GMT
I also changed nothing. Had to make the package udev-146-2 from the contents of the package udev-141 to satisfy dependencies and have a stable job.
Comment by Vladimir (elstop) - Sunday, 25 October 2009, 20:25 GMT
I also changed nothing. Had to make the package udev-146-2 from the contents of the package udev-141 to satisfy dependencies and have a stable job.
Comment by bv (voltam.bence) - Tuesday, 10 November 2009, 02:55 GMT
I think there is two different problems with udev (at least for me), but it's possible that they are related.

1. "udevd uses excessive cpu and spawns too many process on boot-up"
I don't know, if is it a problem, but I have 116 udevd processes. They eat ~25% of my cpu in idle state.
workaround: add the --debug-trace option to udevd (in rc.sysinit for example); it will slow down the init procedure but you get only ~3 udevd instance

2. "udevd still uses (less) excessive cpu without spawning too many..."
After eliminating the first issue I have still ~6% cpu usage (by udevd) in idle state.
workaround: blacklist the module sr_mod (and forget using it :)
Comment by Vladimir (elstop) - Tuesday, 10 November 2009, 05:12 GMT
I propose a third way not to use these crutches
http://aur.archlinux.org/packages.php?ID=31373
Comment by Thomas Bächler (brain0) - Tuesday, 10 November 2009, 09:30 GMT
Hm, it seems the upstream bug report has fallen asleep, and I don't have any new ideas as well. I also cannot reproduce it, udevd behaves sane here on all systems. The only thing people seem to agree about on bugzilla.kernel is that this is related to media change detection on some drives.
Comment by Gerardo Exequiel Pozzi (djgera) - Tuesday, 26 January 2010, 06:44 GMT
  • Field changed: Status (Assigned → Waiting on Response)
status of this with latest udev-150-3? Lots of fixes are made between 145 and 150 in these months.
Comment by Philipp B. (TamCore) - Tuesday, 26 January 2010, 07:44 GMT
It's a little bit better. I have a DualCore and udev took in the stable version 50% of each cpu. Now it's 30-40% of each cpu.
Comment by Gerardo Exequiel Pozzi (djgera) - Tuesday, 26 January 2010, 08:15 GMT
OK, good. Anyway you can limit the numbers of childs that udevd raise with (maybe help)

udevadm control --max-childs=1
Comment by Matthias Schwarz (craw) - Tuesday, 02 February 2010, 22:14 GMT
persisting with udev 151-2.
Comment by Gerardo Exequiel Pozzi (djgera) - Tuesday, 02 February 2010, 22:29 GMT
@craw: please try adding udevadm control --max-childs=1, just after udevd is loaded.
Comment by Matthias Schwarz (craw) - Wednesday, 03 February 2010, 08:30 GMT
With udev-ubuntu from AUR those media change events disappeared for me, also the overall number of udevd processes lowered from ~100 to 6.
The events also stop appearing when using 'udevadm control --max-childs=1' ? When put in the udev hook of initcpio?
Comment by Thomas Bächler (brain0) - Wednesday, 03 February 2010, 08:40 GMT
@craw: This has nothing to do with mkinitcpio at all.
Comment by Matthias Schwarz (craw) - Wednesday, 03 February 2010, 09:09 GMT
i simply keep messing things up. sorry.

@brain0 thank you for pointing that out.
Comment by Thomas Bächler (brain0) - Wednesday, 03 February 2010, 09:29 GMT
Another idea here: These media change events come from the kernel, so a different udev could not stop them, but only ignore them. Have you tried introducing OPTIONS="ignore" rules for media change events to make this stop?

The underlying problem must still be in the kernel, not udev itself.
Comment by Philipp B. (TamCore) - Wednesday, 03 February 2010, 11:09 GMT
I blacklisted the sr_mod module and now everything is fine. If i need my dvd drive, i have to unblacklist the module and reboot.

I agree with brain0, it's a kernel problem.
Comment by Matthias Schwarz (craw) - Wednesday, 03 February 2010, 12:02 GMT
So, to get this straight,
shouldn't 'udevadm monitor' show the events signalled by the kernel even if they are ignored by an udev rule?
Because when using udev-ubuntu 'udevadm' does not output any of those media change events here.
Comment by Thomas Bächler (brain0) - Wednesday, 03 February 2010, 12:35 GMT
@craw: Yes, I suppose they should. Maybe the modified udev you use deactivates those events? Or our udev explicitly activates them?
Comment by Jay Tanzman (jt512) - Wednesday, 03 February 2010, 16:29 GMT
When I tried udev-ubuntu, it didn't seem to recognize the optical drive at all. I have a rather involved kludge to work around the problem with the current udev in post #32 here: http://bbs.archlinux.org/viewtopic.php?id=78946 .
Comment by Martin Zecher (MartinZ) - Wednesday, 10 March 2010, 23:20 GMT
As pointed by astralys, the problem may be easily solved upgrading the firmware of the DVD drive. For the Optiarc AD-7560S, it may be downloaded from here: http://ftp.euro.dell.com/rmsd/AD-7560S%20SD05.zip
Comment by Matthias Schwarz (craw) - Thursday, 11 March 2010, 11:53 GMT
Would try it if Lenovo would offer a firmware update for Optiarc AD-7580S. Mailed to customer support...
Comment by thibault (inflames) - Thursday, 11 March 2010, 12:47 GMT
Worked perfectly for me on my Lenovo but my optical drive is a AD-7560S.
Comment by Kamil Sałaś (Kamil) - Saturday, 13 March 2010, 13:23 GMT
@MartinZ: Ok, but before I update my firmware I should dump the old one. How to do it?

@craw: There is also a firmware for AD-7580S from dell: http://ftp.euro.dell.com/rmsd/AD-7580S%20FD06.zip
Comment by Martin Zecher (MartinZ) - Saturday, 13 March 2010, 16:09 GMT
I really didn't care about backing it up
Comment by Matthias Schwarz (craw) - Saturday, 13 March 2010, 19:26 GMT
I also found that in the meantime, but since it seems that there is no easy solution to flashing/backing up the firmware from linux, i'll just wait for specific instructions. I'd be great if someone could post a little howto on the wiki/forums or via mail how one could accomplish this from GNU/Linux only.
Comment by Kamil Sałaś (Kamil) - Saturday, 13 March 2010, 19:39 GMT
@MartinZ: My laptop vendor is Lenovo, this firmware is from Dell. I'm a bit worried because of this.
Comment by Jay Tanzman (jt512) - Saturday, 13 March 2010, 20:57 GMT
Unfortunately, not all of us have Optiarc models with firmware updates. I have the Optiarc AD-7583S, and there does not appear to be a firmware update available for it.
Comment by Matthias Schwarz (craw) - Wednesday, 17 March 2010, 10:27 GMT
It seems the repeating media change events are no longer signalled after i did the firmware upgrade. Worked like a charm with the AD-7580S firmware from above on Lenovo N500.

(Used unetbootin to create a bootable usbstick with freedos and unzipped the firmware package to it.)

Thank all of you for pointing this out
Comment by Kamil Sałaś (Kamil) - Wednesday, 17 March 2010, 14:14 GMT
Firmware from Dell works perfect on Lenovo Y530 - AD-7560S.
Comment by Stephen E. Baker (TheCycoONE) - Saturday, 27 March 2010, 15:43 GMT
I get similar behavior whenever I plug in my Blackberry 7250, udevd goes to 95% cpu and stays there.
Comment by Dominik Honnef (dominikh) - Thursday, 15 April 2010, 18:56 GMT
I can confirm that updating the firmware helped.
Device: Optiarc AD-7560S
Brand of notebook: Acer Extensa
Used firmware: http://ftp.euro.dell.com/rmsd/AD-7560S%20SD05.zip
Comment by runiq (runiq) - Sunday, 16 May 2010, 12:51 GMT
I took the leap and updated my FSC Amilo optical drive with the firmware from Dell (Optical drive: Optiarc AD-7640S; updated firmware version: A03) using Windows. Worked flawlessly, even though the firmware was for another notebook (and by another manufacturer). The drive still works just fine, I now have a mere 3 udev instances, no more CPU problems and the optical drive doesn't spam useless kernel event messages in udevadm monitor anymore.

The firmware I used is this one:

http://support.dell.com/support/downloads/download.aspx?c=ca&l=en&s=gen&releaseid=R225232&SystemID=STUDIO1737&servicetag=&os=WLH&osl=en&deviceid=16565&devlib=0&typecnt=0&vercnt=1&catid=-1&impid=-1&formatcnt=0&libid=32&typeid=-1&dateid=-1&formatid=-1&source=-1&fileid=323548

A DOS version is included in the zip file.
Comment by Dirk (arathis) - Friday, 02 July 2010, 21:42 GMT
I tried the firmware and udev gives me less trouble now, but the drive became painfully slow. So I wanted to give the original fw a try with new udev and X, but I forgot to make a backup for such a case and can't find the fw online. Anyone got a dump to share?

Loading...