Arch Linux

This is an error message from grub, it has nothing to do with the kernel. Possible things could be:
- wrong hd device specified
- corrupted filesystem
- dmraid configuration broken, either inside grub or the dmraid stuff on your system itself

Huh? Nothing changed in /boot/grub/menu.lst I'll double-check in case we had an intermittent dmraid failure and report back. Right now the box is running just fine on LTS. Thanks.

# (0) Arch Linux
title Arch Linux on Archangel
root (hd1,5)
kernel /vmlinuz26 root=/dev/mapper/nvidia_baaccajap5 ro vga=0x31a
initrd /kernel26.img

# (1) Arch Linux
title Arch Linux Fallback
root (hd1,5)
kernel /vmlinuz26 root=/dev/mapper/nvidia_baaccajap5 ro
initrd /kernel26-fallback.img

# (2) Arch Linux LTS
title Arch Linux LTS on Archangel
root (hd1,5)
kernel /vmlinuz26-lts root=/dev/mapper/nvidia_baaccajap5 ro vga=0x31a
initrd /kernel26-lts.img

# (3) Arch Linux LTS-Fallback
title Arch Linux LTS Fallback
root (hd1,5)
kernel /vmlinuz26-lts root=/dev/mapper/nvidia_baaccajap5 ro
initrd /kernel26-lts-fallback.img

# (4) memtest86+
title Memtest86+ on Archangel
kernel (hd1,5)/memtest86+/memtest.bin

Nope - still broken. It's a bug somewhere Jan. I just tried to boot it again and it fails at the same location. Reboot to LTS - just fine. I don't know what the heck grub is trying to do, but it is really messed up. 3 or 4 of the past 6 kernels have had boot issues on this box. All OS's work fine on the box, except for these boot issues. What say the masters as what to try next?

This looks more like a problem with the location of the kernel on your filesystem, not the kernel itself. Searching google for your error, I've seen Ubuntu users having the same problem with random Ubuntu kernels.

Is this grub 0.x with your kernel on ext4? If so, you might want to consider installing grub2 instead. I've been very unlucky with ext4 and grub 0.x on my systems also.

Thanks Jan,

I read the issues on ext4 and /boot so all my /boot partitions are still ext3. I'll google and see what comes up. We may need Tobias in on this as well as it could be a dmraid issue as well. I know there has been 1 dmraid update between 2.6.35.4-1 and 2.6.35.5-1 so maybe the update of the kernel tripped a problem with dmraid that wasn't present with 2.6.35.4-1.

17:54 archangel:~> mount
proc on /proc type proc (rw,relatime)
sys on /sys type sysfs (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=1022847,mode=755)
/dev/mapper/nvidia_baaccajap5 on / type ext3 (rw,relatime,errors=continue,data=writeback)
none on /dev/pts type devpts (rw)
none on /dev/shm type tmpfs (rw)
/dev/mapper/nvidia_baaccajap6 on /boot type ext3 (rw)
/dev/mapper/nvidia_baaccajap7 on /home type ext3 (rw)
/dev/mapper/nvidia_baaccajap9 on /var type ext3 (rw)
/dev/mapper/nvidia_baaccajap10 on /srv type ext3 (rw)
/dev/mapper/nvidia_baaccajap12 on /data type ext4 (rw)
/dev/mapper/nvidia_baaccajap11 on /home/david/pvt type ext4 (rw)

Jan,

This has got to be a dmraid/kernel/grub issue with my boards chipset. I went ahead and updated my production x86_64 box (Identical s/w config) and I have no issues booting the other box.

However, the box I opened this bug on is still DEAD. I haven't found anything relevant googling yet except that Error 24 supposedly means you have the grub (drive,part) information messed up. That is NOT the case here. This is a problem with the way the boot process inits dmraid with kernel 2.6.35.5-1. If I downgrade to 2.6.35.4-1 everything works FINE. If I upgrade to 2.6.35.5-1, the box fails to boot and I have to reboot into LTS to even bring it up.

Let me know if I can provide anything else that narrows this issue down. If any one of you want direct access to the box, email me privately and I'll send you credentials and port info. I can check anything you like at this end, I just need to know what to look at. Thanks.

Just did another test on the box that fails to boot with 2.6.35.5-1 where I downgraded it again to 2.6.35.4-2. The box boots beautifully with 2.6.35.4-2. The downgrade output is identical to the upgrade output, but this time the box boots. Here is the downgrade output:

01:59 archangel:/home/backup/pkg-old> sudo pacman -U kernel26-2.6.35.4-2-x86_64.pkg.tar.xz kernel26-headers-2.6.35.4-2-x86_64.pkg.tar.xz kernel26-manpages-2.6.35.4-2-x86_64.pkg.tar.xz
resolving dependencies...
looking for inter-conflicts...

Targets (3): kernel26-2.6.35.4-2 kernel26-headers-2.6.35.4-2 kernel26-manpages-2.6.35.4-2

Total Download Size: 0.00 MB
Total Installed Size: 161.81 MB

Proceed with installation? [Y/n]
checking package integrity...
(3/3) checking for file conflicts [#####################################] 100%
(1/3) upgrading kernel26 [#####################################] 100%
>>> Updating module dependencies. Please wait ...
>>> MKINITCPIO SETUP
>>> ----------------
>>> If you use LVM2, Encrypted root or software RAID,
>>> Ensure you enable support in /etc/mkinitcpio.conf .
>>> More information about mkinitcpio setup can be found here:
>>> http://wiki.archlinux.org/index.php/Mkinitcpio

>>> Generating initial ramdisk, using mkinitcpio. Please wait...
==> Building image "default"
==> Running command: /sbin/mkinitcpio -k 2.6.35-ARCH -c /etc/mkinitcpio.conf -g /boot/kernel26.img
:: Begin build
:: Parsing hook [base]
:: Parsing hook [udev]
:: Parsing hook [autodetect]
:: Parsing hook [pata]
:: Parsing hook [scsi]
:: Parsing hook [sata]
:: Parsing hook [dmraid]
:: Parsing hook [filesystems]
:: Generating module dependencies
:: Generating image '/boot/kernel26.img'...SUCCESS
==> SUCCESS
==> Building image "fallback"
==> Running command: /sbin/mkinitcpio -k 2.6.35-ARCH -c /etc/mkinitcpio.conf -g /boot/kernel26-fallback.img -S autodetect
:: Begin build
:: Parsing hook [base]
:: Parsing hook [udev]
:: Parsing hook [pata]
:: Parsing hook [scsi]
:: Parsing hook [sata]
:: Parsing hook [dmraid]
:: Parsing hook [filesystems]
:: Generating module dependencies
:: Generating image '/boot/kernel26-fallback.img'...SUCCESS
==> SUCCESS
(2/3) upgrading kernel26-headers [#####################################] 100%
(3/3) upgrading kernel26-manpages [#####################################] 100%
02:02 archangel:/home/backup/pkg-old> x
logout
Connection to archangel closed.
02:02 alchemy:~> sar 'sudo shutdown -r now'

The box popped right up running 2.6.35.4-2, so there is a problem here with 2.6.35.5-1. (I don't know what it is, but it is there bigger than Texas. Boots with 2.6.35.4-2 perfectly. However if I upgrade the 3 packages to 2.6.35.5-1 (kernel26, kernel26-headers, kernel26-manpages) then the box Locks-Up on boot. That makes no sense unless somebody dorked with KMS again between 2.6.35.4-2 and 2.6.35.5-1. Where there any changes there whereby my kernel line with 0x31a now prevents KMS early in 2.6.35.5-1 where it did not with 2.6.35.4-2??

What say the master archers on what to check next. The grub config hasn't changed in months so that isn't relevant. This thing smells like a dmraid init problem such that grub doesn't see the dmraid paritions and then grub throws a grub error, where the error all along was a kernel dmraid init issue.

I'm basically out of ideas on what I can try next. Can you throw a brother a bone of wisdom here? Thanks.

Just a FYI follow-up - updated to kernel26-lts-2.6.32.22-1-x86_64 and it works just fine. 2.6.35.5-1 still won't boot.

There was one strange curiosity when I did the last update. pacman updated 2.6.35.5-1 again even though there was no minor version change. Here is the update output:

FORGET THE ABOVE COMMENT - I DOWNGRADED SO THE UPGRADE WAS EXPECTED (duh...)

<snip>

Targets (13): bluez-4.72-1 ed-1.5-1 electricsheep-2.7b12-3 gparted-0.6.3-1 kernel26-2.6.35.5-1
kernel26-headers-2.6.35.5-1 kernel26-lts-2.6.32.22-1 kernel26-manpages-2.6.35.5-1
konq-plugins-svn1179246-1 mingw32-binutils-2.20.51-1 sysvinit-2.88-1 transcode-1.1.5-5
xscreensaver-5.12-1

Total Download Size: 28.70 MB
Total Installed Size: 303.24 MB

<snip>

( 5/13) upgrading kernel26 [#####################################] 100%
>>> Updating module dependencies. Please wait ...
>>> MKINITCPIO SETUP
>>> ----------------
>>> If you use LVM2, Encrypted root or software RAID,
>>> Ensure you enable support in /etc/mkinitcpio.conf .
>>> More information about mkinitcpio setup can be found here:
>>> http://wiki.archlinux.org/index.php/Mkinitcpio

>>> Generating initial ramdisk, using mkinitcpio. Please wait...
==> Building image "default"
==> Running command: /sbin/mkinitcpio -k 2.6.35-ARCH -c /etc/mkinitcpio.conf -g /boot/kernel26.img
:: Begin build
:: Parsing hook [base]
:: Parsing hook [udev]
:: Parsing hook [autodetect]
:: Parsing hook [pata]
:: Parsing hook [scsi]
:: Parsing hook [sata]
:: Parsing hook [dmraid]
:: Parsing hook [filesystems]
:: Generating module dependencies
:: Generating image '/boot/kernel26.img'...SUCCESS
==> SUCCESS
==> Building image "fallback"
==> Running command: /sbin/mkinitcpio -k 2.6.35-ARCH -c /etc/mkinitcpio.conf -g /boot/kernel26-fallback.img -S autodetect
:: Begin build
:: Parsing hook [base]
:: Parsing hook [udev]
:: Parsing hook [pata]
:: Parsing hook [scsi]
:: Parsing hook [sata]
:: Parsing hook [dmraid]
:: Parsing hook [filesystems]
:: Generating module dependencies
:: Generating image '/boot/kernel26-fallback.img'...SUCCESS
==> SUCCESS
( 6/13) upgrading kernel26-headers [#####################################] 100%

I tried booting 2.6.35.5-1 and it failed at the same point - first few lines on the screen and then locks up. (the grub Error 24 is gone now, but it still hangs)

I agree it looks like grub reads the boot code from the MBR and then can't find the dmraid partition to boot from. However -- nothing, I mean nothing has changed there between 2.6.35.4-2 and 2.6.35.5-1.

Since this box is a server, I'd rather not move to grub2 until I am more comfortable with it. Why is normal old grub failing?

Here is a bit of additional info about my config.

mkinitcpio.conf:

[09:52 archangel:/home/david] # cat /etc/mkinitcpio.conf
# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run. Advanced users may wish to specify all system modules
# in this array. For instance:
# MODULES="piix ide_disk reiserfs"
MODULES="dm_mod dm_mirror sata_nv"

rc.conf:

MOD_AUTOLOAD="yes"
#MOD_BLACKLIST=() #deprecated
MODULES=(dm_mod dm_mirror sata_sil nvidia)

# Scan for LVM volume groups at startup, required if you use LVM
USELVM="no"

These settings are the same settings I have always used with this box and it 2 nvraid dmraid arrays.

Let me know what else to check. I'd like to figure out why 2.6.35.5-1 won't boot and get back on the normal Arch kernel.

Oops forgot the HOOKS line:

HOOKS="base udev autodetect pata scsi sata dmraid filesystems"

could this be related to the sata_sil issue in:

https://bugs.archlinux.org/task/20614

In rc.conf I've tried changing the MODULES line sata inclusion from sata_sil with sata_nv and rebooted - no change. The normal kernel still fails and LTS still boots fine.

I misstated in my recent comment that the GRUB ERROR: 24 was gone -- it's not, it's still there, I just shut down after the hang but before the grub timeout. GRUB ERROR: 24 is still present. Back to square 1...

Jan,

Any other thoughts/guesses here? I'll try anything. I know you said it looked like grub can't find the kernel, but why? My menu.lst hasn't changed since August 2009 when I added LTS and the contents of /boot look fine:

[14:32 archangel:/home/david] # l /boot
total 28777
drwxr-xr-x 5 root root 1024 Sep 26 18:55 .
drwxr-xr-x 22 root root 4096 Sep 21 15:45 ..
-rw-r--r-- 1 root root 1400334 Sep 21 04:57 System.map26
-rw-r--r-- 1 root root 1345754 Sep 21 04:57 System.map26-lts
drwxr-xr-x 2 root root 1024 Aug 23 13:18 grub
-rw-r--r-- 1 root root 9098203 Sep 26 18:55 kernel26-fallback.img
-rw-r--r-- 1 root root 8305205 Sep 26 18:56 kernel26-lts-fallback.img
-rw-r--r-- 1 root root 2476409 Sep 26 18:55 kernel26-lts.img
-rw-r--r-- 1 root root 2501522 Sep 26 18:54 kernel26.img
drwx------ 2 root root 12288 Apr 20 2009 lost+found
drwxr-xr-x 2 root root 1024 Jul 4 14:40 memtest86+
-rw-r--r-- 1 root root 2154576 Sep 21 04:57 vmlinuz26
-rw-r--r-- 1 root root 2034832 Sep 21 04:57 vmlinuz26-lts

There is something in the way 2.6.35.5-1 inits that's just broken, but what? Any help appreciated.

Also, I looked at the initrd images with 'bsdtar -t -f /boot/kernel26.img' and compared it to LTS 'bsdtar -t -f /boot/kernel26-lts.img'.

The only difference between them other than the obvious '2.6.35-ARCH' and '2.6.32-lts' is the 2.6.35-ARCH initrd contains:

/lib/modules/2.6.35-ARCH/kernel/drivers/ata/ahci.ko

which LTS doesn't. This line is right before the '/dev/mapper' entry. Could ahci.ko be part of the problem?

Guys - update: 2.6.35.6-1 boot but hangs on UDEV activation until the 120 sec timeout occurs. Here is the dmesg error:

INFO: task modprobe:1547 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe D 0000000000000000 0 1547 1545 0x00000000
ffff880226c91a28 0000000000000086 ffff88022fca0000 ffff880200000000
0000000000014f40 0000000000014f40 ffff880226c91fd8 ffff880226c91fd8
ffff880226c91fd8 ffff880226ec6120 ffff880226c91fd8 0000000000014f40
Call Trace:
[<ffffffffa01f7ba5>] usb_kill_urb+0x85/0xc0 [usbcore]
[<ffffffff810719b0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0275831>] usbhid_init_reports+0xb1/0x120 [usbhid]
[<ffffffffa0275d53>] usbhid_start+0x4b3/0x5a0 [usbhid]
[<ffffffffa025f6d8>] hid_device_probe+0x98/0xe0 [hid]
[<ffffffff81287cfa>] ? driver_sysfs_add+0x5a/0x90
[<ffffffff81287fd6>] driver_probe_device+0x96/0x1c0
[<ffffffff812881a0>] ? __device_attach+0x0/0x60
[<ffffffff812881eb>] __device_attach+0x4b/0x60
[<ffffffff81286bb4>] bus_for_each_drv+0x64/0x90
[<ffffffff81287e6f>] device_attach+0x8f/0xb0
[<ffffffff81287625>] bus_probe_device+0x25/0x40
[<ffffffff8128536f>] device_add+0x4ff/0x5e0
[<ffffffffa025f0a7>] hid_add_device+0x87/0x1b0 [hid]
[<ffffffffa02734b9>] usbhid_probe+0x329/0x500 [usbhid]
[<ffffffffa01fc9fb>] usb_probe_interface+0xfb/0x1f0 [usbcore]
[<ffffffff81287fd6>] driver_probe_device+0x96/0x1c0
[<ffffffff8128819b>] __driver_attach+0x9b/0xa0
[<ffffffff81288100>] ? __driver_attach+0x0/0xa0
[<ffffffff81286f0e>] bus_for_each_dev+0x5e/0x90
[<ffffffff81287c99>] driver_attach+0x19/0x20
[<ffffffff812877a7>] bus_add_driver+0xc7/0x2e0
[<ffffffff81288411>] driver_register+0x71/0x140
[<ffffffffa01fb6c8>] usb_register_driver+0xb8/0x170 [usbcore]
[<ffffffffa01c9000>] ? hid_init+0x0/0xd1 [usbhid]
[<ffffffffa01c9093>] hid_init+0x93/0xd1 [usbhid]
[<ffffffff81002149>] do_one_initcall+0x39/0x1a0
[<ffffffff8108cfdb>] sys_init_module+0xbb/0x200
[<ffffffff81009e82>] system_call_fastpath+0x16/0x1b
INFO: task modprobe:1547 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe D 0000000000000000 0 1547 1545 0x00000000
ffff880226c91a28 0000000000000086 ffff88022fca0000 ffff880200000000
0000000000014f40 0000000000014f40 ffff880226c91fd8 ffff880226c91fd8
ffff880226c91fd8 ffff880226ec6120 ffff880226c91fd8 0000000000014f40
Call Trace:
[<ffffffffa01f7ba5>] usb_kill_urb+0x85/0xc0 [usbcore]
[<ffffffff810719b0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0275831>] usbhid_init_reports+0xb1/0x120 [usbhid]
[<ffffffffa0275d53>] usbhid_start+0x4b3/0x5a0 [usbhid]
[<ffffffffa025f6d8>] hid_device_probe+0x98/0xe0 [hid]
[<ffffffff81287cfa>] ? driver_sysfs_add+0x5a/0x90
[<ffffffff81287fd6>] driver_probe_device+0x96/0x1c0
[<ffffffff812881a0>] ? __device_attach+0x0/0x60
[<ffffffff812881eb>] __device_attach+0x4b/0x60
[<ffffffff81286bb4>] bus_for_each_drv+0x64/0x90
[<ffffffff81287e6f>] device_attach+0x8f/0xb0
[<ffffffff81287625>] bus_probe_device+0x25/0x40
[<ffffffff8128536f>] device_add+0x4ff/0x5e0
[<ffffffffa025f0a7>] hid_add_device+0x87/0x1b0 [hid]
[<ffffffffa02734b9>] usbhid_probe+0x329/0x500 [usbhid]
[<ffffffffa01fc9fb>] usb_probe_interface+0xfb/0x1f0 [usbcore]
[<ffffffff81287fd6>] driver_probe_device+0x96/0x1c0
[<ffffffff8128819b>] __driver_attach+0x9b/0xa0
[<ffffffff81288100>] ? __driver_attach+0x0/0xa0
[<ffffffff81286f0e>] bus_for_each_dev+0x5e/0x90
[<ffffffff81287c99>] driver_attach+0x19/0x20
[<ffffffff812877a7>] bus_add_driver+0xc7/0x2e0
[<ffffffff81288411>] driver_register+0x71/0x140
[<ffffffffa01fb6c8>] usb_register_driver+0xb8/0x170 [usbcore]
[<ffffffffa01c9000>] ? hid_init+0x0/0xd1 [usbhid]
[<ffffffffa01c9093>] hid_init+0x93/0xd1 [usbhid]
[<ffffffff81002149>] do_one_initcall+0x39/0x1a0
[<ffffffff8108cfdb>] sys_init_module+0xbb/0x200
[<ffffffff81009e82>] system_call_fastpath+0x16/0x1b
INFO: task usbhid-ups:2023 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
usbhid-ups D 0000000000000000 0 2023 1 0x00000000
ffff8802251d3b98 0000000000000082 ffff8802251d3ae8 ffffffff00000000
0000000000014f40 0000000000014f40 ffff8802251d3fd8 ffff8802251d3fd8
ffff8802251d3fd8 ffff8802258f6810 ffff8802251d3fd8 0000000000014f40
Call Trace:
[<ffffffff81371fc9>] __mutex_lock_slowpath+0x139/0x310
[<ffffffff813721b1>] mutex_lock+0x11/0x30
[<ffffffffa0201165>] usbdev_open+0x145/0x330 [usbcore]
[<ffffffff81127d17>] chrdev_open+0x127/0x270
[<ffffffff81127bf0>] ? chrdev_open+0x0/0x270
[<ffffffff8112223b>] __dentry_open+0x11b/0x3c0
[<ffffffff811a2b2a>] ? security_inode_permission+0x1a/0x20
[<ffffffff81123474>] nameidata_to_filp+0x54/0x70
[<ffffffff81131700>] do_last+0x480/0x740
[<ffffffff81131bad>] do_filp_open+0x1ed/0x680
[<ffffffff8113d184>] ? alloc_fd+0xf4/0x150
[<ffffffff811234f4>] do_sys_open+0x64/0x130
[<ffffffff811235db>] sys_open+0x1b/0x20
[<ffffffff81009e82>] system_call_fastpath+0x16/0x1b
INFO: task modprobe:1547 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe D 0000000000000000 0 1547 1545 0x00000000
ffff880226c91a28 0000000000000086 ffff88022fca0000 ffff880200000000
0000000000014f40 0000000000014f40 ffff880226c91fd8 ffff880226c91fd8
ffff880226c91fd8 ffff880226ec6120 ffff880226c91fd8 0000000000014f40
Call Trace:
[<ffffffffa01f7ba5>] usb_kill_urb+0x85/0xc0 [usbcore]
[<ffffffff810719b0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0275831>] usbhid_init_reports+0xb1/0x120 [usbhid]
[<ffffffffa0275d53>] usbhid_start+0x4b3/0x5a0 [usbhid]
[<ffffffffa025f6d8>] hid_device_probe+0x98/0xe0 [hid]
[<ffffffff81287cfa>] ? driver_sysfs_add+0x5a/0x90
[<ffffffff81287fd6>] driver_probe_device+0x96/0x1c0
[<ffffffff812881a0>] ? __device_attach+0x0/0x60
[<ffffffff812881eb>] __device_attach+0x4b/0x60
[<ffffffff81286bb4>] bus_for_each_drv+0x64/0x90
[<ffffffff81287e6f>] device_attach+0x8f/0xb0
[<ffffffff81287625>] bus_probe_device+0x25/0x40
[<ffffffff8128536f>] device_add+0x4ff/0x5e0
[<ffffffffa025f0a7>] hid_add_device+0x87/0x1b0 [hid]
[<ffffffffa02734b9>] usbhid_probe+0x329/0x500 [usbhid]
[<ffffffffa01fc9fb>] usb_probe_interface+0xfb/0x1f0 [usbcore]
[<ffffffff81287fd6>] driver_probe_device+0x96/0x1c0
[<ffffffff8128819b>] __driver_attach+0x9b/0xa0
[<ffffffff81288100>] ? __driver_attach+0x0/0xa0
[<ffffffff81286f0e>] bus_for_each_dev+0x5e/0x90
[<ffffffff81287c99>] driver_attach+0x19/0x20
[<ffffffff812877a7>] bus_add_driver+0xc7/0x2e0
[<ffffffff81288411>] driver_register+0x71/0x140
[<ffffffffa01fb6c8>] usb_register_driver+0xb8/0x170 [usbcore]
[<ffffffffa01c9000>] ? hid_init+0x0/0xd1 [usbhid]
[<ffffffffa01c9093>] hid_init+0x93/0xd1 [usbhid]
[<ffffffff81002149>] do_one_initcall+0x39/0x1a0
[<ffffffff8108cfdb>] sys_init_module+0xbb/0x200
[<ffffffff81009e82>] system_call_fastpath+0x16/0x1b
INFO: task usbhid-ups:2023 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
usbhid-ups D 0000000000000000 0 2023 1 0x00000000
ffff8802251d3b98 0000000000000082 ffff8802251d3ae8 ffffffff00000000
0000000000014f40 0000000000014f40 ffff8802251d3fd8 ffff8802251d3fd8
ffff8802251d3fd8 ffff8802258f6810 ffff8802251d3fd8 0000000000014f40
Call Trace:
[<ffffffff81371fc9>] __mutex_lock_slowpath+0x139/0x310
[<ffffffff813721b1>] mutex_lock+0x11/0x30
[<ffffffffa0201165>] usbdev_open+0x145/0x330 [usbcore]
[<ffffffff81127d17>] chrdev_open+0x127/0x270
[<ffffffff81127bf0>] ? chrdev_open+0x0/0x270
[<ffffffff8112223b>] __dentry_open+0x11b/0x3c0
[<ffffffff811a2b2a>] ? security_inode_permission+0x1a/0x20
[<ffffffff81123474>] nameidata_to_filp+0x54/0x70
[<ffffffff81131700>] do_last+0x480/0x740
[<ffffffff81131bad>] do_filp_open+0x1ed/0x680
[<ffffffff8113d184>] ? alloc_fd+0xf4/0x150
[<ffffffff811234f4>] do_sys_open+0x64/0x130
[<ffffffff811235db>] sys_open+0x1b/0x20
[<ffffffff81009e82>] system_call_fastpath+0x16/0x1b
INFO: task modprobe:1547 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe D 0000000000000000 0 1547 1545 0x00000000
ffff880226c91a28 0000000000000086 ffff88022fca0000 ffff880200000000
0000000000014f40 0000000000014f40 ffff880226c91fd8 ffff880226c91fd8
ffff880226c91fd8 ffff880226ec6120 ffff880226c91fd8 0000000000014f40
Call Trace:
[<ffffffffa01f7ba5>] usb_kill_urb+0x85/0xc0 [usbcore]
[<ffffffff810719b0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0275831>] usbhid_init_reports+0xb1/0x120 [usbhid]
[<ffffffffa0275d53>] usbhid_start+0x4b3/0x5a0 [usbhid]
[<ffffffffa025f6d8>] hid_device_probe+0x98/0xe0 [hid]
[<ffffffff81287cfa>] ? driver_sysfs_add+0x5a/0x90
[<ffffffff81287fd6>] driver_probe_device+0x96/0x1c0
[<ffffffff812881a0>] ? __device_attach+0x0/0x60
[<ffffffff812881eb>] __device_attach+0x4b/0x60
[<ffffffff81286bb4>] bus_for_each_drv+0x64/0x90
[<ffffffff81287e6f>] device_attach+0x8f/0xb0
[<ffffffff81287625>] bus_probe_device+0x25/0x40
[<ffffffff8128536f>] device_add+0x4ff/0x5e0
[<ffffffffa025f0a7>] hid_add_device+0x87/0x1b0 [hid]
[<ffffffffa02734b9>] usbhid_probe+0x329/0x500 [usbhid]
[<ffffffffa01fc9fb>] usb_probe_interface+0xfb/0x1f0 [usbcore]
[<ffffffff81287fd6>] driver_probe_device+0x96/0x1c0
[<ffffffff8128819b>] __driver_attach+0x9b/0xa0
[<ffffffff81288100>] ? __driver_attach+0x0/0xa0
[<ffffffff81286f0e>] bus_for_each_dev+0x5e/0x90
[<ffffffff81287c99>] driver_attach+0x19/0x20
[<ffffffff812877a7>] bus_add_driver+0xc7/0x2e0
[<ffffffff81288411>] driver_register+0x71/0x140
[<ffffffffa01fb6c8>] usb_register_driver+0xb8/0x170 [usbcore]
[<ffffffffa01c9000>] ? hid_init+0x0/0xd1 [usbhid]
[<ffffffffa01c9093>] hid_init+0x93/0xd1 [usbhid]
[<ffffffff81002149>] do_one_initcall+0x39/0x1a0
[<ffffffff8108cfdb>] sys_init_module+0xbb/0x200
[<ffffffff81009e82>] system_call_fastpath+0x16/0x1b
INFO: task usbhid-ups:2023 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
usbhid-ups D 0000000000000000 0 2023 1 0x00000000
ffff8802251d3b98 0000000000000082 ffff8802251d3ae8 ffffffff00000000
0000000000014f40 0000000000014f40 ffff8802251d3fd8 ffff8802251d3fd8
ffff8802251d3fd8 ffff8802258f6810 ffff8802251d3fd8 0000000000014f40
Call Trace:
[<ffffffff81371fc9>] __mutex_lock_slowpath+0x139/0x310
[<ffffffff813721b1>] mutex_lock+0x11/0x30
[<ffffffffa0201165>] usbdev_open+0x145/0x330 [usbcore]
[<ffffffff81127d17>] chrdev_open+0x127/0x270
[<ffffffff81127bf0>] ? chrdev_open+0x0/0x270
[<ffffffff8112223b>] __dentry_open+0x11b/0x3c0
[<ffffffff811a2b2a>] ? security_inode_permission+0x1a/0x20
[<ffffffff81123474>] nameidata_to_filp+0x54/0x70
[<ffffffff81131700>] do_last+0x480/0x740
[<ffffffff81131bad>] do_filp_open+0x1ed/0x680
[<ffffffff8113d184>] ? alloc_fd+0xf4/0x150
[<ffffffff811234f4>] do_sys_open+0x64/0x130
[<ffffffff811235db>] sys_open+0x1b/0x20
[<ffffffff81009e82>] system_call_fastpath+0x16/0x1b
INFO: task modprobe:1547 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe D 0000000000000000 0 1547 1545 0x00000000
ffff880226c91a28 0000000000000086 ffff88022fca0000 ffff880200000000
0000000000014f40 0000000000014f40 ffff880226c91fd8 ffff880226c91fd8
ffff880226c91fd8 ffff880226ec6120 ffff880226c91fd8 0000000000014f40
Call Trace:
[<ffffffffa01f7ba5>] usb_kill_urb+0x85/0xc0 [usbcore]
[<ffffffff810719b0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0275831>] usbhid_init_reports+0xb1/0x120 [usbhid]
[<ffffffffa0275d53>] usbhid_start+0x4b3/0x5a0 [usbhid]
[<ffffffffa025f6d8>] hid_device_probe+0x98/0xe0 [hid]
[<ffffffff81287cfa>] ? driver_sysfs_add+0x5a/0x90
[<ffffffff81287fd6>] driver_probe_device+0x96/0x1c0
[<ffffffff812881a0>] ? __device_attach+0x0/0x60
[<ffffffff812881eb>] __device_attach+0x4b/0x60
[<ffffffff81286bb4>] bus_for_each_drv+0x64/0x90
[<ffffffff81287e6f>] device_attach+0x8f/0xb0
[<ffffffff81287625>] bus_probe_device+0x25/0x40
[<ffffffff8128536f>] device_add+0x4ff/0x5e0
[<ffffffffa025f0a7>] hid_add_device+0x87/0x1b0 [hid]
[<ffffffffa02734b9>] usbhid_probe+0x329/0x500 [usbhid]
[<ffffffffa01fc9fb>] usb_probe_interface+0xfb/0x1f0 [usbcore]
[<ffffffff81287fd6>] driver_probe_device+0x96/0x1c0
[<ffffffff8128819b>] __driver_attach+0x9b/0xa0
[<ffffffff81288100>] ? __driver_attach+0x0/0xa0
[<ffffffff81286f0e>] bus_for_each_dev+0x5e/0x90
[<ffffffff81287c99>] driver_attach+0x19/0x20
[<ffffffff812877a7>] bus_add_driver+0xc7/0x2e0
[<ffffffff81288411>] driver_register+0x71/0x140
[<ffffffffa01fb6c8>] usb_register_driver+0xb8/0x170 [usbcore]
[<ffffffffa01c9000>] ? hid_init+0x0/0xd1 [usbhid]
[<ffffffffa01c9093>] hid_init+0x93/0xd1 [usbhid]
[<ffffffff81002149>] do_one_initcall+0x39/0x1a0
[<ffffffff8108cfdb>] sys_init_module+0xbb/0x200
[<ffffffff81009e82>] system_call_fastpath+0x16/0x1b
INFO: task usbhid-ups:2023 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
usbhid-ups D 0000000000000000 0 2023 1 0x00000000
ffff8802251d3b98 0000000000000082 ffff8802251d3ae8 ffffffff00000000
0000000000014f40 0000000000014f40 ffff8802251d3fd8 ffff8802251d3fd8
ffff8802251d3fd8 ffff8802258f6810 ffff8802251d3fd8 0000000000014f40
Call Trace:
[<ffffffff81371fc9>] __mutex_lock_slowpath+0x139/0x310
[<ffffffff813721b1>] mutex_lock+0x11/0x30
[<ffffffffa0201165>] usbdev_open+0x145/0x330 [usbcore]
[<ffffffff81127d17>] chrdev_open+0x127/0x270
[<ffffffff81127bf0>] ? chrdev_open+0x0/0x270
[<ffffffff8112223b>] __dentry_open+0x11b/0x3c0
[<ffffffff811a2b2a>] ? security_inode_permission+0x1a/0x20
[<ffffffff81123474>] nameidata_to_filp+0x54/0x70
[<ffffffff81131700>] do_last+0x480/0x740
[<ffffffff81131bad>] do_filp_open+0x1ed/0x680
[<ffffffff8113d184>] ? alloc_fd+0xf4/0x150
[<ffffffff811234f4>] do_sys_open+0x64/0x130
[<ffffffff811235db>] sys_open+0x1b/0x20
[<ffffffff81009e82>] system_call_fastpath+0x16/0x1b
INFO: task modprobe:1547 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe D 0000000000000000 0 1547 1545 0x00000000
ffff880226c91a28 0000000000000086 ffff88022fca0000 ffff880200000000
0000000000014f40 0000000000014f40 ffff880226c91fd8 ffff880226c91fd8
ffff880226c91fd8 ffff880226ec6120 ffff880226c91fd8 0000000000014f40
Call Trace:
[<ffffffffa01f7ba5>] usb_kill_urb+0x85/0xc0 [usbcore]
[<ffffffff810719b0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0275831>] usbhid_init_reports+0xb1/0x120 [usbhid]
[<ffffffffa0275d53>] usbhid_start+0x4b3/0x5a0 [usbhid]
[<ffffffffa025f6d8>] hid_device_probe+0x98/0xe0 [hid]
[<ffffffff81287cfa>] ? driver_sysfs_add+0x5a/0x90
[<ffffffff81287fd6>] driver_probe_device+0x96/0x1c0
[<ffffffff812881a0>] ? __device_attach+0x0/0x60
[<ffffffff812881eb>] __device_attach+0x4b/0x60
[<ffffffff81286bb4>] bus_for_each_drv+0x64/0x90
[<ffffffff81287e6f>] device_attach+0x8f/0xb0
[<ffffffff81287625>] bus_probe_device+0x25/0x40
[<ffffffff8128536f>] device_add+0x4ff/0x5e0
[<ffffffffa025f0a7>] hid_add_device+0x87/0x1b0 [hid]
[<ffffffffa02734b9>] usbhid_probe+0x329/0x500 [usbhid]
[<ffffffffa01fc9fb>] usb_probe_interface+0xfb/0x1f0 [usbcore]
[<ffffffff81287fd6>] driver_probe_device+0x96/0x1c0
[<ffffffff8128819b>] __driver_attach+0x9b/0xa0
[<ffffffff81288100>] ? __driver_attach+0x0/0xa0
[<ffffffff81286f0e>] bus_for_each_dev+0x5e/0x90
[<ffffffff81287c99>] driver_attach+0x19/0x20
[<ffffffff812877a7>] bus_add_driver+0xc7/0x2e0
[<ffffffff81288411>] driver_register+0x71/0x140
[<ffffffffa01fb6c8>] usb_register_driver+0xb8/0x170 [usbcore]
[<ffffffffa01c9000>] ? hid_init+0x0/0xd1 [usbhid]
[<ffffffffa01c9093>] hid_init+0x93/0xd1 [usbhid]
[<ffffffff81002149>] do_one_initcall+0x39/0x1a0
[<ffffffff8108cfdb>] sys_init_module+0xbb/0x200
[<ffffffff81009e82>] system_call_fastpath+0x16/0x1b

Full dmesg output is here:

http://www.3111skyline.com/dl/Archlinux/bugs/dmesg20100927.txt

Whatever the problem was, it looks fixed in kernel26 2.6.35.6-2. Oh well, there are just somethings in the world you just can't explain :p

It would be nice to get at least some explanation of what was broken in kernel26 2.6.35.5-1 and fixed in kernel26 2.6.35.6-2 so I would at least know where to look if we suffer a regression in the future...