FS#7886 - Can't log into arch since the latest kernel upgrades.

Attached to Project: Arch Linux
Opened by Mike Dill (Theoden) - Friday, 24 August 2007, 04:03 GMT
Last edited by Aaron Griffin (phrakture) - Tuesday, 02 October 2007, 22:02 GMT
Task Type Bug Report
Category Kernel
Status Closed
Assigned To Tobias Powalowski (tpowa)
Architecture All
Severity Critical
Priority Normal
Reported Version 2007.08 Don't Panic
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:
Since the new kernel upgrade - I cannot log into my arch system with any kernel except the ck kernels. I get to the login prompt - entranced fails to start. I try to log in - once logged in - it kicks me out to the login again - over and over. When I try to reboot - CTRL-ALT-DEL - the system locks up tight immediately and gives me the following message:

INIT: Switching to runlevel: 6
INIT: Sending processes started by init the TERM signal

> Initiating Shutdown...
acpid: exiting

INIT: no more processes left in this runlevel

After that - I have to shut the system off completely and start it again.
This all started after upgrading the kernel to kernel26-2.6.22.4-1 - then to kernel26-2.6.22.4-2 - and now to kernel26-2.6.22.5-1. None of them work here.

I downgraded the kernel to kernel26-2.6.22.3-1 - but it does the same thing now and it won't work. I have kernel26ck-2.6.22.4.ck1-1 installed as well - it works perfectly. Entranced starts - E17 comes up - everything runs just fine. Currently I appear to be locked to the ck kernel - which is going away. This must be fixed soon.

Additional info:
* package version(s)
* config and/or log files etc.


Steps to reproduce:
This task depends upon

Closed by  Aaron Griffin (phrakture)
Tuesday, 02 October 2007, 22:02 GMT
Reason for closing:  Fixed
Comment by Andreas Radke (AndyRTR) - Friday, 24 August 2007, 04:36 GMT
please check your pam version. also please boot your working ck-kernel, run pacman -Syu and look for entries in system log files (everything.log and auth.log) when login fails.
Comment by Mike Dill (Theoden) - Friday, 24 August 2007, 05:25 GMT
FROM THE EVERYTHING.LOG :

Aug 21 22:25:08 Ringwraith eth0: setting full-duplex.
Aug 21 22:25:11 Ringwraith hpiod: 1.7.4a accepting connections at 2208...
Aug 21 22:25:20 Ringwraith entranced: Starting X server.
Aug 21 22:25:21 Ringwraith entranced: Starting Entrance.
Aug 21 22:25:22 Ringwraith entrance: Debug: ipc_title = /var/run/entrance/entrance_ipc_4321
Aug 21 22:25:22 Ringwraith entrance: entrance_ipc_init: Success
Aug 21 22:25:48 Ringwraith entranced: The session has ended normally.
Aug 21 22:25:48 Ringwraith entranced: The X Server apparently died as well.
Aug 21 22:25:48 Ringwraith entranced: Attempting to restart X server.
Aug 21 22:25:48 Ringwraith entranced: Successfully restarted the X server.
Aug 21 22:25:48 Ringwraith entrance: Debug: ipc_title = /var/run/entrance/entrance_ipc_4321
Aug 21 22:25:48 Ringwraith entrance: entrance_ipc_init: Success
Aug 21 22:25:51 Ringwraith entranced: The session has ended normally.
Aug 21 22:25:52 Ringwraith entranced: The X Server apparently died as well.
Aug 21 22:25:52 Ringwraith entranced: Attempting to restart X server.
Aug 21 22:25:52 Ringwraith entranced: Failed to generate auth cookie for X Server.
Aug 21 22:25:52 Ringwraith entranced: Failed to generate auth cookie for X Server.
Aug 21 22:25:52 Ringwraith entranced: Successfully restarted the X server.
Aug 21 22:25:52 Ringwraith entrance: Debug: ipc_title = /var/run/entrance/entrance_ipc_4321
Aug 21 22:25:52 Ringwraith entrance: entrance_ipc_init: Success
Aug 21 22:25:55 Ringwraith entranced: The session has ended normally.
Aug 21 22:25:55 Ringwraith entranced: The X Server apparently died as well.
Aug 21 22:25:55 Ringwraith entranced: Attempting to restart X server.
Aug 21 22:25:55 Ringwraith entranced: Successfully restarted the X server.
Aug 21 22:25:55 Ringwraith entrance: Debug: ipc_title = /var/run/entrance/entrance_ipc_4321
Aug 21 22:25:55 Ringwraith entrance: entrance_ipc_init: Success
Aug 21 22:26:00 Ringwraith entranced: The session has ended normally.
Aug 21 22:26:00 Ringwraith entranced: The X Server apparently died as well.
Aug 21 22:26:00 Ringwraith entranced: Attempting to restart X server.
Aug 21 22:26:00 Ringwraith entranced: Successfully restarted the X server.
Aug 21 22:26:01 Ringwraith entrance: Debug: ipc_title = /var/run/entrance/entrance_ipc_4321
Aug 21 22:26:01 Ringwraith entrance: entrance_ipc_init: Success
Aug 21 22:26:05 Ringwraith entranced: The session has ended normally.
Aug 21 22:26:05 Ringwraith entranced: The X Server apparently died as well.
Aug 21 22:26:05 Ringwraith entranced: Attempting to restart X server.
Aug 21 22:26:05 Ringwraith entranced: Failed to generate auth cookie for X Server.
Aug 21 22:26:05 Ringwraith entranced: Failed to generate auth cookie for X Server.
Aug 21 22:26:05 Ringwraith entranced: Failed to generate auth cookie for X Server.
Aug 21 22:26:05 Ringwraith entranced: Failed to generate auth cookie for X Server.
Aug 21 22:26:05 Ringwraith entranced: Failed to generate auth cookie for X Server.
Aug 21 22:26:05 Ringwraith entranced: Failed to restart the X server. Aborting.
Aug 21 22:29:00 Ringwraith shutdown[12359]: shutting down for system reboot
Aug 21 22:29:00 Ringwraith init: Switching to runlevel: 6
Aug 21 22:29:04 Ringwraith syslog-ng[4149]: Termination requested via signal, terminating;
Aug 21 22:29:04 Ringwraith syslog-ng[4149]: syslog-ng shutting down; version='2.0.5'
Aug 21 22:30:02 Ringwraith syslog-ng[3959]: syslog-ng starting up; version='2.0.5'
Comment by Mike Dill (Theoden) - Friday, 24 August 2007, 05:37 GMT
Pam version = 0.99.8.1-3.1
Comment by Mike Dill (Theoden) - Friday, 24 August 2007, 06:35 GMT
Ok - did some experimenting. When I boot the new kernel - I log in and it kicks me out in a few seconds. But if I type - killall entranced - before it kicks me out - I can then log in and things are ok.

If I then run - entranced start - E17 starts up fine.

So - it looks like at login time - when entranced is supposed to log in as root and start - it cannot. But the same entranced works fine with kernel26ck-2.6.22.4.ck1-1.
Comment by Travis Willard (Cerebral) - Friday, 24 August 2007, 11:22 GMT
What's in your /var/log/Xorg.0.log for a failed session? ie. why is X not starting?
Comment by Andreas Radke (AndyRTR) - Friday, 24 August 2007, 12:12 GMT
i guess "entranced" is the e17 login daemon. please try to rebuild the corresponding package against new pam.
Comment by Mike Dill (Theoden) - Friday, 24 August 2007, 15:13 GMT
Entranced works fine against the new pam. As I said above - when I choose to boot to the ck kernel (same pam) - entranced works fine and E starts up. It's ONLY - repeat ONLY with the new - non-ck - kernels that it fails.

Comment by Travis Willard (Cerebral) - Friday, 24 August 2007, 15:21 GMT
Okay, but that still doesn't tell us why X is dying on your stock kernels. Pastebin-ing /var/log/Xorg.0.log from a failed session will help us figure that out.
Comment by Mike Dill (Theoden) - Friday, 24 August 2007, 16:19 GMT
Ok - here is the xorg.0.log. Hope you can make out the error section better than I can. :-)
Comment by Travis Willard (Cerebral) - Friday, 24 August 2007, 16:46 GMT
From the logfile:

------8<------8<------8<------8<------8<------8<
Current Operating System: Linux Ringwraith.atlantis 2.6.22-ck #1 SMP PREEMPT Wed Aug 22 03:36:54 IST 2007 i686
------8<------8<------8<------8<------8<------8<

I need a log from within the failing kernel. Boot into the stock kernel, copy the Xorg.0.log file from there, and attach it, please.
Comment by Mike Dill (Theoden) - Friday, 24 August 2007, 21:24 GMT
Here is the log after booting with the new default kernel - entranced failed - I had to kill it ti stay booted up.
Comment by Travis Willard (Cerebral) - Saturday, 25 August 2007, 00:30 GMT
hm. That's odd, 'cause it still says the same line about "current operating system: 2.6.22-ck" which says to me that X isn't writing a logfile when Entranced tries to run. Once you've killed Entranced on the stock kernel, if you run startx does it work?
Comment by Mike Dill (Theoden) - Saturday, 25 August 2007, 00:55 GMT
Yes it does. In fact - once I do that - I can run entranced and it works.
Comment by Mike Dill (Theoden) - Saturday, 25 August 2007, 15:47 GMT
Here - this may be better - hope it's what you are looking for.
Comment by Travis Willard (Cerebral) - Saturday, 25 August 2007, 21:50 GMT
Hm, the only difference is that it fails to connect to acpid:

> (WW) Open ACPI failed (/var/run/acpid.socket) (Connection refused)
> (II) No APM support in BIOS or kernel

I don't think that would cause X to fail to load though. :/ Did you get that after you killed entrance and then successfully re-ran X? Does entrance write any logs anywhere?
Comment by Mike Dill (Theoden) - Saturday, 25 August 2007, 22:03 GMT
Well - I still think it is the kernel in some way and here's why:

1. The system is exactly the same regardless of which kernel I use - ck or default.
2. X is still exactly the same regardless of which kernel I use - ck or default.
3. Entranced is still exactly the same regardless of which kernel I use - ck or default.
4. acpid is still exactly the same regardless of which kernel I use - ck or default.
5. The login and initscripts are still exactly the same regardless of which kernel I use - ck or default.
6. The hardware is still exactly the same regardless of which kernel I use - ck or default.
7. NOTHING is different EXCEPT the kernel.

In troubleshooting - I learned to always determine what is different - what changed - and then narrow down to the problem. When I do that - the kernel keeps coming up as the only thing. When I use the ck kernel everything works. When I use the default kernel - it doesn't. When I break the attempt to start entranced (it is itself loging in as root I believe), it fails with the default - but when I break the attempt to startup entranced - log in - and then run entranced - it works correctly with the default kernel.

I simply don't think the problem is X. It HAS to be kernel related!
Comment by Travis Willard (Cerebral) - Saturday, 25 August 2007, 22:16 GMT
Well, great, but I never said it WASN'T kernel related. Find the comment where I said that. However, an X log saying "could not load module X" for example can tell me what's wrong with the kernel. I'm trying to help you here, and all I have to work with is logfiles, because I don't know kernels all that well, and wouldn't be able to debug a kernel package.

Also, the modules the kernel uses are different, so it's not JUST the kernel. Do you use any external modules? You're using the foss ati driver, and not fglrx, so it wouldn't be that, but if you're using anything like madwifi or something it might, perhaps, make a difference.

When you log in and kill entrance, I assume you re-run entrance as root, with sudo for example. Is there any kind of entranced.log file or something like that in /var/log?
Comment by Mike Dill (Theoden) - Saturday, 25 August 2007, 22:21 GMT
Did mean to put it off on you - sorry. No - I'm not running any external modules of any kind - everything is foss. It must have something to do with running a *dm to bring up a gui loging ... I assume it must loging as root to do that and that is where things are failing. When entranced doesn't try to log in - and I log in - and then run entranced - we fine.

I'm not sure what part of the 'kernel' or 'kernel-modules' that would be. I really am trying to give you all I can to help.
Comment by Travis Willard (Cerebral) - Monday, 27 August 2007, 17:53 GMT
FORUM> Since most users seem to be alright now - this appears to have been dropped.

It's not dropped, but I think I'm out of ideas. I can't fix something when I don't know what the problem truly is - I know the symptoms (your login manager doesn't start at bootup with the stock kernel) but I don't know the cause.

> When entranced doesn't try to log in - and I log in - and then run entranced - we fine.

Hm.. are you running entranced with sudo or as root after you log in, or just as your regular user? If you're running it as a regular user, try running it as root after you've logged in successfully and see if it fails then - that way we can be sure if it's the combination of root + kernel26 + entranced that's causing the problem.
Comment by Mike Dill (Theoden) - Monday, 27 August 2007, 18:11 GMT
I am logging in as root and then starting entranced - and that works fine.
But what is 'supposed' to happen is that entranced logs in as root and starts on it's own.
It does that with the ck kernel - but it fails to do that on the default kernel.
Entranced keeps no log that can find.
Entranced will not start if I log in as a regular user and try and start it.
If I use sudo as a regular user entranced will start after startup.

Entranced works in every way like it is supposede - except when it is supposed to log in and start automatically with the 'default kernel.'
With the ck kernel - it logs in and starts automatically just as it is supposed.
Comment by Travis Willard (Cerebral) - Monday, 27 August 2007, 18:35 GMT
I'm totally grasping at straws now - how do you start entranced? With a DAEMONS entry, or with inittab?
Comment by Mike Dill (Theoden) - Monday, 27 August 2007, 18:49 GMT
I am logging in as root and then starting entranced - and that works fine.
But what is 'supposed' to happen is that entranced logs in as root and starts on it's own.
It does that with the ck kernel - but it fails to do that on the default kernel.
Entranced keeps no log that can find.
Entranced will not start if I log in as a regular user and try and start it.
If I use sudo as a regular user entranced will start after startup.

Entranced works in every way like it is supposede - except when it is supposed to log in and start automatically with the 'default kernel.'
With the ck kernel - it logs in and starts automatically just as it is supposed.
Comment by Mike Dill (Theoden) - Monday, 27 August 2007, 18:51 GMT
Entranced is added to the daemons line in /etc/rc.conf normally. It works very well there. Until the latest default kernel.
Comment by Travis Willard (Cerebral) - Monday, 27 August 2007, 18:58 GMT
http://wiki.archlinux.org/index.php/Startup_files#If_you_want_to_startup_your_computer_to_a_DM.28Desktop_Manger._Like_XDM.2C_GDM.2C_Entrance.29_or_to_a_shell

Can you try the suggestion there (adding entranced to /etc/inittab and removing it from /etc/rc.conf) and see if that works any better?

By the way, if it's still giving you the problem when it's in inittab, you can boot directly to console by appending the number 3 to your kernel boot line (ie, if you're using grub, hit 'e' to edit the entry, then add the 3, then hit 'b' to boot.)
Comment by Mike Dill (Theoden) - Monday, 27 August 2007, 19:36 GMT
Great!! I tried the wiki solution and now I can't boot the system at all - Damn fcking kernel!!
Comment by Travis Willard (Cerebral) - Monday, 27 August 2007, 19:50 GMT
> By the way, if it's still giving you the problem when it's in inittab, you can boot directly to console by appending the number 3 to your kernel boot line

You can use this to get back into your console system, and revert the changes. That was my final idea, I'm afraid - I hope someone with more knowledge than I can get this resolved for you.
Comment by Mike Dill (Theoden) - Monday, 27 August 2007, 19:57 GMT
No it doesn't work!!
Now it boots just to this:

(none) login:

This is the absolute worst damn archlinux issue I've ever had in 3 years. I an ready to jump ship. This is fscking ridiculous!
Now I have a totally hosed system!
:-/
Comment by Travis Willard (Cerebral) - Monday, 27 August 2007, 20:01 GMT
Booting to init level 3 by adding a '3' to the end of your grub line (kind-of shown here, except you'd have to edit the line at bootup: http://wiki.archlinux.org/index.php/Adding_a_login_manager_%28KDM%2C_GDM%2C_or_XDM%29_to_automatically_boot_on_startup#GRUB) doesn't get you back to the regular login?

That is really strange - I assume all you did was edit the two lines in your inittab, correct? You didn't remove anything from it?
Comment by Mike Dill (Theoden) - Monday, 27 August 2007, 20:04 GMT
No - followed it exactly. If we don't find a fix for this soon - arch will be history for me - I have to have this system running - it's imparitive!
Comment by Travis Willard (Cerebral) - Monday, 27 August 2007, 20:12 GMT
I don't suppose you have an install CD or liveCD handy, do you? You can boot from that, mount your / partition, and revert the changes to the inittab and rc.conf files.
Comment by Mike Dill (Theoden) - Monday, 27 August 2007, 20:14 GMT
I did that - and it still boots to:

(none) login:

It just goes from bad to worse since the last kernel upgrade.
Comment by Mike Dill (Theoden) - Monday, 27 August 2007, 20:16 GMT
It's mounting the fs as read-only - and nothing can start.
Comment by Travis Willard (Cerebral) - Monday, 27 August 2007, 20:26 GMT
If you've totally reverted the changes to inittab and rc.conf (id:3:initdefault: in particular) and it's still non-functional, then I'm clueless. I've never seen nor heard of this before - I've put out a call in the IRC channel and mailing list to see if anyone else can help you.
Comment by Douglas Soares de Andrade (dsa) - Monday, 27 August 2007, 20:30 GMT
When you enter in recovery mode or when the mount partition is being mounted as read only you can try this:

mount / -o remount,rw

Do your changes and then:

mount / -o remount,ro

And after that, reboot.

HTH
Comment by Mike Dill (Theoden) - Monday, 27 August 2007, 20:58 GMT
Ok - that got my loging back
Comment by Travis Willard (Cerebral) - Monday, 27 August 2007, 21:00 GMT
Whew - okay, so you're back to the issue where entranced won't start properly with the stock kernel, but it works fine with -ck?
Comment by tardo (tardo) - Monday, 27 August 2007, 21:01 GMT
You said you can't boot... but you didn't say why? what's failing? kernel panic? can't find your rootfs?
Comment by tardo (tardo) - Monday, 27 August 2007, 21:02 GMT
Ack, this is too active... can you try another login manager? gdm/kdm/xdm are decent choices.
Comment by Aaron Griffin (phrakture) - Monday, 27 August 2007, 21:37 GMT
Wow, lots of activity. Let's see if we can discover exactly where the problem is here.
If the system boots and entranced is the _only_ thing that fails, then we most likely have a video module problem.

As a long shot... do you use lilo or grub?
Comment by Mike Dill (Theoden) - Monday, 27 August 2007, 21:40 GMT
Grub
Comment by Glenn Matthys (RedShift) - Monday, 27 August 2007, 22:13 GMT
I'm wondering why it says:

"(none) login:"

looks like rc.conf is not being read or something. Make sure your initscripts are OK. I'd do an fsck too, from some recent livecd (knoppix or something). You never know what all that crashing does to your filesystem :-)

And try a regular login manager like xdm, gdm or kdm. Those always work. Have you tried logging in using startx?
Comment by Milan Jovanovic (milan) - Monday, 27 August 2007, 22:17 GMT
I had similar situation with another Linux distro. The problem was permission mask for /dev/null . Try to chmod it to rw-rw-rw-

Cheers!
Comment by Mike Dill (Theoden) - Monday, 27 August 2007, 23:27 GMT
Ok - looks like I've finally gotten Travis' wikiwork around to work.
Somehow - my fstab file got hosed - probably in one of the lockups - that's why it couldn't read the fs ... reconstructed than manually.
Cleaned the drive. Set /etc/inittab to:

id:5:initdefault:
x:5:respawn:/opt/e17/sbin/entranced -nodaemon

Restarted the system ... it runs entranced fine now, though I cannot break it to get to console only ... have to reboot and set init 3 on the kernel line in grub to get to the command console.

So I'm functional now. Thanks to all who helped - especially Travis (Cerebral).

But now - we still haven't gotten to the reason why this is different in the current kernel. I would really like to know what's up with that.
Comment by Travis Willard (Cerebral) - Tuesday, 28 August 2007, 00:07 GMT
Mike, glad you found out what was screwing up your login, and that the inittab method worked better than the DAEMONS entry. I'm still not sure why the DAEMON wouldn't work, but here's a tip for you - if you want to get back to console, Ctrl+Alt+F1 to Ctrl+Alt+F7 will bring you to a text-only console. Then, to change to text-only multi-user mode, run

telinit 3

as root. That will stop entranced. Then, to return to graphical-login mode, run

telinit 5

That will restart entranced, and usually automatically return you to its VT. Note, though, your console will still likely be logged in wherever you ran telinit 5. You can log it out by Ctrl+Alt+F-key and the 'exit' as usual.
Comment by Travis Willard (Cerebral) - Tuesday, 28 August 2007, 00:10 GMT
Oh, and you can add another entry to your grub menu at /boot/grub/menu.lst so that you have boot-to-console and boot-to-graphical. Useful for when X fails to start.

See http://wiki.archlinux.org/index.php/Adding_a_login_manager_%28KDM%2C_GDM%2C_or_XDM%29_to_automatically_boot_on_startup#GRUB for details.
Comment by Tobias Powalowski (tpowa) - Sunday, 30 September 2007, 11:11 GMT
can we close this?
Comment by Mike Dill (Theoden) - Tuesday, 02 October 2007, 18:26 GMT
Yes - it can be closed. And thanks!

Loading...