FS#3553 - initrd handling suggestions

Attached to Project: Arch Linux
Opened by Aaron Griffin (phrakture) - Wednesday, 30 November 2005, 22:55 GMT
Last edited by Tobias Powalowski (tpowa) - Thursday, 01 December 2005, 08:03 GMT
Task Type Feature Request
Category Kernel
Status Closed
Assigned To Judd Vinet (judd)
Architecture not specified
Severity Medium
Priority Normal
Reported Version 0.7 Wombat
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Ok, this will be a bit of a longish bug report / feature request here - be prepared.

Let me begin by saying that the default initrd changes are great and very much needed. It allows for much more fine tuning without having to recompile an entire kernel. Also, early-userspace is nice and fun and opens up many new avenues for things such as usplash (userspace boot splash).

Ok, that said, I have some issues with how the initrd/hwdetect thing is being handled at current. One of the key features of arch is that 95% of the packages "Just Work" as soon as you install them. It goes hand-in-hand with the little blurb about "sane defaults" in the about page. However, this is not the case with the current initrd implementation. I have heard from *many* users on how disheartened with Arch they have become. Things no longer work out of the box, ever initscript/kernel change is a disaster and causes drama. I have heard from a few very prominent Arch users that they are looking elsewhere for the linux needs.

The current initrd implementation points users to how to create their own initrd images. This is fine and dandy, but there is a high probability of errors when leaving something like this up to the end user. This could be solved in a round-about way, by simply telling the user to add an initrd line pointing to the "full" image, and telling them to read this-or-that document on how to create their own. By the same token, there is _no_reason_ why the full initrd image should be built on the client side on install. It should be built in the PKGBUILD and placed next to the kernel. The "auto" initrd should also not be built on install, as it should be up to the end user to determine if they will optimize their image or not.

Let's move on to the new initscripts stuff. These changes with hwdetect are nice as well, and work very good. However, I have to point out that the constant updates are angering alot of people. I honestly have no idea why something like this is being pushed into rc.sysinit. I see no reason for it. It is replacing the hotplug/hwd initscripts, so it follows that it should, itself, be an initscript. Even better, it should be it's own package. It's self contained enough, and, in all honesty, has about as much to do with the "init" process as modprobe does. With hwdetect as it's own package, people no longer have to constantly update rc.conf and constantly check modules and make sure things are working. It just makes more sense.

Those are my two big issues - the confusion around the initrd system, and the initscripts changes. Solutions have been offered, and even if they are not ideal, they can be molded, and should be - I fear what will happen if things continue in this fashion.

On a related note, initramfs-tools from debian has alot of really nice code that would be *very* useful for building these initrd images. They even supply functionality for you to add userspace "hooks" so one can inject code into the init (linuxrc) script for the image. I think it would be great if you took a look at those scripts (or even just directly ported them to Arch - initramfs has numerous advantages over initrd, and initrd will be replaced by either initramfs or yaird in the future).

I will direct some users here, to add comments to this report. I warn those users not to get out of hand, as if this bug report becomes some sort of personal flame-war, it will most likely be closed and ignored. Please, if you're going to add anything, at least make suggestions, and not just complaints.

- phrakture -
This task depends upon

Closed by  Tobias Powalowski (tpowa)
Thursday, 22 June 2006, 14:55 GMT
Reason for closing:  Fixed
Comment by Woody Gilk (Shadowhand) - Wednesday, 30 November 2005, 23:22 GMT
I've been worried about the implications of hwdetect since it was announced. tpowa has done a wonderful job with it, and it's really an amazing piece of work. However, the fact that it's a boot time process, happening in sysinit, makes me uncomfortable.

I'm going to attempt to outline some of my issues with hwdetect:
1) Hardware detection should be an _optional_, not a _required_ part of Arch.
2) hwdetect should be a rc.d script, and not in rc.sysinit (see #1)
3) If hwdetect is going to be the perferred method of hardware detection, hwd and hotplug should be depreciated or removed
4) The issues with hwdetect using the MODULES array would dissappear if it were a script (see #1 and #2)
5) hwdetect makes rc.conf more complicated by adding the MOD_AUTOLOAD and MOD_BLACKLIST options (see #4) Where's the K.I.S.S. in that?
6) hwdetect should read the modules array and create it's blacklist from the banged ("!") modules, instead of making a new array

I'm sorry some of those are very redundent, but I think that the redundency is justified. Again, I think hwdetect is great and tpowa has been (seemingly) tireless in implementing fixes and features when the community saw a problem. Phrakture is right about the constant updates being a pain, but at the same time, I understand that it means the development cycle is happening very very fast, which is wonderful.
Comment by Tobias Powalowski (tpowa) - Thursday, 01 December 2005, 08:02 GMT
At Phrak:
why do you need to update your rc.conf when initscripts are updated? keep an eye on .pacnew if something changed else you don't have to do anything.
"initscript/kernel change is a disaster and causes drama" ?!?
why? kernel is in testing that means we test it and everyone who wants to join should read the news/blog, write us an email etc., hwdetect is really final and i don't see anyone with real problems anymore, that he wouldn't have by using the modules= array.

hmm, let me sum up:
i don't see where initrd problems are:
you have to add the initrd to your bootloader, that is the only action the user must do.
(that might be a problem, but there are big hints while installing and a news on frontpage will come too).
if you don't even take a look at mkinitrd.conf the normal initrd will contain everything you need to boot.
it's like booting the scsi kernel. If you decide to tweak it go ahead and rebuild it.
the decision of not inculding it in kernel was due to the fact that ppl have their own initrd config and running it as script afterwards makes sense.
If something went wrong you always have the fallback to ...full.img to rescue your system without the need of a rescue cd.

hwdetect/modules:
see it this way, we implemented the load_modules=off option this way you can boot your system wihtout any modules modprobing and be able to boot if modules do something wrong, i think that is really great.
load_modules=off init s + initrd26-full.img at boot prompt and you can repair arch without the need of a rescue cd.
modules were always loaded by rc.sysinit, hwdetect does nothing else so it is there, it's only a helper for modules= array, so it should be there. You can disable it in rc.conf, you can disable it by boot prompt.
The last updates were needed for people that don't run a standard arch kernel and that triggered the fast updates, you never know how people "abuse" their systems apart from very different/exotic hardware setups.

We made Arch more flexible + you have more possibilities to rescue your system.
We try to keep ppl with custom kernels happy and try to fix appearing bugs as fast as possible.
Then initscripts updates will calm down, hey we haven't releases a new one since 3 days ;).

The new documentation will cover these things to make it easier to understand.

hotplug/lshwd/hwd:
hotplug cannot be removed because we provide still kernel24 keep that in mind.
lshwd has really weak points but it's faster then hotplug and is also good for older kernels.
hwd can be used for xorg.conf that is not done by hwdetect.
these are good reasons for not removing them.

Hope i have answered to most issues and you understand now that the changes does make sense and in the long run we will have much benefit from them.
Judd please add more if i missed something, thanks.

greetings
tpowa
Comment by Aaron Griffin (phrakture) - Thursday, 01 December 2005, 16:21 GMT
tpowa:
First off, I never said I have problems with the initscripts updates. It's just what I've gotten from users via the Mailing List, Forums, and IRC - basically, even if things do not change, people wonder everytime if something *did* change, and must go through the hassle of verifying that things stayed the same. You have to understand that you, being the one making these changes, know exactly what is happening and know when and where you have to change things, but for a normal user, they must check every time.

This still does not answer the question of why this belongs in the rc.sysinit. This question was also raised on the mailing list. If hwdetect was it's own package, ala hwd, most of these things would be a non-issue. For instance, look at the steps in the wiki to migrate from hotplug to hwd on boot - it's as simple as "comment this out and add this instead". For hwdetect, it's "comment this out, add in two settings, run this app to see what's going to be loaded, and add things to the blacklist (yes this is very much needed, as my system detects both 8139cp/8139too which causes eth0 to balk)".

As for the initrd changes, I understand it's very simple, and to *me* it isn't a problem. However new users are very much confused (I've seen numerous people questioning the initrd stuff - I simply tell them to use the full image unless they want to optimize the boot). It appears, from an outside perspective, that it's very complicated. I also know two people who cannot boot with even the full image (one has uninstalled the stock kernel and has begun using the archck kernel in community). The key thing here is the lack of the whole (cliche) "KISS principle" - I'm not saying initrd is bad in anyway, it just seems to cause problems for *alot* of people.

As for the last point - I never said hotplug/hwd should be removed, but hwdetect is supposed to superceed them as far as booting goes.

Just to reiterate - I am not here posting this because _I_ am having problems. I am posting based on a general "mood" I've gotten from alot of users. I've received a few PMs and emails from members of the arch community saying they are leaving. And even more questions about why things are so complicated. I'd like to try and do something about it - to try to bring these people back. Many of these thoughts are not even my ideas. I just seem to be the only one who cares that people are leaving. One unnamed person, in an email, told me "I don't want to make waves, but [here's how I feel]" - I have no problems making waves.
I am in no way saying these changes are bad - I am in full support of these changes. I just think they can be handled more transparently than they currently are. Right now, there's a bunch of required steps *after* the install, where most things used to just install with some nice defaults and work fine.
Again, these changes are fine and dandy, but they *need* to "just work" - that is what arch is known for, that is what makes it great.
Comment by Dusty Phillips (Dusty) - Thursday, 01 December 2005, 16:58 GMT
I want to lend support to Aaron's comments. So far, I have had no trouble with the new updates, I simply turned module auto-loading off, and I haven't updated to the initrd kernel so haven't had to deal with this hassle. However, I also do not *want* to have to deal with this hassle. I'm very busy at the moment; the reason I use Arch is it doesn't hassle me. Arch is simple and I always know what it is doing for me. The changes in question, initscripts and kernel do NOT conform to this long-standing philosophy. The module autoloading was supposed to work out of the box but so many people have had trouble with it that I didn't even bother to try it. I recall having to disable hotplug for similar reasons. Having it enabled by default is not in conformance with the Arch philosophy. IIRC, one of the founding principles of Arch was that nothing was enabled unless you intentionally enabled it (fondly recalls the debate over enabling alsa). Like Aaron, I have no problem with the availability of the hwdetect tool; there are many contributed tools that have been made available for a wonderful variety of purposes. I even choose to use some of them. Sadly, I do not want to be forced to use any of them, and worse yet than this, I do not want to be forced to disable them.

With these new changes, I suddenly feel like I am no longer in control of my *default* Arch system. I have to hack it to get what I want. I'm getting the feeling that the design goals of Arch have had a subtle shift. Are we now designing for a new generation of users that are not so famiilar with their systems and not willing to learn about them? This is fine, I am sure, but I appreciated Arch much more when it was designed for me and my peers and I could direct the less interested users to other distributions. I feel if changes continue in this vein I will be left without a distribution.

The above are my own comments. So far, I've kept these ideas to myself, but like Aaron, I've also heard a lot of grumblings from friends and acquaintances in the Arch community. Perhaps these have not reached the ears of the developers because of the increasing size of the gap between userland and devland. I hope this is the case, because it would greatly sadden me if the developers are turning a deaf ear.

Sincerely yours,
Dusty
Comment by Tobias Powalowski (tpowa) - Thursday, 01 December 2005, 17:49 GMT
we have a bugtracker, if stuff doesn'T work why isn't it put there?
shall we smell that a system is not booting ;) ?
what is a lot of ppl?
noone contacted me, nothing is in bugtracker.
tell me names, exact problems then we can investigate the problems and not just alot of ...
is not working ... etc. it's too unprecise and doesn't help at all.
greetings
tpowa
Comment by Travis Willard (Cerebral) - Thursday, 01 December 2005, 18:12 GMT
First off, let me say that I'm impressed by hwdetect - it's very simple and it works great for me.

However, due to its nature - automatic hardware detection (emphasis on automatic) - and the fact that it's already nicely self-contained, I feel it should be packaged on its own, similar to hwd or hotplug, and included with the base packages. This keeps it consistant with other automatic detection solutions available, and can prevent confusion between hwdetect settings like MOD_AUTOLOAD and behaviour of other automatic hardware detection -- I could forsee a situation where a user might expect MOD_AUTOLOAD to stop all autoloading but miss hwd in the DAEMONS.

Additionally, as mentioned already, a seperate package for it would allow it to be updated and version-tracked seperately from the initscripts, which I feel is important -- say, for example, a new change to hwdetect accidentally breaks a handful of people's systems, but they don't want to disable the autoloading until a new version is released, they'll be able to rollback to a version that did work without worrying if anything else important in the initscripts was changed. This would also allow for extra options to be added to the script without the need to change rc.conf should anyone want to extend the script's functionality.

As for initrd, I've been using the archck kernel for ati-drivers compatibility (and 'cause the kernel's pretty darn good) for a long time, and as of yet have been unaffected by this.
Comment by Aaron Griffin (phrakture) - Thursday, 01 December 2005, 18:16 GMT
Before this goes any farther:
tpowa, it seems you are taking personal offense to this - and I apologize if it seems so. I am not here to tell people they are wrong - I am trying to make things better.

As for the last response, it is almost as if you think I am lying - "noone contacted me, nothing is in bugtracker." For whatever the reason, the issues were not logged in the bug tracker. I don't mean to speculate on why these issues were not logged/noted anywhere - that's not my job. I only mean to say that I have heard alot about issues like this - mainly from new users who try the install, and screw everything up after the first update - this "pop in" users aren't going to log anything, they don't care - they were just testing.

Regardless of any of that, you can either assume I'm lying and do nothing, or take it as truth and act on it. I am not pointing a finger. This is not about you. This is about those who have left us, and those who are in the process of leaving.
Comment by Dusty Phillips (Dusty) - Thursday, 01 December 2005, 18:17 GMT
Hey Tobias,

I know you've been posting inappropriately defensive posts to the very long thread to the forums, so you can't sincerely say you unaware of the problems people have had with the script.

The problems people I am talking to are having are not necessarily specific technical problems. There are extremely intelligent people well-versed in the use of Linux. These are much greater problems with philosophy. I'm not going to give names because I'm not sure these people want to be associated with me. Three prominent members of this community have already complained publicly, two TUs and forum advisors, and a former developer and former forum moderator -- Phil, Aaron, and myself. I also know of four others that have complained privately to me or someone I have talked to. Then there are the numerous forum posts I know you've already read, given that its not possible to respond to a post without reading it. And let us not forget the responses to Phil's now-famous mailing list post.

From what I've read, people may be happier if you try these options:
Make hwdetect a daemon instead of built into the initscripts
Use initramfs instead of initrd --> discussed here: http://bbs.archlinux.org/viewtopic.php?t=16800&highlight=initramfs

Dusty
Comment by eliott (cactus) - Thursday, 01 December 2005, 18:44 GMT
I agree with both Aaron and Dusty.
Quite a while ago, I simply put my kernel, and initscripts, into pacman.conf with the very helpful "no upgrade" syntax. I have been debating what to do about it...and largely figured that it was just a case of me having a problem with something that nobody else did. I am glad to see that I am not alone in my concern with the way certain packages (important ones!) are being jiggled around lately.

If the arch philosophy has changed..just let us know. We are all big kids, and we can decide for ourselves what we want to do. But when we are told one thing (simplicity, user control, etc), and are having to deal with something else (automation, autodetection, added complexity, etc)...well, it is a bit confusing to the end users..that is all i can say.

oh. and since this is a bug report, maybe we should take the discussion to the forums? I removed myself from the mailing list a while back...so I don't know if it being carried out there..if so..could someone post the archive thread?

--eliott
<cactus>
Comment by Mike Dill (Theoden) - Thursday, 01 December 2005, 19:08 GMT
I have to begin here by giving a hearty "Amen!" to both Aaron's and Dustry's comments. The feeling that "The changes in question, initscripts and kernel do NOT conform to this long-standing philosophy:, is not an isolated opinion. The KISS principle that has thus far made arch linux the absolute best distro choice for me is seemingly slipping away.

Arch linux has been my only choice for a linux distro for quite some time now precisely because I was not forced to run or accept things that I did not want to, and I had choices that 'I' got to make about it. Now it seems, the choices are being slowly taken away and what remains requires me to hack the system to get back what I used to like about arch. If I have to hack arch to get arch - I can do that with any other distro. I feel like Dusty, that "... if changes continue in this vein I will be left without a distribution."

My personal bugaboo has been the initrd stuff. Yes Tobias, I know it has been documented. And when people have trouble with it anyway, even after carefully checking and double checking their configuration against both the wiki and the instructions found in the forums, they are told:

"Well, it's not that hard - you can figure it out if you try";

or

"Go compile your own kernel then";

or

"Install the archck kernel from community."

Somehow, that kind of response does not breed trust nor send people to the bugtracker. Something is seriously wrong IMHO with the way all of this is being handled and the way those who are having problems are being shunted away like irritating gnats.

As I said, my problem has been with the initrd stuff. When I reboot the system, with either initrd26.img or initrd26-full.img in use, I get thrown into the busybox and that's it - dead. I get "/bin/sh: can't access tty; job control turned off". Nothing I do or change fixes this. But when I raised the issue my problem fell on deaf ears. So yes, I confess to frustration. I finally just compiled my own kernel the old way and things run fine. But I shouldn't have to, except that it appears KISS is a passing concept. And defensive responses from developers only makes it worse. I have the greatest respect for the developers of arch linux. You guys have done a fantastic job. That's why I'm here. But no one is infallible in judgement.

--Mike
<Theoden>


Comment by Judd Vinet (judd) - Thursday, 01 December 2005, 19:50 GMT
Hello all,

First, before I start talking about philosophy and defending my decisions, let me talk about what kind of user I am. I thought most of you knew, but perhaps not. Since Arch was originally intended to be my perfect distro, this may elucidate somewhat.

I like new features and flexibility. I want an operating system that I can do damn well anything with. The initrd opens up new doors for things I want to try. I can have an encrypted LVM root filesystem. That's cool. Sure, it may not mean much to 99% of the people out there, but I think it's dang neat.

While loving new features, I hate dealing with crap that I shouldn't have to (unless my intervention is required). This includes modules. Shit, I could rip *MOD* out of rc.conf and load mine in rc.local, it would be soooo simple. Then all I have to do for each of my seven computers is sit down, cat the PCI bus, pick out the proper modules, and write a custom rc.local for each one. Soooo simple. Rather than spend my free hacking hours on encrypted filesystems and LVM, I can spend it setting up my hardware for these machines.

I know I'm taking things farther than reason here, but it's to illustrate my point. I want the boring stuff abstracted away from me, BUT (big "but" here) with the ability to intervene if the dumb computer can't figure things out on its own. THIS, my friends, is the core philosphy of Arch. The operating system HELPS you, but you are the boss. I don't know if these is equivalent to KISS for you guys or not. I see it as more of a variation of Ockam's Razor than unadulterated "simplicity". I think most of you would agree, or else we wouldn't be here. I'd be the only one running Arch and all of you would be on your own LFS systems, crooning about how simple it all is.

Now, this thread is getting a bit big, but I'll try to tackle the prominent points as I see them. If I miss some, go ahead and remind me.



(*) Why is hwdetect not in the DAEMONS array?

Believe it or not, this was a one-off decision on my part. hwdetect was so simple that I didn't see any advantages to providing this 200-line bash script in it's own package. If any of you see an advantage, please enlighten me. But I'll anticipate some of your replies right now.

- either way, it would run at almost the exact same time in the boot process
- either way, the MODULES array is loaded first, allowing users to force certain module load order
- either way, enabling/disabling it takes a one-line change
- either way, it is not a mandatory part of Arch. The only people that could possibly be using hwdetect right now are people who have manually added MOD_AUTOLOAD="yes" to their rc.conf, or people who have recently installed from FTP.

I have to wonder why you all feel you HAVE to use this. When I added LVM to the initscripts, no one complained. You just didn't use the feature if you didn't need it. Why is this different? Because there's a setting directly in rc.conf?

So to sum up, I could easily convinced to move it into DAEMONS if someone can give me a good solid advantageous reason to do so.



(*) Why the initrd?

Besides saving the hassle of maintaining two kernels, it opens up new possibilities (remember those nifty features I like?) that aren't possible without an initrd. Mostly stuff related to root-device fun, like LVM and encrypted filesystems. Arch likes sane defaults, and mkinitrd.conf has em. The only people that should even need to TOUCH mkinitrd.conf are RAID/LVM users. The defaults should cover everybody else. That means that, for most of us, the only human intervention required is the addition of the initrd line to the bootloader config.

Now I don't know the whole story, maybe there's something else fundamentally wrong with our initrd setup. But as far as module inclusion goes, the mkinitrd defaults are quite sane.

Another interesting thing to note is that at least a couple contributors to this thread have mentioned that they are avoiding the initscripts/kernel updates. To me, this means that you are representing people and problems that you yourself cannot testify to. Be brave, try it out. Maybe I'm just super super intelligent and super-human, but I can still upgrade my kernel, reboot, and continue on my day's work. Maybe I've been blessed with conformant hardware and mundane configurations, but maybe it's not just luck. Give it a shot. Note each manual change you have to make to get things working - maybe we can create a concise list for people wary of the upgrade. Help us.


(*) Why not initramfs?

The honest answer is that I didn't know about it. When I have time, I'll read through and check it out. I like playing with new things. But to be honest, I'm a little scared of making another change like this anytime soon. I wake up each morning to a new slough of emails (from the same 3 or 4 "representatives" of the community) asking for justification for each of the changes we've recently made. We spend a lot of time trying to make these changes as low-impact as we can, but no one seems to notice that part. I can walk over to my roommate's computer, run a --sysupgrade, reboot, and everything Just Works as it always did. hwdetect doesn't even run since I'd have to manually add MOD_AUTOLOAD="yes" to rc.conf.

One more clarification. For me, KISS != no module autodetection. If that's what it means to you, then Arch is not KISS. If this is your idea of KISS, then you could make it so by leaving MOD_AUTOLOAD out of rc.conf and disabling hotplug and hwd.


Now the fun part. What mistakes do I think we've made through this?

- I think we should have kept initscripts in Testing longer. I wasn't consulted before moving initscripts out of Testing, but once in Current, I could really move it back into Testing (cat was pretty far from the bag by then). A week longer in Testing would have saved you the annoyance of all the initscripts updates/fixes.

- I think we could have explained the changes more clearly, so as to stress that very little intervention is required from the user. I think we could have stressed that hwdetect is not a required component of Arch Linux, but that it is there to help with the rather mundane task of loading modules, *IF* one chooses to use it. Chooses.

- I think we could be more patient when helping users. They are smart people and most of their problems are justified. The bug-reporter/bug-fixer relationship is a tenuous one at times. Both sides need to maintain patience and respect through the process. It's also important that language and instructions are clearly written, as well as the problems/symptoms described. Sometimes the reporter doesn't know how to properly report the problem and information related to it, and sometimes the fixer doesn't know how to properly explain the next step in the try-this-and-then-this type of diagnosis.


I think that's all I have for now. I've tried to keep the defensive tone out of my reply, as I know it's not needed here. But please keep in mind that Arch is very important to me. When I feel I am right about decisions made, I will defend them fervently. However, I am big enough to admit mistakes and make changes to right them, if properly convinced.

Also keep in mind that I do indeed HAVE a dayjob so I can't while away too many hours composing replies on the bugtracker. There are bills to be paid, AND work to be done on Arch. My preference leans away from these dialectic discussions and more towards the actual development of Arch.

Thanks for writing, everyone.


- J
Comment by Travis Willard (Cerebral) - Thursday, 01 December 2005, 20:11 GMT
Well spoken, Judd.

I bet I must seem easily swayed, but keep in mind I had no personal problems with it from the beginning; if hwdetect is most likely going to remain relatively static once the initial kinks are worked out, it seems reasonable to have it with initscripts. I was mainly concerned with this trend of frequent updates to the initscripts continuing - if this stops, my stronger argument is dead in the water. ;)

I wonder if setting the default in rc.conf to MOD_AUTOLOAD="no" would satisfy, at least in part, those that don't like it?
Comment by eliott (cactus) - Thursday, 01 December 2005, 20:11 GMT
I appreciate your candor Judd.

For my own personal edification, let me just say that I did try it on my desktop and a test box...had problems, and decided it was not something i wanted to deal with for a while on my more important boxen.

Back to my own job, and my own bills to be paid...
Comment by Mike Dill (Theoden) - Thursday, 01 December 2005, 20:18 GMT
Well - this is all very interesting. But it all fails to address the greater problem of users for whom the changes simply do not work. I don't wish to be uncharitable, but it sounds a lot like, "My mind's made up, so live with it." For those for whom these changes break things and do not work, neither does the explanation. Sorry, but that's how it feels to me.

--Mike
<Theoden>
Comment by Judd Vinet (judd) - Thursday, 01 December 2005, 20:23 GMT
>> I was mainly concerned with this trend of frequent updates to the initscripts continuing

Agreed. It was at least a minor annoyance to most of us, and should not have happened. My apologies for that. It's one of those things that you can't rescind, even if you want to. But we have slowed down on hwdetect now, and it's everyone's (trust me, everyone's ;) hope that we can put it away for awhile.

One thing I forgot to mention was a core reason why hwdetect is better than hwd. Since hwdetect is basically just exploiting knowledge the kernel already has, it can automatically "know" about new modules, even if they're not included in our stock kernel package. For example, hwdetect knows that I need the slamr module (included with the slmodem package) because it shows up in /sys/devices just fine. hwd does not recognize it, however. This is because hwd uses static tables with modules and device IDs, and the maintainers have not included the device ID for my modem chipset.

See the advantage there? If kernel maintainers include a new module for a new device, then hwdetect will just "know" about it. But for it to work with hwd, the hwd maintainers must know that device ID and manually add it to hwd, then we (Arch maintainers) must upgrade hwd with the new static tables. The process is longer and un-needed, how that kernel 2.6 has these modalias exports.


>> For my own personal edification, let me just say that I did try it on my desktop and a test box...had problems, and decided it was not something i wanted to deal with for a while on my more important boxen.

Hehe, very pragmatic of you, Elliot. I maintain a lot of Arch servers too. Some are on the initrd now (no problems) and others aren't. Some days, even though I'm 99% sure of a smooth upgrade, my mood is just not one where I want to deal with the remote possibility of breakage. :)
Comment by Judd Vinet (judd) - Thursday, 01 December 2005, 20:28 GMT
>>> Well - this is all very interesting. But it all fails to address the greater problem of users for whom the changes simply do not work. I don't wish to be uncharitable, but it sounds a lot like, "My mind's made up, so live with it." For those for whom these changes break things and do not work, neither does the explanation. Sorry, but that's how it feels to me.

Hi Mike,

Okay, well let's start working on fixing the problems then. Is it better for me to say "oops, some 5% of users are having inexplicable problems, I'd better revert everything back to normal." First off, that causes the other 95% of people undue aggravation as they have to follow a move-forward, move-backward process.

Perhaps you can help me with the move-forward part of the process. I'm not an avid forum reader. Can you point me at some specific users who are having problems? If you are worried about their anonymity, send me a private email with their usernames.

>>> Arch linux has been my only choice for a linux distro for quite some time now precisely because I was not forced to run or accept things that I did not want to, and I had choices that 'I' got to make about it. Now it seems, the choices are being slowly taken away and what remains requires me to hack the system to get back what I used to like about arch.

This hasn't changed. You want choice? Use your own custom kernel, most power users probably do anyhow. Don't add MOD_AUTOLOAD to rc.conf, or leave it set to "no". What other "hacks" have you been adding to "get back what you used to like" with Arch?
Comment by Aaron Griffin (phrakture) - Thursday, 01 December 2005, 20:41 GMT
Just a few response to the "why make a seperate hwdetect script?" question:
* In hindsight, a seperate package would have prevented a jillion initscripts updates
* Congruency - existing methods are done this way, it makes the change transprent
(by contrast, the LVM stuff had no existing method, so was 100% new)

I admit I have no good response to it that was not outline in you "prediction" list, but think of it this way... if the Apache developers came out and said "well, we just Java here, so we're going to put mod_java into the main Apache code base" - even if you could disable it in some config, it'd still be unsettling.

On a similar note, in order to use hwdetect, I *need* to blacklist 2 things, or else ethernet will not work (8139cp and 8139too both loaded), and I get some odd video issues (intel_agp and agpgart both loaded). To a new user, with the same system, these problems are going to throw them off (admittedly, hotplug did the same thing with the ethernet, but not agp). Having hwdetect become it's own script with it's own configuration, would allow for more complex settings for certain defaults (i.e. conflicting modules - if X and Y load X, loading orders - X before Y, things like that) - it would be nice if some defaults were setup, but that would require rc.conf changes, which I'm assuming people like to ignore, thus the defaults would be ignored.

The initrd things, I have no personal problems with, besides the ones which were glossed over, though they're semantic. An important one, to me, is: Why is the "full" initrd image built on the user's machine? The full image should be built and installed with the main package. This way, a user is not required to have mkinitrd and busybox on his system at all (not a big point, but still important) - the "auto" image can be generated at will *after* a user installs mkinitrd.

As for initramfs - it's honestly a drop-in replacement. The kernel will grab an initrd image and check if it's a cpio archive - if it is, then it uses the initramfs stuff. Otherwise, it loads it like a typical initrd. I did a few minor things in an attempt to port initramfs-tools to arch. It's not that hard, really, I just need to do more testing (and actually reboot with it).
Comment by Tobias Powalowski (tpowa) - Thursday, 01 December 2005, 20:52 GMT
if i offended someone with my repsonse, i apologize for that, it wasn't meant this way.
Comment by Judd Vinet (judd) - Thursday, 01 December 2005, 20:59 GMT
>> * In hindsight, a seperate package would have prevented a jillion initscripts updates

Agreed. Though the end result is the same. A jillion initscripts updates or a jillion hwdetect updates. I know people think an initscripts update means they have to manually examine rc.conf.pacnew and merge changes. But it's not true, do it at your earliest convenience or never at all. Can you think of a time in Arch's history when you HAD to merge in a new setting to rc.conf to maintain functionality? We're pretty good at being backwards-compatible with older versions of config settings.


>> On a similar note, in order to use hwdetect, I *need* to blacklist 2 things, or else ethernet will not work (8139cp and 8139too both loaded), and I get some odd video issues (intel_agp and agpgart both loaded).

Yes, both of these are outstanding issues with hwdetect (and linux modules in general). The 8139cp/8139too issue has been around forever, before tpowa was even a maintainer, I think. I remember him reporting the problem long long ago, and that was with hotplug. So we can say that hwdetect isn't introducing all new module problems, but it's not exactly fixing all of them either.

When tpowa and I have a chance to take off our flame suits, we'll sit down with hwdetect and work on these issues. I think both are fixable. And when we start fixing them, we will do it in Testing first, either as a "hwdetect" package or in initscripts.


>> An important one, to me, is: Why is the "full" initrd image built on the user's machine? The full image should be built and installed with the main package. This way, a user is not required to have mkinitrd and busybox on his system at all (not a big point, but still important

Valid point, Aaron, thank you for that. I don't see why we can't do that. Tpowa, any additions to this, or reasons against?


>> As for initramfs - it's honestly a drop-in replacement. The kernel will grab an initrd image and check if it's a cpio archive - if it is, then it uses the initramfs stuff. Otherwise, it loads it like a typical initrd. I did a few minor things in an attempt to port initramfs-tools to arch. It's not that hard, really, I just need to do more testing (and actually reboot with it).

Excellent. Perhaps we can work on that one together. I haven't had a ton of time lately, but I'd like to explore it.

Of course, there are always timing issues. Do we push for initramfs in 0.7.1 or wait til 0.8?
Comment by Tobias Powalowski (tpowa) - Thursday, 01 December 2005, 21:20 GMT
Hm should be possible to build the full initrd and ship it with it.
then we need a check for mkinitrd in .install file to call it if it's there, for ppl with custom initrd's.
shall i try it Judd?
is the ndevfs patch still needed?
i could upload new kernel on saturday
greetings
tpwoa
Comment by Mike Dill (Theoden) - Thursday, 01 December 2005, 21:20 GMT
Hi Judd:

Okay - you want some names and examples - here you go:

sweiss: Posted: Mon Oct 31, 2005 3:27 am - "Can't boot with this kernel, I get stuck in the useless BusyBox shell. I followed all the hints in the bugtracker so far to no avail, added the proper grub line and changed my fstab to old-style naming scheme."

glad: Posted: Tue Nov 01, 2005 8:39 am - "i updated my kernel yesterday and i added the initrd file to grubs config file but when i reboot my computer i come in to a shell cold busybox what is that?? what did i do wrong"

glad: Posted: Tue Nov 01, 2005 9:46 am - "Mr Green wrote:
"It seems to me that when trying out a new kernel you should always have old one as backup ..."

"Either rename old or new kernel edit grub/lilo & then if something is wrong you can just boot back into old kernel to figure it out ..

my 2c

(Must get round to loading 2.6.14)"

i did but i cant figure out the problem with initrd and kernel 2.6.14"

introspectif: Posted: Wed Nov 02, 2005 4:19 am - "sweiss wrote:
"Can't boot with this kernel, I get stuck in the useless BusyBox shell. I followed all the hints in the bugtracker so far to no avail, added the proper grub line and changed my fstab to old-style naming scheme."

I have this same problem. Any help?"

judland: Posted: Wed Nov 02, 2005 7:45 pm - "Okay... my turn.
I followed the Wiki, regarding the setup of initrd and I still get shunted to the BusyBox prompt. The message I get, just before the # prompt is

"Code:
/bin/sh: can't access tty; job control turned off"

Now, I'm willing to call it 1for initrd, 0 for me and go back to the old kernel"

kakabaratruskia: Posted: Wed Nov 02, 2005 8:25 pm - "judland wrote:

"I followed the Wiki, regarding the setup of initrd and I still get shunted to the BusyBox prompt. The message I get, just before the # prompt is

Code:
/bin/sh: can't access tty; job control turned off"

I have the same problem here... I think it has to do with the new mkinitrd pkg."

orjanp: Posted: Thu Nov 03, 2005 7:32 am - "Is there a way to fix this?"

=====================

That should be sufficient to make the point. The initrd thing IS a REAL problem, not one of user incompetence. And of course Judd, you can add me to that list with the same problem verified on two different machines.

--Mike
<Theoden>
Comment by Tobias Powalowski (tpowa) - Thursday, 01 December 2005, 21:26 GMT
Mike are these problems still valid? the latest mkinitrd script is from 23.11.
does this still show these problems?
Autodetection was turned on some time, now it's disabled by default.
would be really great to fix these issues, if they still exist.
greetings
tpowa
Comment by Judd Vinet (judd) - Thursday, 01 December 2005, 21:29 GMT
Thanks Mike, this is a good start.

I notice the dates on all of these are early November, a month ago. I wonder if these people have since fixed the problems, or just reverted back to an older Arch kernel?

I'll make a forum post to collect all current initrd problems in one place. It may be a more efficient method than having me track these guys down.
Comment by Mike Dill (Theoden) - Thursday, 01 December 2005, 21:30 GMT
Tobias:

I can only speak on that for my issue. It is as current as today. I have this problem right now and that is why I have been adding to this discussion. I downloaded the m kinitrd pkg from testing so it must be the latest I would think.

Thanks for your response Tobias. I appreciate it.

--Mike
<Theoden>
Comment by Mike Dill (Theoden) - Thursday, 01 December 2005, 21:32 GMT
Judd:

Thank you. I didn't intend to sound 'cranky' in my posts. But I have had the feeling that problems are sometimes not taken too seriously unless they are brought forward by someone of 'stature' with the devs - like phrakture or dtw. Anyway, I appreciate it.

--Mike
<Theoden>
Comment by Judd Vinet (judd) - Friday, 02 December 2005, 00:09 GMT
Hi tpowa,

>> shall i try it Judd?

Yes, if you have the time. Don't tag it Testing yet, just commit it so I can have a little peek.

>> is the ndevfs patch still needed?

Not necessary. If it doesn't apply cleanly, you can just leave it aside. It's a convenience, nothing more.

Comment by Rasat (rasat) - Friday, 02 December 2005, 07:24 GMT
>> Comment by Travis Willard (Cerebral) - Thursday, 01 December 2005, 03:11PM
>> I wonder if setting the default in rc.conf to MOD_AUTOLOAD="no" would satisfy, at least
>> in part, those that don't like it?

Yes. Myself I would have had never "complained" in the forum if default had been "no". The "no" stand for the Arch Way. When set to "yes" it went against the philosophy and I was not "psychologically" prepared getting my machine to hang up at booting (hwdetect requires kernel 2.6.12 and above). I think this is the bottom line issue here.

About initrd I cannot comment, haven't studied, but hwdetect is great. Doeas what's already in the newer kernels. Hwd is static depended on a "lazy" maintainer :), so hwdetect is the future. But I would recommend it tries to follow the Arch Way. The Arch Way is *NOT* the written "constitution" of Arch Linux but a guideline why Arch Linux has been appreciated by many users and would like to maintain as such. In short, its about "user-control and freedom of choice", not about hwdetect and initrd. Same applies to any new application.
Comment by Jeff Mickey (codemac) - Saturday, 03 December 2005, 01:30 GMT
I agree with Judd on the hwdetect being in initscripts. Think about it. hwdetect itself is an initscript. Thus, in the initscript package it belongs. That's just my opinion. I don't think the updates have really been that huge of a hassle, you just have those derned .pac* files in /etc. It didn't break anything. If it had broken hotplug or hwd, then I might be more able to understand the hoopla we are making this into. hwdetect not working on your system is NOT your system being broken by hwdetect. You set the MOD_AUTOLOAD, and you decided to give it a try.

As for initramfs and initrd stuff, I feel that we are using initrd because that's all anyone looked into. Initramfs aparantly wasn't looked into. If Judd likes the idea of having initrd standard, then so be it, but to settle for it because it's all you know isn't ever good. So I thank Judd on his understanding of this, and heeding to phrak's suggestion of initramfs.

As a final thought, change is good.
Comment by Tobias Powalowski (tpowa) - Tuesday, 06 December 2005, 08:14 GMT
INITRD in kernel itself is impossible:
Found THE reason for not including the initrd's in kernel, because it is not possible.
You need the root device that can only be detected on user system and not on a developer system.

About initramfs, as far as i understand it,
it's interchangeable, the kernel detects the kind of image, wether it is initrd or initramfs.
The debian utils look a bit complicated atm to me and i have not the time to read that bunch of scripts and docu.
So why not adding initrd setup now and start changing to initramfs after a while, when the devs have more time to have a look at it?

Judd:
we could add this to kernel to track initrd's by pacman:
touch $startdir/pkg/boot/initrd26.img
touch $startdir/pkg/boot/initrd26-full.img

this will result in errors while upgrading, because of existing files,
to solve this:
we could add a pre_install routine to install file that these files were removed before installing/updating.
good idea?

greetings
tpowa
Comment by Aaron Griffin (phrakture) - Tuesday, 06 December 2005, 08:20 GMT
Why do you need to know the root device? It's passed via the bootloader in the root=* kernel param, and should be handled via initrd in the same way the kernel used to do it....
Comment by Brendan Forster (Shifty) - Tuesday, 06 December 2005, 13:46 GMT
I know that some have had their problems with these changes, but I'm quite happy with how hwdetect works.

The original plan was to just upgrade the system and manually attempt to patch all the devfs and initrd changes and cross my fingers. Didn't happen. Some error about being unable to find the root partition in GRUB. No big deal, every cloud has a silver lining.

Testing out the 0.7.1 beta ISO on the system failed to load my ethernet modules on startup after installation. Figured it wasn't something to do with initrd as it was only using defaults. After working out which modules were used (8139too for those playing along at home,the system was trying to install the 8139cp module without success) the manual loading worked fine. Curiosity got the better of me and using the instructions in the Dev blog I got the hwdetect script running. After running the script alongside hotplug, with success, and then disabling hotplug (which in my experience with Arch has been more pain than pleasure, can't wait to see the day that is removed) the system was able to correctly load the ethernet module and it was well on its way to being my file server again! If hwdetect isn't in the final release than I'll just put it in there myself, because Arch has taught me more about Linux and system administration than any other distribution, and hwdetect is so simple and elegant that it would be stupid to not use it on my systems.

I do feel sorry for those who have suffered problems caused by thesen changes. I'm sure that the changes could be fixed by some minor changes, and I hope you don't have to resort to a fresh install like I did. I didn't touch initrd so I can't speak much for that, but I'll certainly look into it now that I'm aware of all the fuss these changes have caused.

Loading...