Arch Linux

Please read this before reporting a bug:
https://wiki.archlinux.org/index.php/Reporting_Bug_Guidelines

Do NOT report bugs when a package is just outdated, or it is in Unsupported. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!
Tasklist

FS#16702 - [linux] Versioned Kernel installs

Attached to Project: Arch Linux
Opened by Harley Laue (losinggeneration) - Friday, 16 October 2009, 20:32 GMT
Last edited by Thomas Bächler (brain0) - Saturday, 30 July 2016, 12:20 GMT
Task Type Feature Request
Category Packages: Core
Status Assigned
Assigned To Tobias Powalowski (tpowa)
Aaron Griffin (phrakture)
Dan McGee (toofishes)
Jan Steffens (heftig)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 189
Private No

Details

The problem:
The current method of upgrading kernels will remove the current kernel, modules, and initrd on upgrades.
This is a problem because: imagine if the new kernel has a serious regression in it on your hardware, or worse yet, it doesn’t boot at all. Now all of a sudden, the user is stuck with no clear, straight forward path to fix the issue.

In a nutshell:
kernel26 will become a dummy package with a dependency on kernel-${version} and symbolic links will be updated post-upgrade to the newly installed kernel/initrd.

Detailed reasoning for this solution:
Every kernel PKGBUILD will have a version number in the pkgname. For instance: instead of it being just kernel26, it’ll now be kernel-2.6.30. kernel26 will now be basically a dummy package that only depends on kernel-2.6.30. When the maintainers go to update the kernel, they'll update the kernel package, and kernel26 package with the new appropriate kernel version number (this will be what? Maybe once a month or so.)

Now, vmlinuz26, kernel26.img will need to obviously be changed. One option would be to automatically upgrade grub/lilo/etc with an appropriate rule. I don't think most people who use Arch would like this option terribly much. So another option would be to have symbolic links in /boot for vmlinuz26->vmlinuz-2.6.30 changed to vmlinuz26->vmlinuz-2.6.31 upon upgrade. This way, the old kernel is still easily accessible on upgrades, and upgrades will still be seamless (and more or less, act like they do now.)

Possible problem:
One problem with this approach would be users will be responsible for periodically removing old kernels. For this, I don't know of a simple solution.

I don't think this one small issue should stop the idea from being shot down entirely.

Ref: http://losinggeneration.homelinux.org/2009/10/16/why-arch-linuxs-kernel-upgrades-suck
This task depends upon

Comment by Aaron Griffin (phrakture) - Friday, 16 October 2009, 20:46 GMT
I'd like some input here. I actually think this is a good idea.

It *does* add maintenance costs - on the dev side: changing the kernel26 dep and renaming the svn dir for each 2.6.X rebuild; and on the user side: manual maintenance in removing the old package when satisfied and changing grub lines.

Additional thought: the kernel26 package can contain dummy symlinks to files in the current kernel package, this way allowing grub and things of the like to remain unchanged for the purpose of tracking the most recent kernel.
Comment by Aaron Griffin (phrakture) - Friday, 16 October 2009, 20:48 GMT
Example:
kernel-2.6.99 contains /boot/vmlinuz26-2.6.99 and builds /boot/kernel26-2.6.99.img
kernel26 package has (broken) symlinks for /boot/vmlinuz26 -> the above, and /boot/kernel26.img -> the above
Comment by Aaron Griffin (phrakture) - Friday, 16 October 2009, 20:49 GMT
Added a reference to the blog post to the original report
Comment by Glenn Matthys (RedShift) - Saturday, 17 October 2009, 15:07 GMT
Didn't we introduce kernel26-lts for issues like this? So people can enjoy a stable kernel?
Comment by Aaron Griffin (phrakture) - Saturday, 17 October 2009, 15:39 GMT
This isn't really about kernel stability. It's more about the fact that if I upgrade my kernel, I can no longer plug in usb devices until I reboot, which is a tad goofy.

Additionally, lts is a different kernel. This mechanism allows for much easier and clearer upgrade paths and testing of new kernels, whereas the lts kernel follows a different path, and releases less often
Comment by Thomas Bächler (brain0) - Saturday, 17 October 2009, 18:21 GMT
lts is way too old, it is not even guaranteed to work right with Arch lon-term (for example, udev will soon break).

I like the idea of having some fallback kernel, and this has been suggested, discussed and dismissed a lot. The problem is that pacman doesn't provide a mechanism to do this cleanly, and the solution suggested will result in a number of small issues we will have to work around, making everything quite complex. In short: I don't like it. I am hesitant to do anything like it without proper support for it in pacman.
Comment by Aaron Griffin (phrakture) - Saturday, 17 October 2009, 23:08 GMT
Do you have an alternate suggestion? I think this one is pretty good, and has a small number of problems which are fairly minor in the grand scheme of things. I'm interested to hear if Dan has any input
Comment by Dan McGee (toofishes) - Sunday, 18 October 2009, 16:38 GMT
Yes, I hit the module loading problem a lot (it really sucks when you need a VPN connection and the tun driver is no longer available). I would love to have some solution to this whole thing.

So a list of things to consider:
* We currently hack the .Y (2.6.X.Y) version out of the kernel; if we do versioned installs do we continue to do this? I don't see how we can. It already seemed a bit odd to me, but I understand the intent (to not have to rebuild external kernel modules). How many of those do we have anymore?
* kernel26 being symlinks makes some sense. How would we pull in the new kernel packages? Bump kernel26 each time to depend on a newer version of the numbered packages?
* It would be trivial, but a cleanup script provided by kernel26 to only keep the last few versions would be nice. We don't see our users as stupid, they can do this as they please. Those with constrained /boot/ partitions will need to be careful, especially since pacman doesn't handle full partitions that well...
* What support from pacman would help here? A concept of a multi-versioned package or something? I can see other cases where this would be helpful (python 2.4, 2.5, etc.) but I'm not sure where we need some extra logic in the package manager. Another meta version parameter?
Comment by Aaron Griffin (phrakture) - Monday, 19 October 2009, 19:18 GMT
@Dan: The idea is that package kernel26 depends=(kernel-2.6.30) [or linux-2.6.30]. On an kernel bump, the kernel-2.6.31 package is added to the repos (svn cp or svn mv) and built. The kernel26 package is bumped to depend on kernel-2.6.31 (most likely when moved to core only). This way, an -Syu will find a kernel26 upgrade, and download the new dependency.

I think the kernel version hackery should stay, and is a separate, unrelated issue. That is more about the EFFORT to rebuild everything, not the impact on the end user.
Comment by Dan McGee (toofishes) - Tuesday, 20 October 2009, 00:57 GMT
So what is the target here? Only major version upgrades? pkgver but not pkgrel bumps? I'm way less for this if that is all we are talking about.
Comment by Aaron Griffin (phrakture) - Tuesday, 20 October 2009, 01:02 GMT
I was imagining something like this:
pkgname=linux-2.6.30
pkgver=2.6.30.1 -> 2.6.30.99

kernel26 pkgver=2.6.30.X depends=(linux-2.6.30)

Then later:
pkgname=linux-2.6.31
pkgver=2.6.31.1

kernel26 pkgver=2.6.31.X depends=(linux-2.6.31)

This allows -Syu's to work exactly as they do now, with the added benefit/hindrance of NOT removing linux-2.6.30 when we upgrade to 2.6.31. That part will be up to the user
Comment by Dan McGee (toofishes) - Tuesday, 20 October 2009, 03:54 GMT
On a side note, our kernel26 PKGBUILD still scares me, it seems overly complex. So instead of beating around the bush, how about something like the attached? Diffing with the current kernel26 PKGBUILD is probably useful as I just hacked that.

Only problem is we slightly lose with the current makepkg, as it doesn't deal with the way we need to do dynamic things very well (note that the function it will end up calling is an alias as bash doesn't support a declare syntax for dynamic function names):

$ makepkg
==> ERROR: missing package function for split package 'linux-2.6.31'
   PKGBUILD (8.6 KiB)
Comment by Mr. K. (KitchM) - Tuesday, 20 October 2009, 06:03 GMT
Thank you, Harley. Its been almost like the emperor's new clothes. Everybody seems to want to ignore the obvious. That this is necessary goes without saying. Imagine how many people are dead in the water after an ill-conceived kernel change. Yes, the only issue is how to implement it.

In this unstable environment, nothing should ever be forever unless the user wishes to deleted the older version. It should always be that the user can go back to the last configuration before the latest upgrade. But wasn't that the purpose of the fallback.img choice on the GRUB menu?
Comment by Harley Laue (losinggeneration) - Tuesday, 20 October 2009, 14:25 GMT
@Dan Yes, the standard kernel26 PKGBUILD is quite complex, but I'm not sure how avoidable that ends up being. If something could be done to work around the limitations of makepkg (and more appropriately, bash) your suggestion may well be a good one (since there would only be one PKGBUILD to maintain.) I wonder if there would be any other larger issues with doing it this way, or like you said before, if other packages (Python 2.4, 2.5, etc) might benefit from being able to do something similar. What I'm getting at is if only the kernel package is going to benefit from a feature/work around like your PKGBUILD, the first way I suggested with two PKGBUILDs may be a more appropriate solution.

@Mr. K. I don't think fallback.img will help at all if the kernel (for whatever reason) doesn't work. fallback.img is just an auto detected initrd that in most cases /should/ work on your system. So if you screwed up an option in /etc/mkinitcpio.conf you'll still be able to boot. For similar reasons, at least one backup kernel/modules is good to have around after a new kernel upgrade. Thus this feature request :)

On a side note, I'm in the process of trying my proposed method above to see how it actually works.
Comment by Harley Laue (losinggeneration) - Tuesday, 20 October 2009, 21:21 GMT
Ok, here's the example pkgbuilds for two kernels, a 2.6.29.2 kernel, and 2.6.30.1 kernel. It also includes kind of what I had in mind for kernel26 called mykernel26 (obviously it's not complete, just to give the idea, and both PKGBUILDs for the two versions are included.) The only thing that really changed between kernel-2.6.29.2 and kernel-2.6.30.1's PKGBUILDs was the md5sums, _basekernel, and pkgver.

The major things I changed from the current PKGBUILD was where the kernel version is declared, most of the time using pkgver instead of kernelname and removed most references to kernel26 in kernel26.install (not a full list, but there really wasn't much more then that.)

Here's a sample listing of what it installs:
kernel-2.6.30.1 /usr/src/linux-2.6.30-ARCH/
kernel-2.6.30.1 /lib/modules/2.6.30-ARCH/
kernel-2.6.30.1 /boot/vmlinuz-2.6.30.1
kernel-2.6.30.1 /etc/mkinitcpio.d/kernel-2.6.30.1.preset

Another question I suppose worth asking is kernel26-firmware, will that need to be updated upon each release as well? That is of course assuming a method like this is adopted.
Comment by Gavin Bisesi (Daenyth) - Wednesday, 11 November 2009, 14:24 GMT
I like the general idea here. This would be a super addition if we can get it implemented well. I can probably help hacking together some maintenance scripts as well.
Comment by Mr. K. (KitchM) - Wednesday, 11 November 2009, 17:05 GMT
Harley, could you please explain in noob terms exactly what this does?

Thanks for the good work.
Comment by Harley Laue (losinggeneration) - Monday, 30 November 2009, 16:40 GMT
@Mr.K. Sorry for the delay. In noob terms what this should do is allow multiple kernel versions to be installed so if a kernel upgrade breaks something, they still have a known working kernel that can be used to boot the system (without the need for a LiveCD or install CD.) It is then left to the user to decide when they want to remove old kernels. (A feature to periodically remove old kernels may be useful, but it's not required.)
Comment by Mr. K. (KitchM) - Monday, 30 November 2009, 20:14 GMT
Oh, yes, very nice! You are so right. That makes a heck of a lot of sense.

In fact, I might add that I had always thought that was the purpose of the option during GRUB boot. If that isn't, then what is the point of GRUB? Who needs a menu when there is no choice. Since I found it wasn't the point, I'm lost with trying to figure out what it does. All very weird.

I've always been used to having a simple menu system that allowed me to boot into the partition I wanted to use. It didn't matter what partition, what OS or combination thereof the user wished to utilize. Then along comes the Arch install that reinvents a wheel that won't roll. I still don't understand the thinking behind this whole thing. How much better it would have been if we had simple gone forward from where we've been for fifteen years. One would add another item to their menu, and go from there. And if the package manager had installed a whole new OS kernel, then one should be able to assume that the fall-back system was standing by and ready to be used as a function of the package manager. We should have always been able to boot into a different (the old) version of an OS. All we need was for users to be able to step back to the previous, working setup. In this case, we don't even need a separate partition, since all is simple software replacement. Instead, someone went to a lot of trouble to make something broken by design. (How much more productive it is when we use our abilities to improve rather than reinvent. What wasted man-hours! But, hey, that's just my opinion.)

I've gone over the dialog here again. There is a huge amount of shorthand-speak that doesn't work in the real world, but I attempted to wade thru it. It appears that the issue can be simply stated as the need to make sure that a kernel does not have dependency problems. Is that the gist of it? Two points: 1. the kernel is supposed to be the lowest common denominator, so that shouldn't be an issue, 2. and in my case, all the upgrades I've done since that kernel was available have not caused any problem when I have reverted back to the older kernel. All is working fine now, including things that didn't work before. I have simply learned to avoid 2.6.30.6-1. I do realize that maybe that is just the chance of this combination of upgrades, but there may be no way to tell. It is at least worth trying another simple backup method.

Instead of changing pacman, why not make a simple reversion script? The purpose would be to take the last setup and put it back as it was. Not being sure of the exact inner workings of pacman, IMHO it may be that pacman would need to have an added function of logging the necessary details of the upgrade so that the added (or associated) reversion script would know what to do. This would be simple add-on script-like coding, rather than a complete rework of the program. I would definately not recommend anything that required any sort of reinvent.

In any case, thanks for the response and clarification. I'm glad I'm not alone in wondering about these things.
Comment by Mr. K. (KitchM) - Monday, 30 November 2009, 20:22 GMT
Oops, that should be 2.6.31.6-1. (Dang, why can't I just edit may comment?)
Comment by Paul Mattal (paul) - Saturday, 06 February 2010, 14:33 GMT
This sounds like a good idea, but nobody has raised their hand to do it. Are we deciding not to implement for now?

If not, what are the next steps?
Comment by Mr. K. (KitchM) - Monday, 08 February 2010, 17:23 GMT
Great question, paul. What do you think, phrakture? Toofishes?
Comment by Michael Henry (drmikehenry) - Saturday, 13 February 2010, 12:59 GMT
Hi,

I'm a very new Arch Linux user (approximately one week so far :-)). After a kernel upgrade, I wanted to revert to the old kernel to see if the graphics performance degradation I was seeing was related to the upgrade. Coming from a long history of other distributions, I was somewhat astonished to find the old kernel had been overwritten instead of preserved. More than any other component in the system, a failed kernel upgrade has the potential to prevent booting. A simple reboot into an old kernel is easy and convenient in other distros, and Grub is tailored for exactly this kind of setup. If the various kernels could be treated as separate packages, it would be easy for a user to periodically use pacman to remove obsolete kernels. I don't know how much impact this has on the larger system, although many of the suggestions in the above comments sound reasonable to my Arch-novice ears. Perhaps Subversion tags could be used to mark the location of older kernel PKGBUILDS, allowing development to continue in the kernel26 directory. These "tags" wouldn't necessarily have to live outside of trunk; they could be dropped peer-level to the kernel26 directory, so they are easily accessible to users via abs. I don't really know enough to make too many implementation suggestions, though. But as a user, I'm very much in favor of supporting the simultaneous installation of multiple kernel versions.

Thanks,
Michael Henry
Comment by Mr. K. (KitchM) - Sunday, 14 February 2010, 17:35 GMT
I couldn't agree more. It is odd that there are so many unnecessary complexities within the subject that seem to create a cloud over the whole thing and stand in the way of progess. We really need to see this in simple terms. I can't help but recognize the truth in one of the things I've learned in the industry; which is that the end-user's impression is usually correct, and that, not only do these problems need to be addressed, but we need to use the KISS method in doing so.

I would suggest that those in the know first explain what is going on in creating the alternative boot setup that we currently have, so that all of us can understand where we stand now. I must admit that I still don't get it.

Thanks.
Comment by Paul Mattal (paul) - Saturday, 06 March 2010, 21:10 GMT
What would we need to (optionally) keep backup copies of the kernel?

It sounds like we'd need to start storing kernels, initcpio images, and other relevant info in directories named for the pkgver and pkgrel. This starts to get complex, doesn't it?

I agree, it would be a great feature to be able to upgrade your kernel and trivially boot back into your old one if something went wrong. Can anyone lay out the full steps we'd need to do to make it happen?
Comment by Mr. K. (KitchM) - Saturday, 20 March 2010, 04:47 GMT
How about this; let's leave it up to disk space. If we have plenty for the boot area, move the old kernel to backup directory. Change GRUB to add extra line. Install new kernel in normal location. If things don't work, one simply chooses last working kernel from GRUB menu. Then the user can run a script to remove new and replace old, changing GRUB in the process. Or something similar. Could symlinks be useful somehow in this process? (I'm just throwing these ideas out there for someone who knows kernels and their installation better than I and can look at these ideas and rearrange them until maybe something fits.)

Sometimes it might be useful to see the OS kernel as just another program to install. We know that it is possible to have more than one version of a particular program run in the same system. The complexity drops when we consider only one backup rather than multiple versions. It must be assumed that the old one was working and then new one was then added. That's only two. We've got that now, although useless, as another GRUB menu item. (Hopefully, we're not talking multiple boot partitions.)
Comment by Davie (daschu117) - Sunday, 18 July 2010, 21:30 GMT
Wouldn't a system like this cause dependency hell with making sure that the appropriate versions of third-party drivers are installed for each retained version of the kernel? All the various graphics and networking drivers come to mind.
Comment by Mr. K. (KitchM) - Sunday, 18 July 2010, 22:32 GMT
Aren't the dependencies actually just a simple list of programs and their appropriate versions in a text file? Then wouldn't that simply be the source for the undo information by comparing the recent one with the similar file in the backup kernel directory? That should actually simplify the whole process so that a simple script can undo what was done.

The only thing we must keep in mind is that the kernel is separate from the other programs which may have been upgraded or installed at the same time. I believe that it is always important to upgrade the kernel separately from the rest of the system, so that the user can reboot and make sure it is working correctly before changing anything else. (I realize that this probably leaves a few things unsaid and assumed, but that's the general idea.)

Perhaps doing a "pacman -Syu" should note that a kernel upgrade exists first and do it separately with a required reboot notification. All other things should halt until this is done.

The alternative is that there is a lack of modularity in Linux, and such must be avoided at all costs. If it might be, as I suspect in this case, we begin to slide down that slippery slope, the whole system begins to be one humugous mess ala micro$oft. We cannot allow such a system to come into being. The issues of diagnosis alone dictate that this cannot be.

Thanks.
Comment by Matthew (piezoelectric) - Thursday, 05 August 2010, 03:07 GMT
Rather than full-blown versioned kernels, what might fit a lot of people's needs would be to at least have the kernel in testing install side-by-side with the core/kernel26.

/lib/modules might not be a problem, except if the same 2.6.X is in testing, then maybe: /lib/modules/2.6.X-TEST

/kernel26-testing.img
/boot/System.map26-testing
/boot/vmlinuz26-testing
/etc/mkinitcpio.d/kernel26-testing.kver
/etc/mkinitcpio.d/kernel26-testing.preset
Comment by Anonymous Submitter - Saturday, 07 August 2010, 16:31 GMT
I think a full-blown versioned kernel is needed. If a kernel does not boot at all (maybe due to wrong config), then initramfs will not help. And if someone does not have a linux live cd/usb handy, then how will those people solve the problem? Simply overwriting old files with the files from the new package is okay even for xorg, kde etc. as the terminal/shell will still be available (init 3) to downgrade the packages? How will you get a working shell with a non-bootable kernel?

Since it is policy of Arch to make the user responsible for the configs, packages etc. it will not be a problem for an Arch user to manually update or remove kernel packages him/herself. Just make sure one kernel package does not overwrite another and leave the maintenance of /boot directory to the user. Arch devs need not worry about whether 1 kernel is installed or 5 or whether the /boot is small for many kernels . Let the user manage those things.

Please version the files in the packages also, like

/boot/vmlinuz26-2.6.35-ARCH
/boot/kernel26-2.6.35-ARCH.img or /boot/initramfs-2.6.35-ARCH.img (similarly for fallback)
/etc/mkinitcpio.d/kernel26-2.6.35-ARCH.kver
/etc/mkinitcpio.d/kernel26-2.6.35-ARCH.preset

I don't think any change is needed for /lib/modules directory
Comment by Anonymous Submitter - Wednesday, 25 August 2010, 07:23 GMT
For versioned kernel installs, pacman should allow installing different version of same package. May be like ignore packages array, a new category of packages for which multiple versions can be allowed, should be created in pacman.conf and pacman source modified accordingly.
Comment by N K (synackfin) - Wednesday, 12 January 2011, 06:20 GMT
I like the idea of having the old kernel stay and not get deleted right away. The details of properly versioning files and such have already been solved by most other distros, so we could just borrow the ideas from them.
Comment by Tom (twall) - Monday, 07 February 2011, 11:39 GMT
I completely agreed to Keshav P R, and i think it the kernel shouldnt be treated as any other program as stated before, because it is crucial for the system to work.

It was also stated before, that the user should not keep track of those things manually, but since arch-linux basically puts all decisions concerning updates to the user, i don't see why this shouldn't apply to the kernel. Maybe it would be possible to let pacman promt the user if he wants to keep old kernels once the user is cleaning the cache. (I mostly clean the cache when the disk is full and when my system is working like a charm)

Actually this problem keeps me from updating my kernel right now, because i need my computer to get some work done, and I don't wont to take the hassle to downgrade the kernel again if something goes wrong. I guess that applies to many users out there.
Comment by Sergej Pupykin (sergej) - Friday, 18 February 2011, 21:14 GMT
I think this idea violates KISS. You can keep lts kernel for recovery issues for example.

I keep tinycore linux in my /boot for recovery - it is nice 10M distro which works completely in initrd and can be booted even if whole arch is broken.

Also I have kernel26 in pacman's IgnorePkg and never update kernel while I have no physical access to PC.
Comment by Harley Laue (losinggeneration) - Friday, 18 February 2011, 22:00 GMT
Sergej Pupykin
"I keep tinycore linux in my /boot for recovery" "Also I have kernel26 in pacman's IgnorePkg and never update kernel while I have no physical access to PC."
The fact that you do this and feel it as a required precaution should pretty strongly reinforce that this is an issue. It's even laughable to consider this (tinycore in /boot) a simpler solution than having a versioned kernel. IMO it's not a fix, it's a work-a-round to the real problem.

Now, if enough users/developers feel this idea isn't following Arch's philosophies close enough, I propose that the kernel26-lts be added to the base group to be installed by default. This still far from ideal, but it'd be a better than nothing. (Don't think of this as conceding defeat. I still strongly believe that versioned kernels are not only a good option, but can even be done in such a way that it follows Arch's philosophies.)
Comment by Ionut Biru (wonder) - Friday, 18 February 2011, 22:06 GMT
having kernel26-lts as default would create more problems than it solves especially when it comes to video drivers that relies on KMS.
Comment by Sergej Pupykin (sergej) - Friday, 18 February 2011, 22:07 GMT
Keeping tinycore in boot is not more difficult than keeping rescue live-cd. All users who had problem once use rescue live disks or alternate boot option.
Comment by Harley Laue (losinggeneration) - Friday, 18 February 2011, 22:54 GMT
lonut
So the kernel26-lts wouldn't be suitable as a stable option for a kernel to recovery boot into upon issues? I hope it didn't look like I was implying for it to be the default kernel to be booted into, rather, a stable kernel that can be relied on to at least always boot. Though, if it's going to cause serious issues, it does nothing for the other case where a versioned kernel would be nice, kernel regressions and driver breakages.

Sergej
I never said anything about it being difficult. It the case of KISS, simple is not defined as easy, it's defined as lacking complexity. Again, I feel that your solution isn't a fix to the issue. It's a way to ignore the issue till it happens, and then work around the issue when it does. It also doesn't do anything for if there's a regression introduced. In that case, all the livecd's and rescue partitions in the world won't be of that much help (because the old kernel is gone.)
Comment by Sergej Pupykin (sergej) - Friday, 18 February 2011, 23:05 GMT
Solution is active kernel testing, so it never breaks anything)

May be repacman (http://aur.archlinux.org/packages.php?ID=35687) can back up kernel before upgrade?
Comment by Ruben Van Boxem (rubenvb) - Tuesday, 19 April 2011, 11:17 GMT
Interesting and heated discussion here. The way I see it:

1. There is a kernel26 (dummy package) will always depend on the latest kernel version available.
2. Installing a kernel upgrade will rename all the old files (kernel26.img etc) to a versioned copy (kernel-2.638.2 os something) so that everythin is still the same to the old Arch user (except for the additional kernel image, which could be deleted by a simple script or tool if the user wants to).
3. Kernel modules will be placed in a versioned directory, which is not deleted anymore when a new kernel is installed.
4. Every time kernel26 is updated, a new package kernel-2.6.x.x is created that has the exact same contents of the old kernel installation. This is the most tricky part: removing this newly added (and automatically installed package, as it was installed as kernel26) should remove every file associated with it. For an example on how this scheme could work: I remember that using the 2010-05 Arch install cd also renames several packages when you first update with pacman. The same renaming can be used here.

Just my two cents :)
Comment by Linas (Linas) - Monday, 02 May 2011, 22:07 GMT
I don't see the need for any renaming there. It can be done just with dependencies.

You create many kernel packages, that place them at eg. /boot/kernels/<version> where installing each of them add an entry to grub, and deleting removes it.

The kernel26 package would have
depends=( kernel26-${_basekernel} )

And the install file would (in addition to any steps to add it to grub) do
post_install() {
ln /boot/{kernels/${_basekernel}/kernel26.img,kernel26.img.new}
ln /boot/{kernels/${_basekernel}/vmlinuz26,vmlinuz26.new}
sync
mv -f /boot/vmlinuz26.new /boot/vmlinuz26
mv -f /boot/kernel26.img.new /boot/kernel26.img
sync
}

Items like /lib/modules/<version>/ would be pinned by the version-specific package, forced by the depends.
You only need to update the vmlinuz26 and kernel26.img hardlinks as they are pointed by grub (you could as
well try to update its reference, but as that may be edited by the user, given how cheap hardlinks are,
it seems easier to do this).
Comment by Anonymous Submitter - Monday, 06 June 2011, 15:45 GMT
Now that officially the next kernel will be 3.0 , I think it is a good time to remove kernel26 naming altogether (aka reconsider implementing any one of the proposals) and move to version specific naming. This will also require change in naming of the initramfs image corresponding to the kernel https://bugs.archlinux.org/task/18719 .
Comment by Kyle Fuller (kylef) - Saturday, 09 July 2011, 21:30 GMT
I think it would be nice if pacman could allow you to install multiple versions of the same package. Now for most packages you wouldn't be able to install multiple versions, but for packages like the linux kernel, and python this would make sense. This should allow making it so that version x of a package doesn't conflict with version y but version z does conflict with version y. For example you have python/2 which doesn't conflict with python/3 but python/2.6 does conflict with python/2.7. I think you could have the syntax in the PKGBUILD like follows:

conflicts=(!python=3.*)

The ! means, "doesn't conflict". The conflict for the kernel26 could look as follows:

conflicts=(!kernel26 kernel26=<3.0)

The above would mean, we don't conflict with any kernel26 version which is more than 3.0, since the older versions of kernel26 might conflict as they were not designed to work with the newer multiple versions system.

This would not only solve the issue for multiple kernel versions, but it will also simplify multiple other packages such as python. The future "linux-lts" package could simply just require a "linux/lts-version" package. The package repositories will need to support multiple versions of a package also, so python/2 and python/3 could be in the repo.

The only issue with multiple kernel versions is with boot loaders because the kernels would have to be in different locations to not conflict. The kernels would be placed somewhere like "/boot/kernels/linux-*" for each version. From what I understand is that there could be a grub2 script or even a grub2 module which could create the boot menu from different installed kernels inside the "/boot/kernels" folder. This would only support grub2. Grub1 and LILO users would need to manually change there configs to reflect new kernels.

For support with LILO/grub1 there could be a package which trials the latest kernel and provides a symlinks and depends upon the real kernel. A user would have 3 options, use grub2 and the automatic menu's, use the kernel trailing package or edit the config themselves. The trailing package could be something like "linux-symlinks", and the new linux package called "linux" (once 3.0 is released).

The only issue is that dependencies would have to support a new syntax, so you can provide a dependency to a certain set of versions such as python2-bla package would depend on python=2.*, if you have python/3 installed the installer should detect and then install the python/2 package alongside python/3.

The package repositores can continue to store python2/PKGBUILD and python/PKGBUILD but the package name inside the package would be the same "python". The linux kernel could adopt a similar pattern.

If a few people like this then I will file it as a complete new bug/feature report for pacman. I have tried to explain this as clearly as I can, I hope it makes sense.
Comment by Bernhard (drahnr) - Thursday, 14 July 2011, 13:37 GMT
@Kyle: Just one point left, how is the user supposed to install the different packages as they are now named the same? I could imagine (but I doubt this will be real) to implement a shallow kind of regex into pacman, so it can easily match pkgname:ver as input (for any kind of versionen sheme). But your effort throws up many questions. Should multiinstall of packages then be enabled for all pkgs? Should we introduce a pacman flag to allow duplicate install? How to handle multiinstall for files in /usr/bin/ /usr/sbin/ /sbin/ and so on...
Comment by Thomas Bächler (brain0) - Thursday, 14 July 2011, 13:50 GMT
Dan and I had a short informal talk recently, and it looks like we can solve this cleanly without having multiple packages of the same name. This requires some changes to the linux package, these changes will probably be done slowly starting with the 3.0 release - most importantly, the different kernel versions must not have file conflicts.

It also requires support in pacman, so that we can automatically remove old versions. This will all go slowly and take a while, just wanted to let you know that we will start a process that will eventually lead to the resolution of this bug report.
Comment by Sergej Pupykin (sergej) - Thursday, 14 July 2011, 14:06 GMT
I think kiss solution is:

$ cp /boot/vmlinuz26 /boot/vmlinuz26.old
$ cp /boot/kernel26.img /boot/kernel26.img.old
$ mv /lib/modules/`uname -r` /lib/modules/`uname -r`.bak
$ pacman -Sy kernel26
$ mv /lib/modules/`uname -r`.bak /lib/modules/`uname -r`
$ reboot

May be add key to pacman --no-delete-only-replace-and-add-new-files instead of 2 mv commands.
Comment by Thomas Bächler (brain0) - Thursday, 14 July 2011, 14:17 GMT
This is not an acceptable solution. This will lead to files being in the system that are not tracked by pacman and have not been installed from any package. This is an absolute no-go and we will never do this (yes, NEVER).
Comment by Marti (intgr) - Thursday, 14 July 2011, 14:19 GMT
brain0, note that Arch is already doing this -- .pacnew and .pacsave files aren't tracked by pacman either.
Comment by N K (synackfin) - Thursday, 14 July 2011, 14:20 GMT
The kernel file should also be versioned (e.g. /boot/vmlinuz26-(ver), and make /boot/vmlinuz26 a symlink to the latest). Thus, two kernel packages would not conflict each other at all except over that one symlink file. Pacman can then have a mode that keeps two packages around so long as the only conflicted files are symlinks (in which case, the symlink is taken from the more recently installed or higher versioned package). The user then manually edits the grub.conf to add older kernels as options. This I think is the KISS approach.

If we want to get fancy, then in addition to the above, we can do something similar to RHEL which updates the grub.conf at every kernel install. Grub would need to have a menuentry auto-added for each installed kernel (and the menuentry auto-removed when a kernel is removed), also like RHEL. The default menuentry would continue pointing to vmlinuz26 (the symlink), only additional grub menuentries would point to specific kernel files. This approach requires zero manual intervention from a user, it will "just work" -- when they reboot, they'll see menus for the latest kernel as well as entries for all the specific versions they kept around. I prefer the KISS approach of not auto-managing grub.conf, but at least creating the kernel packages such that they never conflict with another kernel package except over symlinks.
Comment by Sergej Pupykin (sergej) - Thursday, 14 July 2011, 14:22 GMT
Ok, forget --no-delete-only-replace-and-add-new-files key

But adding kernel26 to IgnorePkg and writing 2 scripts (~/bin/upgrade-kernel.sh and ~/bin/remove-old-kernel.sh to clean up) is easy, simple and covers kernel upgrading issue.
Comment by Thomas Bächler (brain0) - Thursday, 14 July 2011, 14:50 GMT
Marti, those are special files and should probably also be handled by pacman somehow, but this shouldn't be the topic here.

Okay, before tons of bikeshedding starts, here it is: Eventually, each kernel package will have a kernel version like 3.1.X-Y-ARCH where X is the upstream patchlevel, Y is the pkgrel and -ARCH indicates that this is the Arch Linux kernel. This will take care of /lib/modules/* not conflicting. Kernel image and initramfs will have this version appended to its filename.

Now, the packages: The 'linux' package will depend on the following packages: linux-version-3.1.X-Y (this is the pkgname, without version) and linux-version-3.1.X2-Y2. The 'linux' package will also manage symlinks like vmlinuz-current and vmlinuz-old. Now, we need a new feature in pacman: we need to be able to obsolete a package so pacman uninstalls it on its own.

Anyway, this approach is not entirely thought out yet, I still see a problem when someone skips a few kernel versions, and with external modules. As I said, we will transition to this slowly rather than in one step.
Comment by Bernhard (drahnr) - Thursday, 14 July 2011, 15:32 GMT
@Kyle: Just one point left, how is the user supposed to install the different packages as they are now named the same? I could imagine (but I doubt this will be real) to implement a shallow kind of regex into pacman, so it can easily match pkgname:ver as input (for any kind of versionen sheme). But your effort throws up many questions. Should multiinstall of packages then be enabled for all pkgs? Should we introduce a pacman flag to allow duplicate install? How to handle multiinstall for files in /usr/bin/ /usr/sbin/ /sbin/ and so on...
Comment by dan (dangbarry) - Friday, 12 August 2011, 07:50 GMT
Thomas, Just thought i'd throw an idea out there. A script that waits until the nth successful reboot to remove the old kernel. Execute the script through the rc.local. Simply, a successful boot counter. When it's save to proceed, remove the old kernel and the script from the rc.local, until of course next kernel update. That also solves your pacman issue, as the rc.local is launched from root at boot. "uninstalling it on its own".
Comment by Robert Petranovic (jazzfan) - Sunday, 21 August 2011, 12:50 GMT
@dangbarry: I don't think that's good idea. System might not work as expected completely, and the script would increase counter. While user tries to debug it, script will uninstall the good older kernel.

I recommend that there is a config option in pacman.conf: KERNEL_VERSIONS or something simmilar. This should specify how much versions of kernel should be keept.

For example, let's say it's 3. When first 3 kernel versions install, nothing happens. But when 4th kernel update is installing, pacman should remove the first one. This could be done using a workarouond like this in .install file:

# do a standard depmod, mkinitcpio, create symlinks for new kernel to be run as default
#

# move the pacman lock so another instance of pacman can be run.
# This is save because one pacman is just waiting for the second one to finish, so pacman database is safe.
mv /var/lock/pacman.lock{,.bak}

# get number of kernels installed
num=$( ls -1 /boot | grep vmlinuz-linux- | wc -l )
if (( "$num" <= "$KERNEL_VERSIONS" )); then return 0; fi


# get oldest version and remove old kernel
version=$( ls /boot | grep vmlinuz-linux- | sort -n | tail -n 1 )
version=${version%vmlinuz-linux-}

pacman -R linux-$version

# get the lock file back
mv /var/lock/pacman.lock{.bak,}
Comment by Kyle Fuller (kylef) - Sunday, 21 August 2011, 16:45 GMT
@jazzfan, there are two issues with. The first being that a user might have a kernel he wants to keep. Lets say he does this:

pacman -S linux-2.6.32 linux-2.6.33 linux-2.6.34 linux-2.6.35
pacman -S linux

A user who manually installed these packages will not expect them to be removed automatically, but they would be. The other issue is that this may break if linux-lts uses the same system for versioned kernel installs (which it should).

pacman -S linux-lts # This installs linux-2.6.32-1
pacman -S linux # This installs linux-3.0.3-1 (the user already had 3 kernels before this)

The pacman -R will fail due to linux-lts depending on linux-2.6.32.

For these two reasons I think the user should remove a kernel manually and do not do this automatically for a user.
Comment by Nico Schottelius (telmich) - Wednesday, 24 August 2011, 07:41 GMT
Imho, this is pretty easy to solve. Either we do it the Debian way with lazy cleanup or we keep all kernels and have the dummy package depend on the real kernel as described in the first three comments and keep the old packages.

Imho there is no harm in keeping them, we should probably consider a feature like "autoremove" in apt, which removes previously needed, but now unneeded packages, with a hint anyway.

There maybe the problem that a user runs autoremove after installing a new kernel and then removes the current kernel, but I guess this shouldn't be a big issue for arch users.
Comment by Mitchell Richters (mjr4077au) - Sunday, 28 August 2011, 21:12 GMT
I know there's a lot of opinion here and that isn't always productive, but I think the solution is this.

1: Switch to GRUB2. GRUB 0.97 is undeveloped, patched like nothing else and is dead in the water. GRUB2 has grub-mkconfig and will detect everything in /boot, even Windows partitions on another disk with os-detect.
2: Keep one linux package that generates a different initial ramdisk upon installation/upgrade based on whatever version number you want. This will keep the old initial ramdisks available for use since they're not installed with the kernel, they're generated by mkinitcpio.
3: Call "grub-mkconfig -o /boot/grub/grub.cfg" in the kernel upgrade to re-generate the file. If it overwrites the file and someone has manually customised it, shame on them, because thats not how you modify GRUB2. If a user doesn't have GRUB2, the call will simply fail and the user can update their bootloader. If they're not using GRUB2, then they're probably doing something else, so they'd know what they're doing.

I think that's pretty KISS. I personally don't care about 3, because I know Arch is a bit more manual than others and I have no problem with typing a one-liner.


As a sidenote to Sergej, you say "Also I have kernel26 in pacman's IgnorePkg and never update kernel while I have no physical access to PC". Next time you're buying a PC/Server, get something with iLO, you'll have remote access to the entire system with actual video output, including full system BIOS access :).
Comment by Thomas Bächler (brain0) - Sunday, 28 August 2011, 21:34 GMT
Yes, it seems everyone needs to give an opinion here, and none of it is productive.

@Mitchell: As for your opinion - I don't quite understand what you actually want to do, but you seem to think having a different bootloader will solve some problem. You also seem to think that now everyone uses grub and later everyone will use grub2. You also seem to think that regenerating a configuration file on each update using some voodoo script that we have no control over is a KISS solution. Whatever it is you're suggesting, it won't happen.


I am thinking about simply closing this report. It's going nowhere, everyone just posts new "solutions" that we simply won't implement. I posted a game-plan in an earlier comment, which will eventually be implemented. I consider this very low priority, and thus the required changes will be made slowly. Leaving more and more comments with brilliant "ideas" here (which spam my inbox with bugtracker emails) will not speed it up.
Comment by Mitchell Richters (mjr4077au) - Sunday, 28 August 2011, 21:50 GMT
By all means, my opinion is just that. By having grub 2 as a default allows for easier bootloader updates, yes. I stand by having it as a default because it's what it's upstream developed and supported. This it's of topic though.

What I'm ultimately saying is to have versioned initial ramdisks, then new initial ramdisks can be generated upon kernel upgrades.

Lastly, no voodoo script by any means. I am only talking about one line to generate a new config for grub. It wouldn't be any more code than what's used to generate the initial ramdisk upon a kernel upgrade now.

I don't assume everyone to have grub as a bootloader at present, no. but those who don't follow defaults and want to do their own thing would be smart enough to deal with updating their bootloader.
Comment by Mitchell Richters (mjr4077au) - Sunday, 28 August 2011, 21:56 GMT
And as I also said, I myself don't care is grub2 or any bootloader gets upgraded on a kernel upgrade, I can deal with that myself.
Comment by Ernestas (ernetas) - Friday, 18 November 2011, 18:30 GMT
Maybe it would be a great idea to allow not an unlimited number of kernels available in /boot, but only the two most recent. This could be realised by upgrade scripts, just as changing the symlinks.
Comment by alessandro (alelinuxbsd) - Sunday, 29 January 2012, 21:12 GMT
Comment by Thomas Bächler (brain0) - Thursday, 14 July 2011, 14:50 GMT
Dan and I had a short informal talk recently, and it looks like we can solve this cleanly without having multiple packages of the same name. This requires some changes to the linux package, these changes will probably be done slowly starting with the 3.0 release - most importantly, the different kernel versions must not have file conflicts.

It also requires support in pacman, so that we can automatically remove old versions. This will all go slowly and take a while, just wanted to let you know that we will start a process that will eventually lead to the resolution of this bug report.


Could i ask if this particular work is advanced? Just for have a general idea on when this function will be available?

Thanks.


Comment by Jonathan Vasquez (Cows) - Monday, 18 June 2012, 06:18 GMT
This is an issue, but I don't think it's really a big deal.. it really depends the way you look at it.

What I do is I just copy the current kernel with a different name. For example we are currently (at the time of this post) on 3.4.2-2-ARCH.

So I do a `cp /boot/vmlinuz-linuz /boot/kernel-3.4.2-2-ARCH` and `cp /boot/initramfs-linux.img /boot/initramfs-3.4.2-2-ARCH.img` and add the entries into my bootloader. Thus if I ever want to stay on that kernel, I can reboot into the old kernel (or set it as default).

The above method doesn't solve the "Temporarily unable to load my drivers in between kernel upgrades" but it does solve many other issues.
Comment by Vyacheslav Stetskevych (tskevy) - Wednesday, 03 April 2013, 21:17 GMT
I would love to see this feature implemented.
Recently, a kernel upgrade from 3.7.10 to 3.8.3 broke my co-worker's Arch. We had to use a usb drive, unlock LUKS, detect LVM, mount drives, chroot, reinstall 3.7.10.
All would've been simpler, had we the ubuntu-way of choosing older kernels in grub.
Please, think about it.
Thanks.
Comment by Paul Gideon Dann (giddie) - Thursday, 04 April 2013, 09:11 GMT
In the meantime, I'm quite fond of keeping an ArchBoot kernel & initramfs around for quick access to a rescue system. You can add an entry to your bootloader, and so you always have something to fall back on :)

https://wiki.archlinux.org/index.php/Archboot#PXE_booting_.2F_Rescue_system
Comment by Dima Q (dimaqq) - Friday, 13 December 2013, 14:35 GMT
I would very much like to see this feature implemented for a different reason:

Being diligent I run pacman -Syu quite so often.
However I cannot afford to reboot as often.

so here I'm stuck, unable to mount -t cifs //... /..., and why?

uname -a --> 3.12.0-1-ARCH
find /lib/modules/ -name cifs\* --> 3.12.4-1-ARCH

I want at least modules for the currently running kernel to be kept around!

Comment by Thomas Bächler (brain0) - Friday, 13 December 2013, 14:41 GMT
@dimaqq: First of all: IgnorePkg=linux in pacman.conf, please.

We still need a way to install each kernel version under a different file name and automatically update the boot loader afterwards, but I don't see a solution for this (the freedesktop.org bootloader spec is a start, but 1) you'd lose custom kernel arguments on each update, 2) only gummiboot and a patched grub even support the spec right now).
Comment by Paul Gideon Dann (giddie) - Monday, 16 December 2013, 09:53 GMT
Surely we can simply maintain a symlink for the latest kernel in /boot, and point the bootloader at the symlink? That's the traditional method.
Comment by Thomas Bächler (brain0) - Monday, 16 December 2013, 09:56 GMT
It's possible for BIOS folks - for UEFI, /boot will be FAT32 which has no concept of symbolic links.
Comment by Paul Gideon Dann (giddie) - Monday, 16 December 2013, 10:04 GMT
I am booting from UEFI. /boot is ext4, and /boot/efi is FAT. The kernel is in /boot, so a symlink would be fine for me. I have /boot/grub/ symlinked to /boot/efi/efi/grub/. Slightly convoluted, but it works great.
Comment by Thomas Bächler (brain0) - Monday, 16 December 2013, 10:07 GMT
Only works with grub. Grub is the only bootloader that can load kernels from other partitions than the ESP. With all other bootloaders, your kernel must be on the ESP where the firmware can access it directly.
Comment by Dima Q (dimaqq) - Monday, 16 December 2013, 10:18 GMT
I suppose traditional method was to have 2 regular files, vmlinuz (current version) and vmlinuz.old (previous version).

that's not really great either, arch updates kernel quite often and double kernel update will leave the system without a copy of working (currently running) kernel.

I suppose a working solution would look like this:

keep vmlinuz-3.x.y-m-arch # running kernel
keep vmlinuz-3-z-z-m-arch # newest installed kernel
vmlinuz # copy on FAT, symlink on ext*

same for initramfs
same for /lib/modules/xxx
Comment by GI Jack (GI_Jack) - Tuesday, 25 February 2014, 19:45 GMT
I think a better idea is vmlinuz is a symlink to something like vmlinuz310 and vmlinuz312, etc...

This is actually not a bad idea. So the package would be linux310, linux312, etc..
Comment by Paul Gideon Dann (giddie) - Wednesday, 26 February 2014, 10:18 GMT
This has already been suggested. The problem is that some people have a FAT filesystem for /boot, which doesn't support symlinks. It might be possible to get around that by simply copying the file, so long as the filesystem isn't too small.
Comment by Alexandre Bique (babali) - Wednesday, 26 February 2014, 10:21 GMT
God, blocked by FAT ;-)
Comment by Daniel Micay (thestinger) - Sunday, 30 March 2014, 02:28 GMT
In my opinion, the linux-lts package covers this use case well. The ESP partition is usually *really* small and we are going to cause a lot of problems if we fill it with old kernel versions. This is something that users can do if they really want, but I find it to be an overcomplicated, overly magical hack for the package to be involved with an install script.
Comment by Pablo Lezaeta (Jristz) - Wednesday, 23 April 2014, 18:58 GMT
And what about keep only 2 or 3 kernels and remove the oldest ones automagically?
let an example
kernel-3.14.0 provides kernel-3base
kernel-3.14.1 provides kernel-2base
kernel-3.14.2 provides kernel-base

then simply upgrade maintain 1 version of diference and keep only 3.
the drawback is that this will force rebuild of alredy buil packages for changin the kernel-type

or you can copy the Manjaro way and adapt it to Arch (bad idea but somone need propose it)
Comment by Daniel Micay (thestinger) - Wednesday, 23 April 2014, 19:01 GMT
The LTS kernel continues to get security updates and important stability fixes, so it seems more viable for users to install that and use it as a fallback.
Comment by OS Hazard (hazard) - Thursday, 07 August 2014, 02:48 GMT
FYI @falconindy denied the proposed LTS solution.

Additionally, @phrakture and I cannot be the only users irritated by such things as USB storage breaking upon upgrading the kernel. This is due to the old module directory being purged immediately and thus requires a reboot before new devices can be hotplugged.

I propose 3 possible solutions to this annoyance:
a.) Versioned/slotted kernels
b.) Build core drivers into the kernel (why for instance are ext4, btrfs, etc. a module?!?)
c.) Create a hook to optionally keep the old modules directory UNTIL reboot.
Comment by Dima Q (dimaqq) - Thursday, 07 August 2014, 06:53 GMT
I think we need to be specific what is really needed:
* modules for newest installed kernel
* modules for other installed kernels (if allowed)
* modules for current running kernel whether installed or not

It's not only about USB, it's also about all on-demand modules, including firewalls, virtualbox, etc.
Typical scenario is:
* boot, work a bit
* pacman -Syu
* start virtualbox, fail
Comment by GI Jack (GI_Jack) - Sunday, 10 August 2014, 21:17 GMT
Its not that hard to fork module packages, I do some myself for linux-aufs_friendly in AUR.

I volunteer to help maintain packages.

As for kernels, we do it as we had before kernel 3.0, where there was a package kernel26(and a kernel24 when arch first started).

linux34, linux310, linux316.

We don't need patch level granularity as in 3.15.7, just minor version granularity.

I propose supporting all of the longterm kernels as found on the kernel.org homepage, as well as keeping a curret kernel.

I can also volunteer CPU and personal time to work on such if needed.
Comment by Andreas Källberg (Hjulle) - Sunday, 24 August 2014, 10:43 GMT
Bug #28081 makes the lts-solution quite annoying since grub thinks that the lts-version is newer.
Comment by Claire Farron (clfarron4) - Thursday, 30 October 2014, 09:26 GMT
On the topic of LTS kernels supported on kernel.org, I recently got a repository going for them at tiny.cc/clfarron4
But that's not the issue here and it doesn't help solve the one proposed here when you consider that the current version of the kernel (3.17/16) and at least two of the LTS releases (3.14 and 3.10) get updated at the same time.
Comment by Gary van der Merwe (garyvdm) - Monday, 19 January 2015, 11:29 GMT
An alternative solution is to implement systemd offline updates[1]. I would be nice if it only updated packages that need to be done offline, offline. The only one I know of is the kernal, but there may be others. This will require some small additions to pacman.


[1] http://www.freedesktop.org/wiki/Software/systemd/SystemUpdates/
Comment by Claire Farron (clfarron4) - Monday, 19 January 2015, 16:36 GMT
@garyvdm: I wonder if anyone has actually implemented this in practice (it'd be good to study what they've done in code to get it to work)? Have any of the Arch Devs seen this in practice?

If I were to implement something like that, at least for the initial introduction, I'd leave it as a service to be enabled, instead of implementing it as something that just happens.
Comment by Andreas Radke (AndyRTR) - Monday, 20 April 2015, 08:15 GMT
Is anyone brave enough to close this one as "Won't implement"? We have pacman pkg cache and also LTS for fallback. This is working pretty well for almost all users.
Comment by Paul Gideon Dann (giddie) - Monday, 20 April 2015, 09:00 GMT
Well we still haven't solved the problem of the running kernel's modules being removed when installing a new kernel... This can cause issues e.g. when plugging a USB keyboard into a server after remotely installing a new kernel. The kernel is unable to load the relevant module, and the keyboard is unusable. It's quite a confusing situation for someone who doesn't realise why it's happening.
Comment by Andreas Källberg (Hjulle) - Monday, 11 May 2015, 14:53 GMT
LTS for fallback is also not sufficient if both the normal version and the LTS-kernel was updated at the same time (or at least between your updates). You could of course manually postpone upgrading the LTS kernel, but that makes upgrading more convoluted.
Comment by Amos Onn (amos) - Tuesday, 02 June 2015, 21:32 GMT
Another issue which might affect some users is that hibernate (to disk) breaks if you boot a different kernel than then one which performed the hibernate. i.e. you upgrade kernel, hibernate, and then when you power on the new version of kernel doesn't manage the loading well and crashes. This was at least true somewhere at the beginning of 3.x, I haven't tried in a long time.
Comment by Jason (replica9000) - Thursday, 27 August 2015, 18:44 GMT
Coming from Debian, I am used to having multiple kernels installed. Having a known working kernel is important in case something doesn't work with the new kernel.

As I have my Arch install on ZFS from the AUR, I wouldn't want to be without my data due to some possible issue with modules not building, or being unavailable until a reboot. (I may go months without a reboot.)

I have currently installed a kernel from the ABS. I've simply edited the PKGBUILD to change pkgbase=linux to pkgbase=linux-4.1.6-1, and built the package(s) as usual. The result was being able to install a new kernel and headers without overwriting the old ones. I've also created an empty dummy linux package for compatibility/dependencies.

Now obviously that means that building new modules such as ZFS means pointing the dependency to linux-x.x.x instead of just linux. I've also changed the module names in PKGBUILD to match the kernel it was build for (zfs-git-4.1.6-1). This means I can, if necessary, boot into the older kernel, and still having working modules for it too.

I know this sounds like extra work, but we use Arch because we aren't afraid of that right??

Perhaps instead of having AUR packages look for a specific version of the kernel, why not specify the kernel to build against during makepkg? (I had to fake uname -r to report the new kernel instead of the running kernel.)

makepkg --option <kernel-version>
-- or --
export KERNELVERSON=4.x.x
makepkg

For now, I'll keep experimenting with this to see how to runs in the long term.
Comment by Jason (replica9000) - Saturday, 29 August 2015, 15:41 GMT
doublepost :(
Comment by Dennis Schridde (devurandom) - Tuesday, 02 February 2016, 08:35 GMT
systemd comes with the `kernel-install` tool, which takes a kernel version and image as argument and then runs /etc/kernel/install.d and /usr/lib/kernel/install.d on it. The latter contains 50-mkinitcpio.install, which compiles a initramfs for the given version and places it in /boot/$machine-id/$version/initrd, as well as 90-loaderentry.install, which will copy the given kernel image to /boot/$machine-id/$version/linux and create a versioned entry in /boot/loader/entries. This is, as far as I understand, what you want.
Comment by Dima Q (dimaqq) - Tuesday, 02 February 2016, 09:16 GMT
Integration of `kernel-install` is probably a good idea.
There's more to this ticket than kernel in /boot though:

Most importantly, /usr/lib/modules/<version> must be preserved.
Ideally, DKMS should also be built for all preserved versions.

IMO, entire `linux` package content should be available for a) current version, and b) every version installed into /boot
Comment by Alexandre Bique (babali) - Tuesday, 02 February 2016, 10:57 GMT
That systemd kernel install sounds like a great idea!
Comment by ITwrx (andriesinfoserv) - Wednesday, 08 June 2016, 23:34 GMT
from an end user's perspective: I'd be in favor of anything (within reason) that doesn't require manual removal of old kernels. Some other well known distros will happily fill up the boot partition until a novice user can't update anymore and dependencies are borked. I understand arch is not for novices, but if a piece of software can install multiple kernels, it can clean up after itself too. Requiring pre config is fine. Periodic manual intervention, not so much. The current way arch handles it is something i've been happy about for years, though i have seen issues with modules so i understand trying to fix that.
Comment by Alif (alive4ever) - Sunday, 12 June 2016, 07:34 GMT
I suggest to add "old" suffix to the old kernel and initramfs files for the last kernel. So that, when a new kernel update appears, there will be vmlinuz-linux-old, initramfs-linux-old.img, and initramfs-linux-fallback-old.img that can be booted after updating grub.cfg entry. This can be triggered by libalpm hooks (pre-transaction and post-transaction).
Comment by Behnam Momeni (sbmomeni) - Thursday, 21 July 2016, 07:12 GMT
Here is an alternative scheme which keeps N latest kernel versions installed and removes all older kernels automatically (by pacman):

1. Each kernel version has its own version-named package as described above (e.g. linux-1.2.3.4-1.2.3.4-1 where linux-1.2.3.4 is the pkgname and the second 1.2.3.4 is the pkgver),
2. Each one of these version-named packages will "provide" the linux-real=${version} package. That is, although each version-named package looks like a completely different package, it also provides a version of a common named target (i.e. linux-real),
3. A dummy package like linux-virtual-${version} will be created which always depends on the latest linux-real-${version} package and also contains the above mentioned symlinks. This installs the new kernel version as its dependency and leaves older versions installed as unused dependencies,
4. The linux-virtual dummy package also "conflicts" with the "linux-real<=${N older version as you wish}" and so all older kernel versions which provide those older linux-real targets are found by pacman and will be removed after getting a y/N confirmation from the user.

This requires no special maintenance script for removing old kernels nor any special support from the pacman.
Comment by Ernestas (ernetas) - Thursday, 21 July 2016, 07:39 GMT
I'm surprised nobody has mentioned this yet, but since Linux 4.0, kGraft and kpatch were included in the kernel. Not to mention ksplice, which has been available for years. While having multiple kernel versions installed seems important right now, I think that live kernel updates is the future.
Comment by Alexandre Bique (babali) - Thursday, 21 July 2016, 09:24 GMT
So how does the live kernel update work? Can we do it right now on Archlinux?
Is that thing well tested?

From my first thoughts I would not do that automatically, but I'd love to have a command that could do it.
Comment by Daniel Micay (thestinger) - Thursday, 21 July 2016, 09:43 GMT
Live updates are aimed at making small security fixes. It's not a way to do arbitrary upgrades, and doesn't simply Just Work without work to create the live patches. I also don't really see how it's relevant to keeping multiple kernel versions installed...
Comment by Andreas Radke (AndyRTR) - Thursday, 21 July 2016, 15:17 GMT
We now have 3 official kernels in our repos: stick kernel, -lts and -zen. I'm for closing this bug
as nobody seems to be interested in implementing this and there's no real need to have fallback kernels of
the same branch.
Comment by Alexandre Bique (babali) - Thursday, 21 July 2016, 15:21 GMT
No way to close that bug :)
I don't think that no one is interested in implementing it, I think that we just didn't come to an agreement on the solution to this issue.
Comment by Dennis Schridde (devurandom) - Thursday, 21 July 2016, 15:27 GMT
While there might be no real need for YOU to have a copy of the running kernel in the filesystems, there is definitely a strong need for others.

E.g. read my comment above (https://bugs.archlinux.org/task/16702#comment143419). When you have a kernel in /boot and EFI is setup to start it (e.g. using gummiboot or systemd-boot), but the /lib/modules/${KV} directory associated with it is gone, the boot will fail. There will be no option left to boot into the system and one has to boot from another medium (e.g. usb drive) to fix this up by copying the newly installed kernel image into the right place (the place expected by EFI/gummiboot/systemd-boot).
Comment by Radek Podgorny (rpodgorny) - Thursday, 21 July 2016, 15:30 GMT
i think this bug report is not really about having lots of different kernels installed simultaneously. it's about retaining the running kernel and its modules. otherwise this can happen:

1) boot up kernel X
2) update to kernel X+1
3) plug in usb stick/mount some filesystem/try to load any module not yet loaded
4) cry after you find out old modules (for the running kernel) are gone and you need to reboot
Comment by Amos Onn (amos) - Thursday, 21 July 2016, 15:35 GMT
Another reason for retaining the running kernel is for hibernating - otherwise you hibernate, but when powering up, the new kernel version tries to restore the state of an older version and fails.

But still, it's just about the running kernel, not about multiple kernels.
Comment by Andreas Radke (AndyRTR) - Thursday, 21 July 2016, 15:37 GMT
After 7 years none of the kernel maintainers or other developer stepped in and wanted to implement this.

This won't change anytime soon. So "Won't implement" and closing this one still makes sense to me.
Comment by Alexandre Bique (babali) - Thursday, 21 July 2016, 15:58 GMT
I can implement it. It is not a very complicated task.
Do we have any specification on how it is supposed to be implemented?
If not, then what i can do is:
- a systemd init script which copies the current modules into /lib/modules/current + cleanup of the old symlinks to current
- a post uninstall hook for the kernel which will detect and symlink if necessary 4.6.4-1-ARCH/ to current/

Is that solution OK?
Comment by Behnam Momeni (sbmomeni) - Thursday, 21 July 2016, 16:06 GMT
You do not need to write any script for retaining the current version modules. The pacman can handle it automatically.
Read my comment (https://bugs.archlinux.org/task/16702#comment149162).
Comment by Eli Schwartz (eschwartz) - Thursday, 21 July 2016, 16:19 GMT
Comment by Thomas Bächler (brain0) - Thursday, 14 July 2011, 09:50 GMT-5

Okay, before tons of bikeshedding starts, here it is: Eventually, each kernel package will have a kernel version like 3.1.X-Y-ARCH where X is the upstream patchlevel, Y is the pkgrel and -ARCH indicates that this is the Arch Linux kernel. This will take care of /lib/modules/* not conflicting. Kernel image and initramfs will have this version appended to its filename.

Now, the packages: The 'linux' package will depend on the following packages: linux-version-3.1.X-Y (this is the pkgname, without version) and linux-version-3.1.X2-Y2. The 'linux' package will also manage symlinks like vmlinuz-current and vmlinuz-old. Now, we need a new feature in pacman: we need to be able to obsolete a package so pacman uninstalls it on its own.

Anyway, this approach is not entirely thought out yet, I still see a problem when someone skips a few kernel versions, and with external modules. As I said, we will transition to this slowly rather than in one step.

----------------------------------------------------------------------------------------------------------

So what is the actual holdup on this?

As far as I can see, this can be done today, as long as users are responsible for uninstalling their own kernels (which is not an unreasonable idea).

Do we just need (questionable) support in pacman for obsoleting a package according to user-defined rules?
If so, has anyone opened a ticket against pacman and/or discussed it on the pacman-dev mailing list?

Because an excellent way to make sure it never happens is to forget all about it and then say no one is interested in it because no one is sure what they actually want.
Comment by Behnam Momeni (sbmomeni) - Thursday, 21 July 2016, 16:33 GMT
It is possible to obsolete old kernel versions with the current pacman.
It is as simple as defining a "conflict" rule from the virtual linux package to the older real linux packages.

What if a user skipped some kernel versions?
Define the "conflict" rule to N (e.g. 10) older versions and it will work flawlessly for users who are not N versions behind the Arch release cycle.
Comment by Eli Schwartz (eschwartz) - Thursday, 21 July 2016, 16:46 GMT
Yeah, for your arbitrarily chosen target users and your arbitrarily chosen number of manually-specified older kernels, which kind of defeats the purpose.

... I still don't see what is wrong with requiring users to manage their own orphaned packages, in which case this can be implemented as soon as someone decides they want to do it.
But if automatic purging of old kernels is what is stopping this from being implemented, it would be nice to at least know that that is the reason.
Comment by Amos Onn (amos) - Thursday, 21 July 2016, 17:38 GMT
It seems to me that people are trying to address two different (though similar) problems:

One is the inability to have multiple versions of the kernel at the same time. This is a problem in case you're not sure the new version works, and want an old one to fall back on.
The other is the inability to leave a copy in the filesystem of only the *currently running* kernel and modules. This is a problem for loading new drivers after an upgrade, or for hibernating and resuming system with the same kernel.

In order to solve the first problem, something akin to having a different package for each kernel version is required; and then there is the question of how to manage the old versions, via package conflicts, obsolete package feature for pacman, or just leaving it to the users.
In order to solve the second problem, a script which saves the current kernel and modules at init, perhaps with a bit more (see babali's comment https://bugs.archlinux.org/task/16702#comment149179) is enough.

Now, the question is, which of the two problems are we trying to solve? It is true that a solution for the first one also solves the second one, but is probably more complicated (hence the stalling).

I, for one, think solving the second problem is enough; and as AndyRTR commented here https://bugs.archlinux.org/task/16702#comment149173, for the fallback issue you have -lts kernel versions.
Comment by Radek Podgorny (rpodgorny) - Thursday, 21 July 2016, 18:11 GMT
i totally agree (with solving just the second problem). ...adding the need for manual uninstalling of old kernels would be a step back, imho. (for example, many of my systems are quite limited on disk space)
Comment by Vyacheslav Stetskevych (tskevy) - Thursday, 21 July 2016, 18:25 GMT
I'd like to
1) have all new kernels installed by default (i.e. have the linux package depend on the latest real kernel package).
2) have a post-install hook which will remove kernel packages, with a configuration file to specify how many releases of the kernel it should keep. It should always leave both the latest and the current running kernel in place.
What do you think?
Comment by Eli Schwartz (eschwartz) - Thursday, 21 July 2016, 18:46 GMT
Automatic uninstalling of old versioned kernels requires either ugly usage of the conflicts field, or else extending pacman. (Post-install hooks cannot invoke pacman, since the database will be locked by the parent pacman process.)

Copying the kernel to outside of pacman's bookkeeping is ugly and not simple at all, besides which in order to actually use it in any way you'd need to copy it right back. In other words, you are literally trying to trick pacman into thinking a package is uninstalled while leaving the files exactly where they were.

...

As per the original intent of this ticket, the solution is *still* versioned kernel installs.
So-called "simpler" solutions are in fact more complicated, and IMHO against the Arch philosophy.

So, it would be nice to hear from the kernel maintainers if "automatic obsoletion of old kernels, possibly according to user-defined rules" is the only blocker.

Alternatively, bikeshedding is more fun than solutions, so carry on.
Comment by Behnam Momeni (sbmomeni) - Thursday, 21 July 2016, 19:08 GMT
I filled a feature request for the pacman package (https://bugs.archlinux.org/task/50121) which allows installing the latest kernel version while keeping the current kernel version even if the user has skipped N (unknown) kernel versions.
Comment by Amos Onn (amos) - Thursday, 21 July 2016, 19:26 GMT
Not all discussion is bikeshedding. Of course a plethora of solutions, minutely different from each other, can be suggested and would get us nowhere; but I think that discussing the options we have broadly, not to mention the requirements of a solution, does help dispelling our current "no one is sure what they actually want" situation, to quote you. If we manage to break down and formulate a few (two-to-three) options with their merits and downsides, then it will be much easier to choose one of them and implement it. It seems to me that we have now two main (categories of) approaches, and what we need is to flesh them out.

I agree that copying files outside of pacman's bookkeeping is not the most elegant of solutions, but it is a possible one. There are precedents for unowned symlinks, such as /etc/localtime, admittedly more in configuration than in binaries. Having a temporary copy of the current modules doesn't sound that bad, perhaps it can even sit outside /usr/lib/modules.

That said, I agree that having multiple kernel versions is definitely also a reasonable solution, and that Arch users should be able to manually manage them. At the very least, it is also possible to have a linux-latest package, so users which don't want to have this feature can go on with the current way of things. So, I second the question whether outside of the obsoletion managament, there are no problems with implementing this.
Comment by Dennis Schridde (devurandom) - Thursday, 21 July 2016, 20:58 GMT
You could install multiple kernel packages, each carrying the version in their name, and then use something like Gentoo's eclean-kernel, which is a script that scans /boot for old kernels and removes them. With Arch Linux binary packages of kernels you could instead just call pacman -R from that script.

[1]: https://packages.gentoo.org/packages/app-admin/eclean-kernel

Loading...