FS#16702 - [linux] Versioned Kernel installs

Attached to Project: Arch Linux
Opened by Harley Laue (losinggeneration) - Friday, 16 October 2009, 20:32 GMT
Last edited by Buggy McBugFace (bugbot) - Saturday, 25 November 2023, 20:22 GMT
Task Type Feature Request
Category Packages: Core
Status Closed
Assigned To Tobias Powalowski (tpowa)
Jan Alexander Steffens (heftig)
David Runge (dvzrv)
Levente Polyak (anthraxx)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 314
Private No

Details

The problem:
The current method of upgrading kernels will remove the current kernel, modules, and initrd on upgrades.
This is a problem because: imagine if the new kernel has a serious regression in it on your hardware, or worse yet, it doesn’t boot at all. Now all of a sudden, the user is stuck with no clear, straight forward path to fix the issue.

In a nutshell:
kernel26 will become a dummy package with a dependency on kernel-${version} and symbolic links will be updated post-upgrade to the newly installed kernel/initrd.

Detailed reasoning for this solution:
Every kernel PKGBUILD will have a version number in the pkgname. For instance: instead of it being just kernel26, it’ll now be kernel-2.6.30. kernel26 will now be basically a dummy package that only depends on kernel-2.6.30. When the maintainers go to update the kernel, they'll update the kernel package, and kernel26 package with the new appropriate kernel version number (this will be what? Maybe once a month or so.)

Now, vmlinuz26, kernel26.img will need to obviously be changed. One option would be to automatically upgrade grub/lilo/etc with an appropriate rule. I don't think most people who use Arch would like this option terribly much. So another option would be to have symbolic links in /boot for vmlinuz26->vmlinuz-2.6.30 changed to vmlinuz26->vmlinuz-2.6.31 upon upgrade. This way, the old kernel is still easily accessible on upgrades, and upgrades will still be seamless (and more or less, act like they do now.)

Possible problem:
One problem with this approach would be users will be responsible for periodically removing old kernels. For this, I don't know of a simple solution.

I don't think this one small issue should stop the idea from being shot down entirely.

Ref: http://losinggeneration.homelinux.org/2009/10/16/why-arch-linuxs-kernel-upgrades-suck
This task depends upon

Closed by  Buggy McBugFace (bugbot)
Saturday, 25 November 2023, 20:22 GMT
Reason for closing:  Moved
Additional comments about closing:  https://gitlab.archlinux.org/archlinux/p ackaging/packages/linux/issues/10
Comment by Aaron Griffin (phrakture) - Friday, 16 October 2009, 20:46 GMT
I'd like some input here. I actually think this is a good idea.

It *does* add maintenance costs - on the dev side: changing the kernel26 dep and renaming the svn dir for each 2.6.X rebuild; and on the user side: manual maintenance in removing the old package when satisfied and changing grub lines.

Additional thought: the kernel26 package can contain dummy symlinks to files in the current kernel package, this way allowing grub and things of the like to remain unchanged for the purpose of tracking the most recent kernel.
Comment by Aaron Griffin (phrakture) - Friday, 16 October 2009, 20:48 GMT
Example:
kernel-2.6.99 contains /boot/vmlinuz26-2.6.99 and builds /boot/kernel26-2.6.99.img
kernel26 package has (broken) symlinks for /boot/vmlinuz26 -> the above, and /boot/kernel26.img -> the above
Comment by Aaron Griffin (phrakture) - Friday, 16 October 2009, 20:49 GMT
Added a reference to the blog post to the original report
Comment by Glenn Matthys (RedShift) - Saturday, 17 October 2009, 15:07 GMT
Didn't we introduce kernel26-lts for issues like this? So people can enjoy a stable kernel?
Comment by Aaron Griffin (phrakture) - Saturday, 17 October 2009, 15:39 GMT
This isn't really about kernel stability. It's more about the fact that if I upgrade my kernel, I can no longer plug in usb devices until I reboot, which is a tad goofy.

Additionally, lts is a different kernel. This mechanism allows for much easier and clearer upgrade paths and testing of new kernels, whereas the lts kernel follows a different path, and releases less often
Comment by Thomas Bächler (brain0) - Saturday, 17 October 2009, 18:21 GMT
lts is way too old, it is not even guaranteed to work right with Arch lon-term (for example, udev will soon break).

I like the idea of having some fallback kernel, and this has been suggested, discussed and dismissed a lot. The problem is that pacman doesn't provide a mechanism to do this cleanly, and the solution suggested will result in a number of small issues we will have to work around, making everything quite complex. In short: I don't like it. I am hesitant to do anything like it without proper support for it in pacman.
Comment by Aaron Griffin (phrakture) - Saturday, 17 October 2009, 23:08 GMT
Do you have an alternate suggestion? I think this one is pretty good, and has a small number of problems which are fairly minor in the grand scheme of things. I'm interested to hear if Dan has any input
Comment by Dan McGee (toofishes) - Sunday, 18 October 2009, 16:38 GMT
Yes, I hit the module loading problem a lot (it really sucks when you need a VPN connection and the tun driver is no longer available). I would love to have some solution to this whole thing.

So a list of things to consider:
* We currently hack the .Y (2.6.X.Y) version out of the kernel; if we do versioned installs do we continue to do this? I don't see how we can. It already seemed a bit odd to me, but I understand the intent (to not have to rebuild external kernel modules). How many of those do we have anymore?
* kernel26 being symlinks makes some sense. How would we pull in the new kernel packages? Bump kernel26 each time to depend on a newer version of the numbered packages?
* It would be trivial, but a cleanup script provided by kernel26 to only keep the last few versions would be nice. We don't see our users as stupid, they can do this as they please. Those with constrained /boot/ partitions will need to be careful, especially since pacman doesn't handle full partitions that well...
* What support from pacman would help here? A concept of a multi-versioned package or something? I can see other cases where this would be helpful (python 2.4, 2.5, etc.) but I'm not sure where we need some extra logic in the package manager. Another meta version parameter?
Comment by Aaron Griffin (phrakture) - Monday, 19 October 2009, 19:18 GMT
@Dan: The idea is that package kernel26 depends=(kernel-2.6.30) [or linux-2.6.30]. On an kernel bump, the kernel-2.6.31 package is added to the repos (svn cp or svn mv) and built. The kernel26 package is bumped to depend on kernel-2.6.31 (most likely when moved to core only). This way, an -Syu will find a kernel26 upgrade, and download the new dependency.

I think the kernel version hackery should stay, and is a separate, unrelated issue. That is more about the EFFORT to rebuild everything, not the impact on the end user.
Comment by Dan McGee (toofishes) - Tuesday, 20 October 2009, 00:57 GMT
So what is the target here? Only major version upgrades? pkgver but not pkgrel bumps? I'm way less for this if that is all we are talking about.
Comment by Aaron Griffin (phrakture) - Tuesday, 20 October 2009, 01:02 GMT
I was imagining something like this:
pkgname=linux-2.6.30
pkgver=2.6.30.1 -> 2.6.30.99

kernel26 pkgver=2.6.30.X depends=(linux-2.6.30)

Then later:
pkgname=linux-2.6.31
pkgver=2.6.31.1

kernel26 pkgver=2.6.31.X depends=(linux-2.6.31)

This allows -Syu's to work exactly as they do now, with the added benefit/hindrance of NOT removing linux-2.6.30 when we upgrade to 2.6.31. That part will be up to the user
Comment by Dan McGee (toofishes) - Tuesday, 20 October 2009, 03:54 GMT
On a side note, our kernel26 PKGBUILD still scares me, it seems overly complex. So instead of beating around the bush, how about something like the attached? Diffing with the current kernel26 PKGBUILD is probably useful as I just hacked that.

Only problem is we slightly lose with the current makepkg, as it doesn't deal with the way we need to do dynamic things very well (note that the function it will end up calling is an alias as bash doesn't support a declare syntax for dynamic function names):

$ makepkg
==> ERROR: missing package function for split package 'linux-2.6.31'
   PKGBUILD (8.6 KiB)
Comment by Mr. K. (KitchM) - Tuesday, 20 October 2009, 06:03 GMT
Thank you, Harley. Its been almost like the emperor's new clothes. Everybody seems to want to ignore the obvious. That this is necessary goes without saying. Imagine how many people are dead in the water after an ill-conceived kernel change. Yes, the only issue is how to implement it.

In this unstable environment, nothing should ever be forever unless the user wishes to deleted the older version. It should always be that the user can go back to the last configuration before the latest upgrade. But wasn't that the purpose of the fallback.img choice on the GRUB menu?
Comment by Harley Laue (losinggeneration) - Tuesday, 20 October 2009, 14:25 GMT
@Dan Yes, the standard kernel26 PKGBUILD is quite complex, but I'm not sure how avoidable that ends up being. If something could be done to work around the limitations of makepkg (and more appropriately, bash) your suggestion may well be a good one (since there would only be one PKGBUILD to maintain.) I wonder if there would be any other larger issues with doing it this way, or like you said before, if other packages (Python 2.4, 2.5, etc) might benefit from being able to do something similar. What I'm getting at is if only the kernel package is going to benefit from a feature/work around like your PKGBUILD, the first way I suggested with two PKGBUILDs may be a more appropriate solution.

@Mr. K. I don't think fallback.img will help at all if the kernel (for whatever reason) doesn't work. fallback.img is just an auto detected initrd that in most cases /should/ work on your system. So if you screwed up an option in /etc/mkinitcpio.conf you'll still be able to boot. For similar reasons, at least one backup kernel/modules is good to have around after a new kernel upgrade. Thus this feature request :)

On a side note, I'm in the process of trying my proposed method above to see how it actually works.
Comment by Harley Laue (losinggeneration) - Tuesday, 20 October 2009, 21:21 GMT
Ok, here's the example pkgbuilds for two kernels, a 2.6.29.2 kernel, and 2.6.30.1 kernel. It also includes kind of what I had in mind for kernel26 called mykernel26 (obviously it's not complete, just to give the idea, and both PKGBUILDs for the two versions are included.) The only thing that really changed between kernel-2.6.29.2 and kernel-2.6.30.1's PKGBUILDs was the md5sums, _basekernel, and pkgver.

The major things I changed from the current PKGBUILD was where the kernel version is declared, most of the time using pkgver instead of kernelname and removed most references to kernel26 in kernel26.install (not a full list, but there really wasn't much more then that.)

Here's a sample listing of what it installs:
kernel-2.6.30.1 /usr/src/linux-2.6.30-ARCH/
kernel-2.6.30.1 /lib/modules/2.6.30-ARCH/
kernel-2.6.30.1 /boot/vmlinuz-2.6.30.1
kernel-2.6.30.1 /etc/mkinitcpio.d/kernel-2.6.30.1.preset

Another question I suppose worth asking is kernel26-firmware, will that need to be updated upon each release as well? That is of course assuming a method like this is adopted.
Comment by Gavin Bisesi (Daenyth) - Wednesday, 11 November 2009, 14:24 GMT
I like the general idea here. This would be a super addition if we can get it implemented well. I can probably help hacking together some maintenance scripts as well.
Comment by Mr. K. (KitchM) - Wednesday, 11 November 2009, 17:05 GMT
Harley, could you please explain in noob terms exactly what this does?

Thanks for the good work.
Comment by Harley Laue (losinggeneration) - Monday, 30 November 2009, 16:40 GMT
@Mr.K. Sorry for the delay. In noob terms what this should do is allow multiple kernel versions to be installed so if a kernel upgrade breaks something, they still have a known working kernel that can be used to boot the system (without the need for a LiveCD or install CD.) It is then left to the user to decide when they want to remove old kernels. (A feature to periodically remove old kernels may be useful, but it's not required.)
Comment by Mr. K. (KitchM) - Monday, 30 November 2009, 20:14 GMT
Oh, yes, very nice! You are so right. That makes a heck of a lot of sense.

In fact, I might add that I had always thought that was the purpose of the option during GRUB boot. If that isn't, then what is the point of GRUB? Who needs a menu when there is no choice. Since I found it wasn't the point, I'm lost with trying to figure out what it does. All very weird.

I've always been used to having a simple menu system that allowed me to boot into the partition I wanted to use. It didn't matter what partition, what OS or combination thereof the user wished to utilize. Then along comes the Arch install that reinvents a wheel that won't roll. I still don't understand the thinking behind this whole thing. How much better it would have been if we had simple gone forward from where we've been for fifteen years. One would add another item to their menu, and go from there. And if the package manager had installed a whole new OS kernel, then one should be able to assume that the fall-back system was standing by and ready to be used as a function of the package manager. We should have always been able to boot into a different (the old) version of an OS. All we need was for users to be able to step back to the previous, working setup. In this case, we don't even need a separate partition, since all is simple software replacement. Instead, someone went to a lot of trouble to make something broken by design. (How much more productive it is when we use our abilities to improve rather than reinvent. What wasted man-hours! But, hey, that's just my opinion.)

I've gone over the dialog here again. There is a huge amount of shorthand-speak that doesn't work in the real world, but I attempted to wade thru it. It appears that the issue can be simply stated as the need to make sure that a kernel does not have dependency problems. Is that the gist of it? Two points: 1. the kernel is supposed to be the lowest common denominator, so that shouldn't be an issue, 2. and in my case, all the upgrades I've done since that kernel was available have not caused any problem when I have reverted back to the older kernel. All is working fine now, including things that didn't work before. I have simply learned to avoid 2.6.30.6-1. I do realize that maybe that is just the chance of this combination of upgrades, but there may be no way to tell. It is at least worth trying another simple backup method.

Instead of changing pacman, why not make a simple reversion script? The purpose would be to take the last setup and put it back as it was. Not being sure of the exact inner workings of pacman, IMHO it may be that pacman would need to have an added function of logging the necessary details of the upgrade so that the added (or associated) reversion script would know what to do. This would be simple add-on script-like coding, rather than a complete rework of the program. I would definately not recommend anything that required any sort of reinvent.

In any case, thanks for the response and clarification. I'm glad I'm not alone in wondering about these things.
Comment by Mr. K. (KitchM) - Monday, 30 November 2009, 20:22 GMT
Oops, that should be 2.6.31.6-1. (Dang, why can't I just edit may comment?)
Comment by Paul Mattal (paul) - Saturday, 06 February 2010, 14:33 GMT
This sounds like a good idea, but nobody has raised their hand to do it. Are we deciding not to implement for now?

If not, what are the next steps?
Comment by Mr. K. (KitchM) - Monday, 08 February 2010, 17:23 GMT
Great question, paul. What do you think, phrakture? Toofishes?
Comment by Michael Henry (drmikehenry) - Saturday, 13 February 2010, 12:59 GMT
Hi,

I'm a very new Arch Linux user (approximately one week so far :-)). After a kernel upgrade, I wanted to revert to the old kernel to see if the graphics performance degradation I was seeing was related to the upgrade. Coming from a long history of other distributions, I was somewhat astonished to find the old kernel had been overwritten instead of preserved. More than any other component in the system, a failed kernel upgrade has the potential to prevent booting. A simple reboot into an old kernel is easy and convenient in other distros, and Grub is tailored for exactly this kind of setup. If the various kernels could be treated as separate packages, it would be easy for a user to periodically use pacman to remove obsolete kernels. I don't know how much impact this has on the larger system, although many of the suggestions in the above comments sound reasonable to my Arch-novice ears. Perhaps Subversion tags could be used to mark the location of older kernel PKGBUILDS, allowing development to continue in the kernel26 directory. These "tags" wouldn't necessarily have to live outside of trunk; they could be dropped peer-level to the kernel26 directory, so they are easily accessible to users via abs. I don't really know enough to make too many implementation suggestions, though. But as a user, I'm very much in favor of supporting the simultaneous installation of multiple kernel versions.

Thanks,
Michael Henry
Comment by Mr. K. (KitchM) - Sunday, 14 February 2010, 17:35 GMT
I couldn't agree more. It is odd that there are so many unnecessary complexities within the subject that seem to create a cloud over the whole thing and stand in the way of progess. We really need to see this in simple terms. I can't help but recognize the truth in one of the things I've learned in the industry; which is that the end-user's impression is usually correct, and that, not only do these problems need to be addressed, but we need to use the KISS method in doing so.

I would suggest that those in the know first explain what is going on in creating the alternative boot setup that we currently have, so that all of us can understand where we stand now. I must admit that I still don't get it.

Thanks.
Comment by Paul Mattal (paul) - Saturday, 06 March 2010, 21:10 GMT
What would we need to (optionally) keep backup copies of the kernel?

It sounds like we'd need to start storing kernels, initcpio images, and other relevant info in directories named for the pkgver and pkgrel. This starts to get complex, doesn't it?

I agree, it would be a great feature to be able to upgrade your kernel and trivially boot back into your old one if something went wrong. Can anyone lay out the full steps we'd need to do to make it happen?
Comment by Mr. K. (KitchM) - Saturday, 20 March 2010, 04:47 GMT
How about this; let's leave it up to disk space. If we have plenty for the boot area, move the old kernel to backup directory. Change GRUB to add extra line. Install new kernel in normal location. If things don't work, one simply chooses last working kernel from GRUB menu. Then the user can run a script to remove new and replace old, changing GRUB in the process. Or something similar. Could symlinks be useful somehow in this process? (I'm just throwing these ideas out there for someone who knows kernels and their installation better than I and can look at these ideas and rearrange them until maybe something fits.)

Sometimes it might be useful to see the OS kernel as just another program to install. We know that it is possible to have more than one version of a particular program run in the same system. The complexity drops when we consider only one backup rather than multiple versions. It must be assumed that the old one was working and then new one was then added. That's only two. We've got that now, although useless, as another GRUB menu item. (Hopefully, we're not talking multiple boot partitions.)
Comment by Davie (daschu117) - Sunday, 18 July 2010, 21:30 GMT
Wouldn't a system like this cause dependency hell with making sure that the appropriate versions of third-party drivers are installed for each retained version of the kernel? All the various graphics and networking drivers come to mind.
Comment by Mr. K. (KitchM) - Sunday, 18 July 2010, 22:32 GMT
Aren't the dependencies actually just a simple list of programs and their appropriate versions in a text file? Then wouldn't that simply be the source for the undo information by comparing the recent one with the similar file in the backup kernel directory? That should actually simplify the whole process so that a simple script can undo what was done.

The only thing we must keep in mind is that the kernel is separate from the other programs which may have been upgraded or installed at the same time. I believe that it is always important to upgrade the kernel separately from the rest of the system, so that the user can reboot and make sure it is working correctly before changing anything else. (I realize that this probably leaves a few things unsaid and assumed, but that's the general idea.)

Perhaps doing a "pacman -Syu" should note that a kernel upgrade exists first and do it separately with a required reboot notification. All other things should halt until this is done.

The alternative is that there is a lack of modularity in Linux, and such must be avoided at all costs. If it might be, as I suspect in this case, we begin to slide down that slippery slope, the whole system begins to be one humugous mess ala micro$oft. We cannot allow such a system to come into being. The issues of diagnosis alone dictate that this cannot be.

Thanks.
Comment by Matthew (piezoelectric) - Thursday, 05 August 2010, 03:07 GMT
Rather than full-blown versioned kernels, what might fit a lot of people's needs would be to at least have the kernel in testing install side-by-side with the core/kernel26.

/lib/modules might not be a problem, except if the same 2.6.X is in testing, then maybe: /lib/modules/2.6.X-TEST

/kernel26-testing.img
/boot/System.map26-testing
/boot/vmlinuz26-testing
/etc/mkinitcpio.d/kernel26-testing.kver
/etc/mkinitcpio.d/kernel26-testing.preset
Comment by Anonymous Submitter - Saturday, 07 August 2010, 16:31 GMT
I think a full-blown versioned kernel is needed. If a kernel does not boot at all (maybe due to wrong config), then initramfs will not help. And if someone does not have a linux live cd/usb handy, then how will those people solve the problem? Simply overwriting old files with the files from the new package is okay even for xorg, kde etc. as the terminal/shell will still be available (init 3) to downgrade the packages? How will you get a working shell with a non-bootable kernel?

Since it is policy of Arch to make the user responsible for the configs, packages etc. it will not be a problem for an Arch user to manually update or remove kernel packages him/herself. Just make sure one kernel package does not overwrite another and leave the maintenance of /boot directory to the user. Arch devs need not worry about whether 1 kernel is installed or 5 or whether the /boot is small for many kernels . Let the user manage those things.

Please version the files in the packages also, like

/boot/vmlinuz26-2.6.35-ARCH
/boot/kernel26-2.6.35-ARCH.img or /boot/initramfs-2.6.35-ARCH.img (similarly for fallback)
/etc/mkinitcpio.d/kernel26-2.6.35-ARCH.kver
/etc/mkinitcpio.d/kernel26-2.6.35-ARCH.preset

I don't think any change is needed for /lib/modules directory
Comment by Anonymous Submitter - Wednesday, 25 August 2010, 07:23 GMT
For versioned kernel installs, pacman should allow installing different version of same package. May be like ignore packages array, a new category of packages for which multiple versions can be allowed, should be created in pacman.conf and pacman source modified accordingly.
Comment by N K (synackfin) - Wednesday, 12 January 2011, 06:20 GMT
I like the idea of having the old kernel stay and not get deleted right away. The details of properly versioning files and such have already been solved by most other distros, so we could just borrow the ideas from them.
Comment by Tom (twall) - Monday, 07 February 2011, 11:39 GMT
I completely agreed to Keshav P R, and i think it the kernel shouldnt be treated as any other program as stated before, because it is crucial for the system to work.

It was also stated before, that the user should not keep track of those things manually, but since arch-linux basically puts all decisions concerning updates to the user, i don't see why this shouldn't apply to the kernel. Maybe it would be possible to let pacman promt the user if he wants to keep old kernels once the user is cleaning the cache. (I mostly clean the cache when the disk is full and when my system is working like a charm)

Actually this problem keeps me from updating my kernel right now, because i need my computer to get some work done, and I don't wont to take the hassle to downgrade the kernel again if something goes wrong. I guess that applies to many users out there.
Comment by Sergej Pupykin (sergej) - Friday, 18 February 2011, 21:14 GMT
I think this idea violates KISS. You can keep lts kernel for recovery issues for example.

I keep tinycore linux in my /boot for recovery - it is nice 10M distro which works completely in initrd and can be booted even if whole arch is broken.

Also I have kernel26 in pacman's IgnorePkg and never update kernel while I have no physical access to PC.
Comment by Harley Laue (losinggeneration) - Friday, 18 February 2011, 22:00 GMT
Sergej Pupykin
"I keep tinycore linux in my /boot for recovery" "Also I have kernel26 in pacman's IgnorePkg and never update kernel while I have no physical access to PC."
The fact that you do this and feel it as a required precaution should pretty strongly reinforce that this is an issue. It's even laughable to consider this (tinycore in /boot) a simpler solution than having a versioned kernel. IMO it's not a fix, it's a work-a-round to the real problem.

Now, if enough users/developers feel this idea isn't following Arch's philosophies close enough, I propose that the kernel26-lts be added to the base group to be installed by default. This still far from ideal, but it'd be a better than nothing. (Don't think of this as conceding defeat. I still strongly believe that versioned kernels are not only a good option, but can even be done in such a way that it follows Arch's philosophies.)
Comment by Ionut Biru (wonder) - Friday, 18 February 2011, 22:06 GMT
having kernel26-lts as default would create more problems than it solves especially when it comes to video drivers that relies on KMS.
Comment by Sergej Pupykin (sergej) - Friday, 18 February 2011, 22:07 GMT
Keeping tinycore in boot is not more difficult than keeping rescue live-cd. All users who had problem once use rescue live disks or alternate boot option.
Comment by Harley Laue (losinggeneration) - Friday, 18 February 2011, 22:54 GMT
lonut
So the kernel26-lts wouldn't be suitable as a stable option for a kernel to recovery boot into upon issues? I hope it didn't look like I was implying for it to be the default kernel to be booted into, rather, a stable kernel that can be relied on to at least always boot. Though, if it's going to cause serious issues, it does nothing for the other case where a versioned kernel would be nice, kernel regressions and driver breakages.

Sergej
I never said anything about it being difficult. It the case of KISS, simple is not defined as easy, it's defined as lacking complexity. Again, I feel that your solution isn't a fix to the issue. It's a way to ignore the issue till it happens, and then work around the issue when it does. It also doesn't do anything for if there's a regression introduced. In that case, all the livecd's and rescue partitions in the world won't be of that much help (because the old kernel is gone.)
Comment by Sergej Pupykin (sergej) - Friday, 18 February 2011, 23:05 GMT
Solution is active kernel testing, so it never breaks anything)

May be repacman (http://aur.archlinux.org/packages.php?ID=35687) can back up kernel before upgrade?
Comment by Ruben Van Boxem (rubenvb) - Tuesday, 19 April 2011, 11:17 GMT
Interesting and heated discussion here. The way I see it:

1. There is a kernel26 (dummy package) will always depend on the latest kernel version available.
2. Installing a kernel upgrade will rename all the old files (kernel26.img etc) to a versioned copy (kernel-2.638.2 os something) so that everythin is still the same to the old Arch user (except for the additional kernel image, which could be deleted by a simple script or tool if the user wants to).
3. Kernel modules will be placed in a versioned directory, which is not deleted anymore when a new kernel is installed.
4. Every time kernel26 is updated, a new package kernel-2.6.x.x is created that has the exact same contents of the old kernel installation. This is the most tricky part: removing this newly added (and automatically installed package, as it was installed as kernel26) should remove every file associated with it. For an example on how this scheme could work: I remember that using the 2010-05 Arch install cd also renames several packages when you first update with pacman. The same renaming can be used here.

Just my two cents :)
Comment by Linas (Linas) - Monday, 02 May 2011, 22:07 GMT
I don't see the need for any renaming there. It can be done just with dependencies.

You create many kernel packages, that place them at eg. /boot/kernels/<version> where installing each of them add an entry to grub, and deleting removes it.

The kernel26 package would have
depends=( kernel26-${_basekernel} )

And the install file would (in addition to any steps to add it to grub) do
post_install() {
ln /boot/{kernels/${_basekernel}/kernel26.img,kernel26.img.new}
ln /boot/{kernels/${_basekernel}/vmlinuz26,vmlinuz26.new}
sync
mv -f /boot/vmlinuz26.new /boot/vmlinuz26
mv -f /boot/kernel26.img.new /boot/kernel26.img
sync
}

Items like /lib/modules/<version>/ would be pinned by the version-specific package, forced by the depends.
You only need to update the vmlinuz26 and kernel26.img hardlinks as they are pointed by grub (you could as
well try to update its reference, but as that may be edited by the user, given how cheap hardlinks are,
it seems easier to do this).
Comment by Anonymous Submitter - Monday, 06 June 2011, 15:45 GMT
Now that officially the next kernel will be 3.0 , I think it is a good time to remove kernel26 naming altogether (aka reconsider implementing any one of the proposals) and move to version specific naming. This will also require change in naming of the initramfs image corresponding to the kernel https://bugs.archlinux.org/task/18719 .
Comment by Kyle Fuller (kylef) - Saturday, 09 July 2011, 21:30 GMT
I think it would be nice if pacman could allow you to install multiple versions of the same package. Now for most packages you wouldn't be able to install multiple versions, but for packages like the linux kernel, and python this would make sense. This should allow making it so that version x of a package doesn't conflict with version y but version z does conflict with version y. For example you have python/2 which doesn't conflict with python/3 but python/2.6 does conflict with python/2.7. I think you could have the syntax in the PKGBUILD like follows:

conflicts=(!python=3.*)

The ! means, "doesn't conflict". The conflict for the kernel26 could look as follows:

conflicts=(!kernel26 kernel26=<3.0)

The above would mean, we don't conflict with any kernel26 version which is more than 3.0, since the older versions of kernel26 might conflict as they were not designed to work with the newer multiple versions system.

This would not only solve the issue for multiple kernel versions, but it will also simplify multiple other packages such as python. The future "linux-lts" package could simply just require a "linux/lts-version" package. The package repositories will need to support multiple versions of a package also, so python/2 and python/3 could be in the repo.

The only issue with multiple kernel versions is with boot loaders because the kernels would have to be in different locations to not conflict. The kernels would be placed somewhere like "/boot/kernels/linux-*" for each version. From what I understand is that there could be a grub2 script or even a grub2 module which could create the boot menu from different installed kernels inside the "/boot/kernels" folder. This would only support grub2. Grub1 and LILO users would need to manually change there configs to reflect new kernels.

For support with LILO/grub1 there could be a package which trials the latest kernel and provides a symlinks and depends upon the real kernel. A user would have 3 options, use grub2 and the automatic menu's, use the kernel trailing package or edit the config themselves. The trailing package could be something like "linux-symlinks", and the new linux package called "linux" (once 3.0 is released).

The only issue is that dependencies would have to support a new syntax, so you can provide a dependency to a certain set of versions such as python2-bla package would depend on python=2.*, if you have python/3 installed the installer should detect and then install the python/2 package alongside python/3.

The package repositores can continue to store python2/PKGBUILD and python/PKGBUILD but the package name inside the package would be the same "python". The linux kernel could adopt a similar pattern.

If a few people like this then I will file it as a complete new bug/feature report for pacman. I have tried to explain this as clearly as I can, I hope it makes sense.
Comment by Bernhard (drahnr) - Thursday, 14 July 2011, 13:37 GMT
@Kyle: Just one point left, how is the user supposed to install the different packages as they are now named the same? I could imagine (but I doubt this will be real) to implement a shallow kind of regex into pacman, so it can easily match pkgname:ver as input (for any kind of versionen sheme). But your effort throws up many questions. Should multiinstall of packages then be enabled for all pkgs? Should we introduce a pacman flag to allow duplicate install? How to handle multiinstall for files in /usr/bin/ /usr/sbin/ /sbin/ and so on...
Comment by Thomas Bächler (brain0) - Thursday, 14 July 2011, 13:50 GMT
Dan and I had a short informal talk recently, and it looks like we can solve this cleanly without having multiple packages of the same name. This requires some changes to the linux package, these changes will probably be done slowly starting with the 3.0 release - most importantly, the different kernel versions must not have file conflicts.

It also requires support in pacman, so that we can automatically remove old versions. This will all go slowly and take a while, just wanted to let you know that we will start a process that will eventually lead to the resolution of this bug report.
Comment by Sergej Pupykin (sergej) - Thursday, 14 July 2011, 14:06 GMT
I think kiss solution is:

$ cp /boot/vmlinuz26 /boot/vmlinuz26.old
$ cp /boot/kernel26.img /boot/kernel26.img.old
$ mv /lib/modules/`uname -r` /lib/modules/`uname -r`.bak
$ pacman -Sy kernel26
$ mv /lib/modules/`uname -r`.bak /lib/modules/`uname -r`
$ reboot

May be add key to pacman --no-delete-only-replace-and-add-new-files instead of 2 mv commands.
Comment by Thomas Bächler (brain0) - Thursday, 14 July 2011, 14:17 GMT
This is not an acceptable solution. This will lead to files being in the system that are not tracked by pacman and have not been installed from any package. This is an absolute no-go and we will never do this (yes, NEVER).
Comment by Marti (intgr) - Thursday, 14 July 2011, 14:19 GMT
brain0, note that Arch is already doing this -- .pacnew and .pacsave files aren't tracked by pacman either.
Comment by N K (synackfin) - Thursday, 14 July 2011, 14:20 GMT
The kernel file should also be versioned (e.g. /boot/vmlinuz26-(ver), and make /boot/vmlinuz26 a symlink to the latest). Thus, two kernel packages would not conflict each other at all except over that one symlink file. Pacman can then have a mode that keeps two packages around so long as the only conflicted files are symlinks (in which case, the symlink is taken from the more recently installed or higher versioned package). The user then manually edits the grub.conf to add older kernels as options. This I think is the KISS approach.

If we want to get fancy, then in addition to the above, we can do something similar to RHEL which updates the grub.conf at every kernel install. Grub would need to have a menuentry auto-added for each installed kernel (and the menuentry auto-removed when a kernel is removed), also like RHEL. The default menuentry would continue pointing to vmlinuz26 (the symlink), only additional grub menuentries would point to specific kernel files. This approach requires zero manual intervention from a user, it will "just work" -- when they reboot, they'll see menus for the latest kernel as well as entries for all the specific versions they kept around. I prefer the KISS approach of not auto-managing grub.conf, but at least creating the kernel packages such that they never conflict with another kernel package except over symlinks.
Comment by Sergej Pupykin (sergej) - Thursday, 14 July 2011, 14:22 GMT
Ok, forget --no-delete-only-replace-and-add-new-files key

But adding kernel26 to IgnorePkg and writing 2 scripts (~/bin/upgrade-kernel.sh and ~/bin/remove-old-kernel.sh to clean up) is easy, simple and covers kernel upgrading issue.
Comment by Thomas Bächler (brain0) - Thursday, 14 July 2011, 14:50 GMT
Marti, those are special files and should probably also be handled by pacman somehow, but this shouldn't be the topic here.

Okay, before tons of bikeshedding starts, here it is: Eventually, each kernel package will have a kernel version like 3.1.X-Y-ARCH where X is the upstream patchlevel, Y is the pkgrel and -ARCH indicates that this is the Arch Linux kernel. This will take care of /lib/modules/* not conflicting. Kernel image and initramfs will have this version appended to its filename.

Now, the packages: The 'linux' package will depend on the following packages: linux-version-3.1.X-Y (this is the pkgname, without version) and linux-version-3.1.X2-Y2. The 'linux' package will also manage symlinks like vmlinuz-current and vmlinuz-old. Now, we need a new feature in pacman: we need to be able to obsolete a package so pacman uninstalls it on its own.

Anyway, this approach is not entirely thought out yet, I still see a problem when someone skips a few kernel versions, and with external modules. As I said, we will transition to this slowly rather than in one step.
Comment by Bernhard (drahnr) - Thursday, 14 July 2011, 15:32 GMT
@Kyle: Just one point left, how is the user supposed to install the different packages as they are now named the same? I could imagine (but I doubt this will be real) to implement a shallow kind of regex into pacman, so it can easily match pkgname:ver as input (for any kind of versionen sheme). But your effort throws up many questions. Should multiinstall of packages then be enabled for all pkgs? Should we introduce a pacman flag to allow duplicate install? How to handle multiinstall for files in /usr/bin/ /usr/sbin/ /sbin/ and so on...
Comment by dan (dangbarry) - Friday, 12 August 2011, 07:50 GMT
Thomas, Just thought i'd throw an idea out there. A script that waits until the nth successful reboot to remove the old kernel. Execute the script through the rc.local. Simply, a successful boot counter. When it's save to proceed, remove the old kernel and the script from the rc.local, until of course next kernel update. That also solves your pacman issue, as the rc.local is launched from root at boot. "uninstalling it on its own".
Comment by Robert Petranovic (jazzfan) - Sunday, 21 August 2011, 12:50 GMT
@dangbarry: I don't think that's good idea. System might not work as expected completely, and the script would increase counter. While user tries to debug it, script will uninstall the good older kernel.

I recommend that there is a config option in pacman.conf: KERNEL_VERSIONS or something simmilar. This should specify how much versions of kernel should be keept.

For example, let's say it's 3. When first 3 kernel versions install, nothing happens. But when 4th kernel update is installing, pacman should remove the first one. This could be done using a workarouond like this in .install file:

# do a standard depmod, mkinitcpio, create symlinks for new kernel to be run as default
#

# move the pacman lock so another instance of pacman can be run.
# This is save because one pacman is just waiting for the second one to finish, so pacman database is safe.
mv /var/lock/pacman.lock{,.bak}

# get number of kernels installed
num=$( ls -1 /boot | grep vmlinuz-linux- | wc -l )
if (( "$num" <= "$KERNEL_VERSIONS" )); then return 0; fi


# get oldest version and remove old kernel
version=$( ls /boot | grep vmlinuz-linux- | sort -n | tail -n 1 )
version=${version%vmlinuz-linux-}

pacman -R linux-$version

# get the lock file back
mv /var/lock/pacman.lock{.bak,}
Comment by Kyle Fuller (kylef) - Sunday, 21 August 2011, 16:45 GMT
@jazzfan, there are two issues with. The first being that a user might have a kernel he wants to keep. Lets say he does this:

pacman -S linux-2.6.32 linux-2.6.33 linux-2.6.34 linux-2.6.35
pacman -S linux

A user who manually installed these packages will not expect them to be removed automatically, but they would be. The other issue is that this may break if linux-lts uses the same system for versioned kernel installs (which it should).

pacman -S linux-lts # This installs linux-2.6.32-1
pacman -S linux # This installs linux-3.0.3-1 (the user already had 3 kernels before this)

The pacman -R will fail due to linux-lts depending on linux-2.6.32.

For these two reasons I think the user should remove a kernel manually and do not do this automatically for a user.
Comment by Nico Schottelius (telmich) - Wednesday, 24 August 2011, 07:41 GMT
Imho, this is pretty easy to solve. Either we do it the Debian way with lazy cleanup or we keep all kernels and have the dummy package depend on the real kernel as described in the first three comments and keep the old packages.

Imho there is no harm in keeping them, we should probably consider a feature like "autoremove" in apt, which removes previously needed, but now unneeded packages, with a hint anyway.

There maybe the problem that a user runs autoremove after installing a new kernel and then removes the current kernel, but I guess this shouldn't be a big issue for arch users.
Comment by Mitchell Richters (mjr4077au) - Sunday, 28 August 2011, 21:12 GMT
I know there's a lot of opinion here and that isn't always productive, but I think the solution is this.

1: Switch to GRUB2. GRUB 0.97 is undeveloped, patched like nothing else and is dead in the water. GRUB2 has grub-mkconfig and will detect everything in /boot, even Windows partitions on another disk with os-detect.
2: Keep one linux package that generates a different initial ramdisk upon installation/upgrade based on whatever version number you want. This will keep the old initial ramdisks available for use since they're not installed with the kernel, they're generated by mkinitcpio.
3: Call "grub-mkconfig -o /boot/grub/grub.cfg" in the kernel upgrade to re-generate the file. If it overwrites the file and someone has manually customised it, shame on them, because thats not how you modify GRUB2. If a user doesn't have GRUB2, the call will simply fail and the user can update their bootloader. If they're not using GRUB2, then they're probably doing something else, so they'd know what they're doing.

I think that's pretty KISS. I personally don't care about 3, because I know Arch is a bit more manual than others and I have no problem with typing a one-liner.


As a sidenote to Sergej, you say "Also I have kernel26 in pacman's IgnorePkg and never update kernel while I have no physical access to PC". Next time you're buying a PC/Server, get something with iLO, you'll have remote access to the entire system with actual video output, including full system BIOS access :).
Comment by Thomas Bächler (brain0) - Sunday, 28 August 2011, 21:34 GMT
Yes, it seems everyone needs to give an opinion here, and none of it is productive.

@Mitchell: As for your opinion - I don't quite understand what you actually want to do, but you seem to think having a different bootloader will solve some problem. You also seem to think that now everyone uses grub and later everyone will use grub2. You also seem to think that regenerating a configuration file on each update using some voodoo script that we have no control over is a KISS solution. Whatever it is you're suggesting, it won't happen.


I am thinking about simply closing this report. It's going nowhere, everyone just posts new "solutions" that we simply won't implement. I posted a game-plan in an earlier comment, which will eventually be implemented. I consider this very low priority, and thus the required changes will be made slowly. Leaving more and more comments with brilliant "ideas" here (which spam my inbox with bugtracker emails) will not speed it up.
Comment by Mitchell Richters (mjr4077au) - Sunday, 28 August 2011, 21:50 GMT
By all means, my opinion is just that. By having grub 2 as a default allows for easier bootloader updates, yes. I stand by having it as a default because it's what it's upstream developed and supported. This it's of topic though.

What I'm ultimately saying is to have versioned initial ramdisks, then new initial ramdisks can be generated upon kernel upgrades.

Lastly, no voodoo script by any means. I am only talking about one line to generate a new config for grub. It wouldn't be any more code than what's used to generate the initial ramdisk upon a kernel upgrade now.

I don't assume everyone to have grub as a bootloader at present, no. but those who don't follow defaults and want to do their own thing would be smart enough to deal with updating their bootloader.
Comment by Mitchell Richters (mjr4077au) - Sunday, 28 August 2011, 21:56 GMT
And as I also said, I myself don't care is grub2 or any bootloader gets upgraded on a kernel upgrade, I can deal with that myself.
Comment by Ernestas (ernetas) - Friday, 18 November 2011, 18:30 GMT
Maybe it would be a great idea to allow not an unlimited number of kernels available in /boot, but only the two most recent. This could be realised by upgrade scripts, just as changing the symlinks.
Comment by alessandro (alelinuxbsd) - Sunday, 29 January 2012, 21:12 GMT
Comment by Thomas Bächler (brain0) - Thursday, 14 July 2011, 14:50 GMT
Dan and I had a short informal talk recently, and it looks like we can solve this cleanly without having multiple packages of the same name. This requires some changes to the linux package, these changes will probably be done slowly starting with the 3.0 release - most importantly, the different kernel versions must not have file conflicts.

It also requires support in pacman, so that we can automatically remove old versions. This will all go slowly and take a while, just wanted to let you know that we will start a process that will eventually lead to the resolution of this bug report.


Could i ask if this particular work is advanced? Just for have a general idea on when this function will be available?

Thanks.


Comment by Jonathan Vasquez (Cows) - Monday, 18 June 2012, 06:18 GMT
This is an issue, but I don't think it's really a big deal.. it really depends the way you look at it.

What I do is I just copy the current kernel with a different name. For example we are currently (at the time of this post) on 3.4.2-2-ARCH.

So I do a `cp /boot/vmlinuz-linuz /boot/kernel-3.4.2-2-ARCH` and `cp /boot/initramfs-linux.img /boot/initramfs-3.4.2-2-ARCH.img` and add the entries into my bootloader. Thus if I ever want to stay on that kernel, I can reboot into the old kernel (or set it as default).

The above method doesn't solve the "Temporarily unable to load my drivers in between kernel upgrades" but it does solve many other issues.
Comment by Vyacheslav Stetskevych (tskevy) - Wednesday, 03 April 2013, 21:17 GMT
I would love to see this feature implemented.
Recently, a kernel upgrade from 3.7.10 to 3.8.3 broke my co-worker's Arch. We had to use a usb drive, unlock LUKS, detect LVM, mount drives, chroot, reinstall 3.7.10.
All would've been simpler, had we the ubuntu-way of choosing older kernels in grub.
Please, think about it.
Thanks.
Comment by Paul Gideon Dann (giddie) - Thursday, 04 April 2013, 09:11 GMT
In the meantime, I'm quite fond of keeping an ArchBoot kernel & initramfs around for quick access to a rescue system. You can add an entry to your bootloader, and so you always have something to fall back on :)

https://wiki.archlinux.org/index.php/Archboot#PXE_booting_.2F_Rescue_system
Comment by Dima Q (dimaqq) - Friday, 13 December 2013, 14:35 GMT
I would very much like to see this feature implemented for a different reason:

Being diligent I run pacman -Syu quite so often.
However I cannot afford to reboot as often.

so here I'm stuck, unable to mount -t cifs //... /..., and why?

uname -a --> 3.12.0-1-ARCH
find /lib/modules/ -name cifs\* --> 3.12.4-1-ARCH

I want at least modules for the currently running kernel to be kept around!

Comment by Thomas Bächler (brain0) - Friday, 13 December 2013, 14:41 GMT
@dimaqq: First of all: IgnorePkg=linux in pacman.conf, please.

We still need a way to install each kernel version under a different file name and automatically update the boot loader afterwards, but I don't see a solution for this (the freedesktop.org bootloader spec is a start, but 1) you'd lose custom kernel arguments on each update, 2) only gummiboot and a patched grub even support the spec right now).
Comment by Paul Gideon Dann (giddie) - Monday, 16 December 2013, 09:53 GMT
Surely we can simply maintain a symlink for the latest kernel in /boot, and point the bootloader at the symlink? That's the traditional method.
Comment by Thomas Bächler (brain0) - Monday, 16 December 2013, 09:56 GMT
It's possible for BIOS folks - for UEFI, /boot will be FAT32 which has no concept of symbolic links.
Comment by Paul Gideon Dann (giddie) - Monday, 16 December 2013, 10:04 GMT
I am booting from UEFI. /boot is ext4, and /boot/efi is FAT. The kernel is in /boot, so a symlink would be fine for me. I have /boot/grub/ symlinked to /boot/efi/efi/grub/. Slightly convoluted, but it works great.
Comment by Thomas Bächler (brain0) - Monday, 16 December 2013, 10:07 GMT
Only works with grub. Grub is the only bootloader that can load kernels from other partitions than the ESP. With all other bootloaders, your kernel must be on the ESP where the firmware can access it directly.
Comment by Dima Q (dimaqq) - Monday, 16 December 2013, 10:18 GMT
I suppose traditional method was to have 2 regular files, vmlinuz (current version) and vmlinuz.old (previous version).

that's not really great either, arch updates kernel quite often and double kernel update will leave the system without a copy of working (currently running) kernel.

I suppose a working solution would look like this:

keep vmlinuz-3.x.y-m-arch # running kernel
keep vmlinuz-3-z-z-m-arch # newest installed kernel
vmlinuz # copy on FAT, symlink on ext*

same for initramfs
same for /lib/modules/xxx
Comment by GI Jack (GI_Jack) - Tuesday, 25 February 2014, 19:45 GMT
I think a better idea is vmlinuz is a symlink to something like vmlinuz310 and vmlinuz312, etc...

This is actually not a bad idea. So the package would be linux310, linux312, etc..
Comment by Paul Gideon Dann (giddie) - Wednesday, 26 February 2014, 10:18 GMT
This has already been suggested. The problem is that some people have a FAT filesystem for /boot, which doesn't support symlinks. It might be possible to get around that by simply copying the file, so long as the filesystem isn't too small.
Comment by Alexandre Bique (babali) - Wednesday, 26 February 2014, 10:21 GMT
God, blocked by FAT ;-)
Comment by Daniel Micay (thestinger) - Sunday, 30 March 2014, 02:28 GMT
In my opinion, the linux-lts package covers this use case well. The ESP partition is usually *really* small and we are going to cause a lot of problems if we fill it with old kernel versions. This is something that users can do if they really want, but I find it to be an overcomplicated, overly magical hack for the package to be involved with an install script.
Comment by Pablo Lezaeta (Jristz) - Wednesday, 23 April 2014, 18:58 GMT
And what about keep only 2 or 3 kernels and remove the oldest ones automagically?
let an example
kernel-3.14.0 provides kernel-3base
kernel-3.14.1 provides kernel-2base
kernel-3.14.2 provides kernel-base

then simply upgrade maintain 1 version of diference and keep only 3.
the drawback is that this will force rebuild of alredy buil packages for changin the kernel-type

or you can copy the Manjaro way and adapt it to Arch (bad idea but somone need propose it)
Comment by Daniel Micay (thestinger) - Wednesday, 23 April 2014, 19:01 GMT
The LTS kernel continues to get security updates and important stability fixes, so it seems more viable for users to install that and use it as a fallback.
Comment by OS Hazard (hazard) - Thursday, 07 August 2014, 02:48 GMT
FYI @falconindy denied the proposed LTS solution.

Additionally, @phrakture and I cannot be the only users irritated by such things as USB storage breaking upon upgrading the kernel. This is due to the old module directory being purged immediately and thus requires a reboot before new devices can be hotplugged.

I propose 3 possible solutions to this annoyance:
a.) Versioned/slotted kernels
b.) Build core drivers into the kernel (why for instance are ext4, btrfs, etc. a module?!?)
c.) Create a hook to optionally keep the old modules directory UNTIL reboot.
Comment by Dima Q (dimaqq) - Thursday, 07 August 2014, 06:53 GMT
I think we need to be specific what is really needed:
* modules for newest installed kernel
* modules for other installed kernels (if allowed)
* modules for current running kernel whether installed or not

It's not only about USB, it's also about all on-demand modules, including firewalls, virtualbox, etc.
Typical scenario is:
* boot, work a bit
* pacman -Syu
* start virtualbox, fail
Comment by GI Jack (GI_Jack) - Sunday, 10 August 2014, 21:17 GMT
Its not that hard to fork module packages, I do some myself for linux-aufs_friendly in AUR.

I volunteer to help maintain packages.

As for kernels, we do it as we had before kernel 3.0, where there was a package kernel26(and a kernel24 when arch first started).

linux34, linux310, linux316.

We don't need patch level granularity as in 3.15.7, just minor version granularity.

I propose supporting all of the longterm kernels as found on the kernel.org homepage, as well as keeping a curret kernel.

I can also volunteer CPU and personal time to work on such if needed.
Comment by Andreas Källberg (Hjulle) - Sunday, 24 August 2014, 10:43 GMT
Bug #28081 makes the lts-solution quite annoying since grub thinks that the lts-version is newer.
Comment by Claire Farron (clfarron4) - Thursday, 30 October 2014, 09:26 GMT
On the topic of LTS kernels supported on kernel.org, I recently got a repository going for them at tiny.cc/clfarron4
But that's not the issue here and it doesn't help solve the one proposed here when you consider that the current version of the kernel (3.17/16) and at least two of the LTS releases (3.14 and 3.10) get updated at the same time.
Comment by Gary van der Merwe (garyvdm) - Monday, 19 January 2015, 11:29 GMT
An alternative solution is to implement systemd offline updates[1]. I would be nice if it only updated packages that need to be done offline, offline. The only one I know of is the kernal, but there may be others. This will require some small additions to pacman.


[1] http://www.freedesktop.org/wiki/Software/systemd/SystemUpdates/
Comment by Claire Farron (clfarron4) - Monday, 19 January 2015, 16:36 GMT
@garyvdm: I wonder if anyone has actually implemented this in practice (it'd be good to study what they've done in code to get it to work)? Have any of the Arch Devs seen this in practice?

If I were to implement something like that, at least for the initial introduction, I'd leave it as a service to be enabled, instead of implementing it as something that just happens.
Comment by Andreas Radke (AndyRTR) - Monday, 20 April 2015, 08:15 GMT
Is anyone brave enough to close this one as "Won't implement"? We have pacman pkg cache and also LTS for fallback. This is working pretty well for almost all users.
Comment by Paul Gideon Dann (giddie) - Monday, 20 April 2015, 09:00 GMT
Well we still haven't solved the problem of the running kernel's modules being removed when installing a new kernel... This can cause issues e.g. when plugging a USB keyboard into a server after remotely installing a new kernel. The kernel is unable to load the relevant module, and the keyboard is unusable. It's quite a confusing situation for someone who doesn't realise why it's happening.
Comment by Andreas Källberg (Hjulle) - Monday, 11 May 2015, 14:53 GMT
LTS for fallback is also not sufficient if both the normal version and the LTS-kernel was updated at the same time (or at least between your updates). You could of course manually postpone upgrading the LTS kernel, but that makes upgrading more convoluted.
Comment by Amos Onn (amos) - Tuesday, 02 June 2015, 21:32 GMT
Another issue which might affect some users is that hibernate (to disk) breaks if you boot a different kernel than then one which performed the hibernate. i.e. you upgrade kernel, hibernate, and then when you power on the new version of kernel doesn't manage the loading well and crashes. This was at least true somewhere at the beginning of 3.x, I haven't tried in a long time.
Comment by Jason (replica9000) - Thursday, 27 August 2015, 18:44 GMT
Coming from Debian, I am used to having multiple kernels installed. Having a known working kernel is important in case something doesn't work with the new kernel.

As I have my Arch install on ZFS from the AUR, I wouldn't want to be without my data due to some possible issue with modules not building, or being unavailable until a reboot. (I may go months without a reboot.)

I have currently installed a kernel from the ABS. I've simply edited the PKGBUILD to change pkgbase=linux to pkgbase=linux-4.1.6-1, and built the package(s) as usual. The result was being able to install a new kernel and headers without overwriting the old ones. I've also created an empty dummy linux package for compatibility/dependencies.

Now obviously that means that building new modules such as ZFS means pointing the dependency to linux-x.x.x instead of just linux. I've also changed the module names in PKGBUILD to match the kernel it was build for (zfs-git-4.1.6-1). This means I can, if necessary, boot into the older kernel, and still having working modules for it too.

I know this sounds like extra work, but we use Arch because we aren't afraid of that right??

Perhaps instead of having AUR packages look for a specific version of the kernel, why not specify the kernel to build against during makepkg? (I had to fake uname -r to report the new kernel instead of the running kernel.)

makepkg --option <kernel-version>
-- or --
export KERNELVERSON=4.x.x
makepkg

For now, I'll keep experimenting with this to see how to runs in the long term.
Comment by Jason (replica9000) - Saturday, 29 August 2015, 15:41 GMT
doublepost :(
Comment by Dennis Schridde (devurandom) - Tuesday, 02 February 2016, 08:35 GMT
systemd comes with the `kernel-install` tool, which takes a kernel version and image as argument and then runs /etc/kernel/install.d and /usr/lib/kernel/install.d on it. The latter contains 50-mkinitcpio.install, which compiles a initramfs for the given version and places it in /boot/$machine-id/$version/initrd, as well as 90-loaderentry.install, which will copy the given kernel image to /boot/$machine-id/$version/linux and create a versioned entry in /boot/loader/entries. This is, as far as I understand, what you want.
Comment by Dima Q (dimaqq) - Tuesday, 02 February 2016, 09:16 GMT
Integration of `kernel-install` is probably a good idea.
There's more to this ticket than kernel in /boot though:

Most importantly, /usr/lib/modules/<version> must be preserved.
Ideally, DKMS should also be built for all preserved versions.

IMO, entire `linux` package content should be available for a) current version, and b) every version installed into /boot
Comment by Alexandre Bique (babali) - Tuesday, 02 February 2016, 10:57 GMT
That systemd kernel install sounds like a great idea!
Comment by ITwrx (andriesinfoserv) - Wednesday, 08 June 2016, 23:34 GMT
from an end user's perspective: I'd be in favor of anything (within reason) that doesn't require manual removal of old kernels. Some other well known distros will happily fill up the boot partition until a novice user can't update anymore and dependencies are borked. I understand arch is not for novices, but if a piece of software can install multiple kernels, it can clean up after itself too. Requiring pre config is fine. Periodic manual intervention, not so much. The current way arch handles it is something i've been happy about for years, though i have seen issues with modules so i understand trying to fix that.
Comment by Alif (alive4ever) - Sunday, 12 June 2016, 07:34 GMT
I suggest to add "old" suffix to the old kernel and initramfs files for the last kernel. So that, when a new kernel update appears, there will be vmlinuz-linux-old, initramfs-linux-old.img, and initramfs-linux-fallback-old.img that can be booted after updating grub.cfg entry. This can be triggered by libalpm hooks (pre-transaction and post-transaction).
Comment by Behnam Momeni (sbmomeni) - Thursday, 21 July 2016, 07:12 GMT
Here is an alternative scheme which keeps N latest kernel versions installed and removes all older kernels automatically (by pacman):

1. Each kernel version has its own version-named package as described above (e.g. linux-1.2.3.4-1.2.3.4-1 where linux-1.2.3.4 is the pkgname and the second 1.2.3.4 is the pkgver),
2. Each one of these version-named packages will "provide" the linux-real=${version} package. That is, although each version-named package looks like a completely different package, it also provides a version of a common named target (i.e. linux-real),
3. A dummy package like linux-virtual-${version} will be created which always depends on the latest linux-real-${version} package and also contains the above mentioned symlinks. This installs the new kernel version as its dependency and leaves older versions installed as unused dependencies,
4. The linux-virtual dummy package also "conflicts" with the "linux-real<=${N older version as you wish}" and so all older kernel versions which provide those older linux-real targets are found by pacman and will be removed after getting a y/N confirmation from the user.

This requires no special maintenance script for removing old kernels nor any special support from the pacman.
Comment by Ernestas (ernetas) - Thursday, 21 July 2016, 07:39 GMT
I'm surprised nobody has mentioned this yet, but since Linux 4.0, kGraft and kpatch were included in the kernel. Not to mention ksplice, which has been available for years. While having multiple kernel versions installed seems important right now, I think that live kernel updates is the future.
Comment by Alexandre Bique (babali) - Thursday, 21 July 2016, 09:24 GMT
So how does the live kernel update work? Can we do it right now on Archlinux?
Is that thing well tested?

From my first thoughts I would not do that automatically, but I'd love to have a command that could do it.
Comment by Daniel Micay (thestinger) - Thursday, 21 July 2016, 09:43 GMT
Live updates are aimed at making small security fixes. It's not a way to do arbitrary upgrades, and doesn't simply Just Work without work to create the live patches. I also don't really see how it's relevant to keeping multiple kernel versions installed...
Comment by Andreas Radke (AndyRTR) - Thursday, 21 July 2016, 15:17 GMT
We now have 3 official kernels in our repos: stick kernel, -lts and -zen. I'm for closing this bug
as nobody seems to be interested in implementing this and there's no real need to have fallback kernels of
the same branch.
Comment by Alexandre Bique (babali) - Thursday, 21 July 2016, 15:21 GMT
No way to close that bug :)
I don't think that no one is interested in implementing it, I think that we just didn't come to an agreement on the solution to this issue.
Comment by Dennis Schridde (devurandom) - Thursday, 21 July 2016, 15:27 GMT
While there might be no real need for YOU to have a copy of the running kernel in the filesystems, there is definitely a strong need for others.

E.g. read my comment above (https://bugs.archlinux.org/task/16702#comment143419). When you have a kernel in /boot and EFI is setup to start it (e.g. using gummiboot or systemd-boot), but the /lib/modules/${KV} directory associated with it is gone, the boot will fail. There will be no option left to boot into the system and one has to boot from another medium (e.g. usb drive) to fix this up by copying the newly installed kernel image into the right place (the place expected by EFI/gummiboot/systemd-boot).
Comment by Radek Podgorny (rpodgorny) - Thursday, 21 July 2016, 15:30 GMT
i think this bug report is not really about having lots of different kernels installed simultaneously. it's about retaining the running kernel and its modules. otherwise this can happen:

1) boot up kernel X
2) update to kernel X+1
3) plug in usb stick/mount some filesystem/try to load any module not yet loaded
4) cry after you find out old modules (for the running kernel) are gone and you need to reboot
Comment by Amos Onn (amos) - Thursday, 21 July 2016, 15:35 GMT
Another reason for retaining the running kernel is for hibernating - otherwise you hibernate, but when powering up, the new kernel version tries to restore the state of an older version and fails.

But still, it's just about the running kernel, not about multiple kernels.
Comment by Andreas Radke (AndyRTR) - Thursday, 21 July 2016, 15:37 GMT
After 7 years none of the kernel maintainers or other developer stepped in and wanted to implement this.

This won't change anytime soon. So "Won't implement" and closing this one still makes sense to me.
Comment by Alexandre Bique (babali) - Thursday, 21 July 2016, 15:58 GMT
I can implement it. It is not a very complicated task.
Do we have any specification on how it is supposed to be implemented?
If not, then what i can do is:
- a systemd init script which copies the current modules into /lib/modules/current + cleanup of the old symlinks to current
- a post uninstall hook for the kernel which will detect and symlink if necessary 4.6.4-1-ARCH/ to current/

Is that solution OK?
Comment by Behnam Momeni (sbmomeni) - Thursday, 21 July 2016, 16:06 GMT
You do not need to write any script for retaining the current version modules. The pacman can handle it automatically.
Read my comment (https://bugs.archlinux.org/task/16702#comment149162).
Comment by Eli Schwartz (eschwartz) - Thursday, 21 July 2016, 16:19 GMT
Comment by Thomas Bächler (brain0) - Thursday, 14 July 2011, 09:50 GMT-5

Okay, before tons of bikeshedding starts, here it is: Eventually, each kernel package will have a kernel version like 3.1.X-Y-ARCH where X is the upstream patchlevel, Y is the pkgrel and -ARCH indicates that this is the Arch Linux kernel. This will take care of /lib/modules/* not conflicting. Kernel image and initramfs will have this version appended to its filename.

Now, the packages: The 'linux' package will depend on the following packages: linux-version-3.1.X-Y (this is the pkgname, without version) and linux-version-3.1.X2-Y2. The 'linux' package will also manage symlinks like vmlinuz-current and vmlinuz-old. Now, we need a new feature in pacman: we need to be able to obsolete a package so pacman uninstalls it on its own.

Anyway, this approach is not entirely thought out yet, I still see a problem when someone skips a few kernel versions, and with external modules. As I said, we will transition to this slowly rather than in one step.

----------------------------------------------------------------------------------------------------------

So what is the actual holdup on this?

As far as I can see, this can be done today, as long as users are responsible for uninstalling their own kernels (which is not an unreasonable idea).

Do we just need (questionable) support in pacman for obsoleting a package according to user-defined rules?
If so, has anyone opened a ticket against pacman and/or discussed it on the pacman-dev mailing list?

Because an excellent way to make sure it never happens is to forget all about it and then say no one is interested in it because no one is sure what they actually want.
Comment by Behnam Momeni (sbmomeni) - Thursday, 21 July 2016, 16:33 GMT
It is possible to obsolete old kernel versions with the current pacman.
It is as simple as defining a "conflict" rule from the virtual linux package to the older real linux packages.

What if a user skipped some kernel versions?
Define the "conflict" rule to N (e.g. 10) older versions and it will work flawlessly for users who are not N versions behind the Arch release cycle.
Comment by Eli Schwartz (eschwartz) - Thursday, 21 July 2016, 16:46 GMT
Yeah, for your arbitrarily chosen target users and your arbitrarily chosen number of manually-specified older kernels, which kind of defeats the purpose.

... I still don't see what is wrong with requiring users to manage their own orphaned packages, in which case this can be implemented as soon as someone decides they want to do it.
But if automatic purging of old kernels is what is stopping this from being implemented, it would be nice to at least know that that is the reason.
Comment by Amos Onn (amos) - Thursday, 21 July 2016, 17:38 GMT
It seems to me that people are trying to address two different (though similar) problems:

One is the inability to have multiple versions of the kernel at the same time. This is a problem in case you're not sure the new version works, and want an old one to fall back on.
The other is the inability to leave a copy in the filesystem of only the *currently running* kernel and modules. This is a problem for loading new drivers after an upgrade, or for hibernating and resuming system with the same kernel.

In order to solve the first problem, something akin to having a different package for each kernel version is required; and then there is the question of how to manage the old versions, via package conflicts, obsolete package feature for pacman, or just leaving it to the users.
In order to solve the second problem, a script which saves the current kernel and modules at init, perhaps with a bit more (see babali's comment https://bugs.archlinux.org/task/16702#comment149179) is enough.

Now, the question is, which of the two problems are we trying to solve? It is true that a solution for the first one also solves the second one, but is probably more complicated (hence the stalling).

I, for one, think solving the second problem is enough; and as AndyRTR commented here https://bugs.archlinux.org/task/16702#comment149173, for the fallback issue you have -lts kernel versions.
Comment by Radek Podgorny (rpodgorny) - Thursday, 21 July 2016, 18:11 GMT
i totally agree (with solving just the second problem). ...adding the need for manual uninstalling of old kernels would be a step back, imho. (for example, many of my systems are quite limited on disk space)
Comment by Vyacheslav Stetskevych (tskevy) - Thursday, 21 July 2016, 18:25 GMT
I'd like to
1) have all new kernels installed by default (i.e. have the linux package depend on the latest real kernel package).
2) have a post-install hook which will remove kernel packages, with a configuration file to specify how many releases of the kernel it should keep. It should always leave both the latest and the current running kernel in place.
What do you think?
Comment by Eli Schwartz (eschwartz) - Thursday, 21 July 2016, 18:46 GMT
Automatic uninstalling of old versioned kernels requires either ugly usage of the conflicts field, or else extending pacman. (Post-install hooks cannot invoke pacman, since the database will be locked by the parent pacman process.)

Copying the kernel to outside of pacman's bookkeeping is ugly and not simple at all, besides which in order to actually use it in any way you'd need to copy it right back. In other words, you are literally trying to trick pacman into thinking a package is uninstalled while leaving the files exactly where they were.

...

As per the original intent of this ticket, the solution is *still* versioned kernel installs.
So-called "simpler" solutions are in fact more complicated, and IMHO against the Arch philosophy.

So, it would be nice to hear from the kernel maintainers if "automatic obsoletion of old kernels, possibly according to user-defined rules" is the only blocker.

Alternatively, bikeshedding is more fun than solutions, so carry on.
Comment by Behnam Momeni (sbmomeni) - Thursday, 21 July 2016, 19:08 GMT
I filled a feature request for the pacman package (https://bugs.archlinux.org/task/50121) which allows installing the latest kernel version while keeping the current kernel version even if the user has skipped N (unknown) kernel versions.
Comment by Amos Onn (amos) - Thursday, 21 July 2016, 19:26 GMT
Not all discussion is bikeshedding. Of course a plethora of solutions, minutely different from each other, can be suggested and would get us nowhere; but I think that discussing the options we have broadly, not to mention the requirements of a solution, does help dispelling our current "no one is sure what they actually want" situation, to quote you. If we manage to break down and formulate a few (two-to-three) options with their merits and downsides, then it will be much easier to choose one of them and implement it. It seems to me that we have now two main (categories of) approaches, and what we need is to flesh them out.

I agree that copying files outside of pacman's bookkeeping is not the most elegant of solutions, but it is a possible one. There are precedents for unowned symlinks, such as /etc/localtime, admittedly more in configuration than in binaries. Having a temporary copy of the current modules doesn't sound that bad, perhaps it can even sit outside /usr/lib/modules.

That said, I agree that having multiple kernel versions is definitely also a reasonable solution, and that Arch users should be able to manually manage them. At the very least, it is also possible to have a linux-latest package, so users which don't want to have this feature can go on with the current way of things. So, I second the question whether outside of the obsoletion managament, there are no problems with implementing this.
Comment by Dennis Schridde (devurandom) - Thursday, 21 July 2016, 20:58 GMT
You could install multiple kernel packages, each carrying the version in their name, and then use something like Gentoo's eclean-kernel, which is a script that scans /boot for old kernels and removes them. With Arch Linux binary packages of kernels you could instead just call pacman -R from that script.

[1]: https://packages.gentoo.org/packages/app-admin/eclean-kernel
Comment by Sergey (versusvoid) - Sunday, 04 September 2016, 19:28 GMT
Hacky, fully automatic solution for keeping several last kernels, with small caveats:
https://github.com/versusvoid/archlinux-kernels-backup
Comment by Hugo Landau (hlandau) - Saturday, 24 September 2016, 23:17 GMT
Here's my suggestion for solving problem no. 2:

This should be rectified by causing the modules in the linux package not to be removed on upgrade if they correspond to the running version. Instead, the module directory should be marked for deletion. A script running at boot can then remove any old modules marked for deletion where they do not correspond to the running version.
Comment by deusstultus (deusstultus) - Wednesday, 16 November 2016, 07:11 GMT
Primary need here is access to a recovery console in a catastrophic case right? Coming to mind after #51818 today...

Fix here seems simple using pacman builtin backup. pacman manages backup history depth already and $(pacdiff -o) will flag the offending files for the clean system purists in us all. No less elegant than the way /etc/pacman.d/mirrorlist is handled -- well, excepting that these are binaries. These would be natural pairs having been owned by the same package before orphaning.

In kernel packages:
-backup=("etc/mkinitcpio.d/${pkgbase}.preset")
+backup=("etc/mkinitcpio.d/${pkgbase}.preset" "boot/vmlinuz-${pkgbase}" "boot/initramfs-${pkgbase}.img" "boot/initramfs-${pkgbase}-fallback.img") #fallback merit contestable

Bootloaders then need to know how to find these and register them. Looks like grub already does in 10_linux through flagrant wildcard use. Otherwise, light sed/awk lines in the PKGBUILD can clone out for finding .pacsave suffixes and adding below the primary boot cases.

EDIT: The first word in this report is "Versioned" and I acknowledge I ignored that, but it's not an altogether inelegant idea despite being admittedly limited.
Comment by Jan Alexander Steffens (heftig) - Wednesday, 16 November 2016, 07:37 GMT
No, backup is not the right approach. Firstly, it only applies to modified files, so you'd never get a backup of the kernel. Secondly, the initcpios aren't even part of the package (and can't be sensibly made so), so you'd never get a backup of the them, either.
Comment by Richard (archer666) - Monday, 21 November 2016, 11:35 GMT
As a fairly new Arch Linux user (but otherwise using Linux since 1994) I have been bitten by this issue, described here:
https://bugs.archlinux.org/task/51873

As this issue has been closed with reference to this task (which is ok) and a thoroughly shitty comment (which is not ok), I move here.

If I understand correctly, this issue is in "undecided" state since 2009. Looking at the technicalities, I am not sure why.
The Linux Kernel deployment mechanism is such that several kernel versions may be installed in parallel on a given system.
Why does Arch not make use of that?
Where is the insurmountable problem of having a (configurable) keep last N kernels option, where N can be - say - 3?
What problem is there to at least provide a compatibility symlink for same kernel versions of an older patchlevel?

And finally in reference to the aforementioned shitty comment in task 51873: So if Arch decides to update the Kernel package it really is considered a "bug" on the users side to not immediately reboot? Srsly?

Comment by Jan Alexander Steffens (heftig) - Monday, 21 November 2016, 11:42 GMT
Pacman does not support that use case. You cannot install multiple versions of the same package. Putting a version into the package name causes a large maintenance burden on the side of the repository (the repository management being nearly unmaintained is the reason we're still stuck with SVN).

There is no guaranteed compatibility even between kernels of the same minor version, so allowing new modules to be loaded into an old kernel is unsafe and may cause crashes or irrecoverable corruption. This has bitten us in the past.

Yes, after a kernel upgrade you need to reboot if you want to load any modules.
Comment by Richard (archer666) - Monday, 21 November 2016, 11:53 GMT
If you paid close attention to https://bugs.archlinux.org/task/51873 we are not even talking about different minor versions.

BOTH kernels were 4.8.8

And you are wrong - BTW - with the compatibility between kernels with minor version difference.
see here:
https://lkml.org/lkml/2011/5/29/204
https://en.wikipedia.org/wiki/Linux_kernel#Version_numbering

Quote: "An additional number (now the third number) would be added on when necessary to designate security and bug fixes, as for example with 3.0.18"

You will never have API, IOCTL or otherwise binary incompatible changes with different minor versions. Show some counter-example.


I moved my notebook from Gentoo to Arch to test how things go here. Not having to compile is nice. But do I understand correctly that there are no different versions (ok) and not even "Slots"? of packages?

IIRC the whole Nvidia-OpenCL stuff pulls in a gcc5, while the system has a gcc6?

So no way to have PHP 5.x and PHP 7.x at the same time?


And no, I cannot accept to have to reboot after a 4.8.8-1 to 4.8.8-2 "upgrade" (you may want to look up the meaning of "upgrade")
Comment by Jan Alexander Steffens (heftig) - Monday, 21 November 2016, 12:12 GMT
KConfig changes can break ABI. Any package release might change the configuration, even if the version remains the same.

There are no slots. There can be at most one package with a given name installed. You can have multiple versions of the same software, but you need to give them separate names. As far as the repository and package management is concerned, that makes it separate software.
Comment by Michael G. (Commod0re) - Monday, 21 November 2016, 18:34 GMT
"it's more work to maintain" is a poor reason to waffle on supporting this use case. If better repo management tools are needed, I am more than happy to lend some spare hours working on improv(ed|ing) tools and/or package install scripts if implementation of improvements is the only thing holding back a resolution of this usability wart. Having a known working backup kernel is not the only use case where having multiple kernels installed can be desirable.

In the Debian/Ubuntu world, `linux-image...` is a metapackage that always points to the current kernel. This allows for tracking new kernels without immediately removing the old one, and users/admins the freedom to reboot when they choose to, as there are no slots in that packaging system either (and authoring a deb is a lot more work than writing an equivalent PKGBUILD). This could be an acceptable mechanism to implement this without needing to make significant changes to pacman and packaging tools.


With regards to KConfig having the possibility to break ABI for modules, that is true. Kernel ABI compatibility is guaranteed for the userspace ABI only (see: https://www.kernel.org/doc/Documentation/stable_api_nonsense.txt)
Comment by Hugo Landau (hlandau) - Tuesday, 22 November 2016, 03:08 GMT
I already pointed out in my comment https://bugs.archlinux.org/task/16702#comment151222 a solution that doesn't require multiple versions of the same package to be installed, nor a new package name for each version.

pacman is capable of not removing files it installed on install when a package is removed, like any package manager. Simply mark kernel modules as such files and use a hook script executed on boot/shutdown/after pacman execution which cleans up any modules which match neither the installed package version nor the currently running kernel version.
Comment by Eli Schwartz (eschwartz) - Tuesday, 22 November 2016, 04:48 GMT
@hlandau,

The problem with your solution is that it doesn't actually work. pacman is not, in fact, capable of *not* removing files in a package when it uninstalls the package. Not unless you extend pacman first, by which logic you can do a lot of things. And as has already been noted, extending pacman itself for the specific use case of the linux kernel is ugly.

@heftig,

"Putting a version into the package name causes a large maintenance burden on the side of the repository (the repository management being nearly unmaintained is the reason we're still stuck with SVN)."

What exactly is this maintenance burden? I cannot say I have personal experience, but I would assume the repo management already knows how to handle split packages when the pkgname array changes from one version to the next. Unless you are saying that when part of a split package is removed from the PKGBUILD, it falls out of sight and manual work is required to purge it from the repos? In that case, maybe this is an opportunity to fix something. :p

Saying "this particular solution won't work because the repo management is unmaintained" is, as @Commod0re said, a poor reason to waffle, but moreover, it is essentially saying "we have no intention of dealing with this bugreport in any way, because it involves WORK".
So it is unmaintained -- if you (rhet.) are motivated to solve this bug, you can one-off patch the repo scripts, and then let it languish unmaintained again. Unmaintained just means nobody cares about it, not that it is impossible to work on it.


Is that the blocker then? (Were you wearing your Arch Dev hat when you said that?)

...

Or, does anyone know what the blocker is?

I really want to hear from someone who can say "this is what we need in order to have versioned kernels, if you implement it I will push in support of finally fixing this".

Going over the comments again, it seems there are two serious concerns raised by the people actually in charge around here.
@brain0 mentions earlier ( https://bugs.archlinux.org/task/16702#comment80122 ), that he wants a way for pacman to obsolete and remove old versions automatically. Additionally, the bootloader would need to be reconfigured on every update because not all bootloaders support symlinking a *-latest kernel/initramfs. (I am looking at you, UEFI/ESP on stupid FAT!)

Old versions IMHO are the user's responsibility, but an argument could be made that stuffing the ESP with obsolete kernels is a bad idea. ;)
Grub could ship a pacman hook to update via grub-mkconfig after (versioned) kernel installs -- this might not be a terrible idea anyway, and users are as always free to disable hooks they don't like. So much better than in the install script which can't be disabled!
I don't know about other bootloaders, how easy it is to update automagically.
Comment by Hugo Landau (hlandau) - Wednesday, 23 November 2016, 04:47 GMT
pacman seems perfectly capable of not removing config files. Or just ship with a directory /lib/modules/$foo.tmp and use a postinstall script to rename it to /lib/modules/$foo, or similar, preventing pacman's removal process from finding it.
Comment by Eli Schwartz (eschwartz) - Wednesday, 23 November 2016, 05:41 GMT
1) You are not comparing apples and oranges, you are comparing apples and decorative china knicknacks. pacman knows how to leave alone any files explicitly marked as backup files, in the event that they have been modified from the original packaged version -- which is how it handles user-editable config files. Non-edited files are deleted as usual, and in any event marking 4822 files as backup files would be a terrible idea even if it worked, which it doesn't.

2) If you really want to install a kernel outside of pacman, there are plenty of ways already. Any solution MUST involve pacman continuing to know that the files exist, or else there is no point of using package managers and we might as well go back to the old days of installing software via `./configure && make && sudo make install`. Or untar'ing the *.pkg.tar.xz into / so it is present on disk but we don't have to worry about pacman knowing about it.
Untar'ing it by hand actually works really well, if you just want to cut it adrift from pacman, why keep around a dummy package which doesn't do anything?

To add insult to injury, pacman would get all worried when about 5000 files drop off the face of pacman, and end up being profoundly irritating.

...

Both those ideas were rejected already, and not only because they have nothing to do with the actual request of this task.
If you have any more horrible and ill-thought-out ideas, please don't mention them. Your comment is the poster child for @brain0's previous comment:

"I am thinking about simply closing this report. It's going nowhere, everyone just posts new "solutions" that we simply won't implement. I posted a game-plan in an earlier comment, which will eventually be implemented. I consider this very low priority, and thus the required changes will be made slowly. Leaving more and more comments with brilliant "ideas" here (which spam my inbox with bugtracker emails) will not speed it up."

Might I add that the aforementioned brilliant "ideas" have already long since driven that particular Arch dev to unsubscribe from this task? Good going, everyone, hope you are happy now you had a chance to talk...

I, personally, just want to know what the status is of the official, blessed "best solution", and how that can be brought to fruition faster if possible.
Comment by Richard (archer666) - Wednesday, 23 November 2016, 10:37 GMT
LOL ... just LOL.
One month in Arch and already great cinema.
You know - if you "all" want to have a "task" open for over 7 years and dancing around why a solution is not possible, it's actually fine with me.

I'm perfectly fine to keep Arch contained on my notebook only and spare the other dozens of machines that are on a rolling release distro. I also can endure this little hassle, because for me, it is either putting the linux kernel in IgnorePkg=linux or

# /lib/modules
drwxr-xr-x 4 root root 4096 Nov 22 20:04 4.8.10-1-ARCH
lrwxrwxrwx 1 root root 13 Nov 18 09:53 4.8.8-1-ARCH -> 4.8.8-2-ARCH/
lrwxrwxrwx 1 root root 14 Nov 22 22:17 4.8.8-2-ARCH -> 4.8.10-1-ARCH/

Because after it upgraded to 4.8.10 yesterday and NFS stopped working and you really F***ING expect the user to reboot, then you have a distro that is worse than Windows95. But hey - as long as *I* can fix it, no hard feelings.

Comment by Alexandre Bique (babali) - Wednesday, 23 November 2016, 10:55 GMT
This issue is a joke. I'm also getting super pissed by those reboot, "oh shit I did update and now my computer does not work anymore, I have to reboot".
It should become top priority.
Comment by Hugo Landau (hlandau) - Thursday, 24 November 2016, 04:58 GMT
> 2) If you really want to install a kernel outside of pacman, there are plenty of ways already. Any solution MUST involve pacman continuing to know that the files exist, or else there is no point of using package managers and we might as well go back to the old days of installing software via `./configure && make && sudo make install`. Or untar'ing the *.pkg.tar.xz into / so it is present on disk but we don't have to worry about pacman knowing about it.
>Untar'ing it by hand actually works really well, if you just want to cut it adrift from pacman, why keep around a dummy package which doesn't do anything?
Old modules can get cleaned up by the cleanup script, not the package manager. There's no particular reason why we should be bothered as to whether it's pacman or another mechanism managing the lifetime of certain files, and many files on the system with managed lifecycles are managed by other processes. The proposed solution provides mechanisms which ensure that all of the files exempted from package manager cleanup are cleaned up by other means. There is no leakage here. No currently available solution, or your suggestion of just installing a kernel manually, offers this.

>To add insult to injury, pacman would get all worried when about 5000 files drop off the face of pacman, and end up being profoundly irritating.
What on earth does 'worry' mean in this context? This is not scientific language.
Comment by Gaelic (gaelic) - Monday, 09 January 2017, 06:34 GMT
I just found this on reddit. I do not know if this is already known, but its a quite simple and useful solution and should be incorporated as a default:

https://www.reddit.com/r/archlinux/comments/4zrsc3/keep_your_system_fully_functional_after_a_kernel/

It just keeps the running version of kernel modules when upgrading.
Comment by Dennis Schridde (devurandom) - Friday, 24 March 2017, 19:23 GMT
A hook + script to partially mitigate this issue, if you are using systemd-boot, can be found here: https://github.com/devurandom/archlinux-hook-kernel-install
Comment by Liam (networkimprov) - Monday, 15 May 2017, 18:16 GMT
A related issue which should have a simpler fix is that any problem which stops mkinitcpio after a kernel update renders the system unbootable. I suggested:

Let the kernel package install a flag, e.g. /boot/outdated-initramfs.
Let mkinitcpio remove the flag on successful completion.
Let grub skip the initramfs if the flag is present.

The response by Dave Reisner (falconindy) - Monday, 15 May 2017, 09:54 GMT-8
>You have a fundamental misunderstanding as to how the boot process works.

That may be, I'm used to u-boot. Does grub have no way to do the above?
Comment by Eli Schwartz (eschwartz) - Monday, 15 May 2017, 23:19 GMT
@networkimprov,

Please don't spam unrelated bugs, just because that bug ( FS#54052 ) was closed doesn't mean you can just carry on that bug's conversation here instead.
Comment by Sergey (versusvoid) - Saturday, 28 October 2017, 07:59 GMT
Example how kernel can be packaged to support versioning (and a solutions to bug for those who want it right now):
https://github.com/versusvoid/arch-linux-rolling

Grub hook must be more sophisticated though.
Comment by Christian Rebischke (Shibumi) - Friday, 18 May 2018, 13:08 GMT
If we ever think about switching from wpa_supplicant to the new internet wireless daemon, we should think about this issue here as well.
IWD is strongly relying on kernel crypto. Therefore if you install a new kernel and don't reboot after it and try to restart the iwd daemon it will not work because of missing kernel crypto modules..
Comment by Ivan Agarkov (kreon) - Wednesday, 23 May 2018, 21:11 GMT
Kernel wipe mechanism was introduced in centos/redhat distros with installonly_limit=X parameter in yum.conf. So we could use the same.
Comment by Tommy Schmitt (spinka) - Saturday, 11 August 2018, 10:48 GMT
There is hook available which keeps current kernel functional after update: https://github.com/saber-nyan/kernel-modules-hook
Comment by Terry Tibbles (terry_tibbles) - Wednesday, 09 January 2019, 07:24 GMT
What about an option in pacman that lets you choose how many kernel versions to keep?
Comment by Terry Tibbles (terry_tibbles) - Friday, 29 March 2019, 04:40 GMT
This seems to be down to 1 thing: the ability of pacman to keep multiple versions of the same package. If it was able to keep multiple versions of the same package, multiple kernel versions could be installed (for example), or if the kernel modules were made into a separate package, multiple versions of kernel modules could be kept (to prevent the problem of the kernel modules not being available).
Comment by Sergey (versusvoid) - Friday, 29 March 2019, 05:48 GMT
@terry_tibbles
Nope, it's not. See discussion (https://lists.archlinux.org/pipermail/pacman-dev/2016-September/021400.html) about patch I proposed with this "ability".
Kernel package can be versioned and managed somewhat similar to java-runtime, but maintainers just do not bother.
Comment by Paul Gideon Dann (giddie) - Friday, 29 March 2019, 08:42 GMT
This problem goes away entirely with the hook & service provided by the package linked above (https://aur.archlinux.org/packages/kernel-modules-hook/). Seems like it wouldn't be too difficult to make this behaviour more official. Seems like a pretty reasonable solution to me.
Comment by Harold (MrHaroldA) - Friday, 29 March 2019, 08:45 GMT
kernel-modules-hook is an excellent solution; it (or something similar) should most definitely be included in Arch.
Comment by Daan van Rossum (drrossum) - Friday, 29 March 2019, 08:54 GMT
What is the advantage of installing a kernel a 'long' time before you reboot?

Missing kernel modules is not the only issue. There is also the hibernate-restore issue.

IMHO, implementation of a convenience feature that enables installing a new kernel 'long' before a reboot is not worth much effort and/or code complication.
Comment by Daan van Rossum (drrossum) - Friday, 29 March 2019, 08:55 GMT
What about the following simple alternative:

Add a --unignore flag to pacman's -S mode that will tell pacman to treat all packages in IgnorePkg as if they were explicit targets. Then, `pacman -Suy --unignore` would ask:
:: linux is in IgnorePkg/IgnoreGroup. Install anyway? [Y/n]
:: linux-headers is in IgnorePkg/IgnoreGroup. Install anyway? [Y/n]
:: acpi_call is in IgnorePkg/IgnoreGroup. Install anyway? [Y/n]

That would simplify a just-before-reboot upgrade without having to remember which packages are currently being ignored by pacman.
Comment by Alexandre Bique (babali) - Friday, 29 March 2019, 08:59 GMT
drrossum, your argument does not stand.

The problem is that running pacman -Syu might break your system and force you to reboot.

This issue is quite trivial. To be honest I can't explain why it is not fixed already.
Comment by Christian Rebischke (Shibumi) - Friday, 29 March 2019, 10:13 GMT
Another good reason for versioned kernels installs is the new systemd-boot timer feature. With this feature you are able to automatically reboot a fallback kernel if the current kernel fails. A fallback kernel could be a prior version of the kernel.
Comment by Daan van Rossum (drrossum) - Friday, 29 March 2019, 10:18 GMT
Would it not be more useful to fallback to linux-lts in that case? At least there is a simple alternative for this as well...
Comment by Christian Rebischke (Shibumi) - Friday, 29 March 2019, 10:23 GMT
Are there any objections against moving https://aur.archlinux.org/packages/kernel-modules-hook/ into community and forking the project kernel-modules-hook on github to our github organisation?
I would like to support this and care for it. The current situation is indeed annoying and I don't see any other solution for it in the near future?

Opinions?
Comment by Christian Rebischke (Shibumi) - Friday, 29 March 2019, 12:08 GMT
@drrossum
Yes falling back to LTS kernel works as well.
Comment by Daan van Rossum (drrossum) - Friday, 29 March 2019, 12:28 GMT
@babali If you accidentally update your kernel but were not planning to reboot you can always still downgrade the kernel again.
Comment by Alexandre Bique (babali) - Friday, 29 March 2019, 13:00 GMT
@drrossum, not if your cache was cleared.
Comment by Eli Schwartz (eschwartz) - Friday, 29 March 2019, 15:10 GMT
@shibumi,

Manjaro has similar scripts, which provably resulted in unbootable systems because they were broken and deleted /usr/lib/modules, thereby removing all modules for all kernels, both the booted kernel as well as any newer one that was installed. (Also, I believe they installed the broken script by default.)

This "solution" has been suggested and rejected a number of times in this bug report. It is attempting to solve one half of the problem at completely the wrong level, results in pacman not knowing what files exist on the system, and is racy. I don't believe we should do it.

The status has not really changed since 2011 (per https://bugs.archlinux.org/task/16702#comment80122 and https://bugs.archlinux.org/task/16702#comment82408 ), as far as I'm aware. The kernel can and should be versioned, which allows you to both boot from and load modules from multiple kernels, and currently the only conflicts between packages are the files in /boot, /etc/mkinitcpio.d and /usr/share/libalpm/hooks. All we need is, for kernelname=linux :


1) Rename those files from *${kernelname}* to *${kernelname}-${kernelversion}*, and rename the package to pkgname=${kernelname}-${kernelversion} and create a dummy "${kernelname}" package which depends=("${kernelname}-${kernelversion}").

2) Copy/symlink the bootloader files. A post_install/post_upgrade script in "linux" which checks whether /boot is a fat* filesystem and either symlinks /boot/vmlinuz-${kernelname} to /boot/vmlinuz-${kernelname}-${kernelversion} or copies it, for example, by running `if [ -f "/boot/vmlinuz-${kernelname}-${kernelname}" ]; then ln -sf ... 2>/dev/null || cp ...; fi`. mkinitcpio could have an extra hook which generates a second initramfs, but ideally would learn to copy over the existing one. Since the preset is written in bash, we could just inject the same ln command with naming tweaks, and then exit 0.

3) Someone needs to db-repo-remove old ${kernelname}-${kernelversion} packages from the official repositories, to prune cruft. This could be automated via a small shellscript stored in svn instead of running testingpkg + testing2x, or for bonus points, implement automatic removal of no-longer-existing split pkgnames in dbscripts and I will gratefully merge it as there are already plenty of cases where this is needed by hand (even if it isn't a regular fact of life for any given package).

...

Points #1 & #2 are trivial to implement, but it adds maintenance burden for #3.
Comment by Mathew (Bird-or-cage) - Friday, 29 March 2019, 15:11 GMT
> Would it not be more useful to fallback to linux-lts in that case?
Assuming linux-lts is regularly confirmed as bootable on the machine in question, and was not updated at the same time as linux (thus becoming not confirmed as bootable) , yes, it would be more useful.
Comment by Jake Kreiger (Magali75) - Friday, 29 March 2019, 18:15 GMT
@eschwartz

Accidentally that solution works and integrating it is trivial. I bet your solution won't happen in next few years.
Comment by Bertrand Michas (berturion) - Tuesday, 28 May 2019, 05:05 GMT
@shibumi, since the submitted solution of @eschewartz, that looks like the ideal one, may not be implemented very soon, integrating the existing "kernel-modules-hook" in the community repo is not so bad in my opinion. I already installed this AUR package in all my Arch systems and on some friend's ones and I am very happy with it. It's up to the user to install it or not. As I see that the current maintainer says that he has not very much time to take care of it, I think that your initiative is good. Even if it will be obsolete in the long term, it is, for now, helpful for many people.
Comment by Avi H.D (Strykar) - Tuesday, 19 November 2019, 09:03 GMT
What are the technical merits to not adopting the pacman hook approach of https://github.com/saber-nyan/kernel-modules-hook/issues/4 which has a TU, @shibumi, willing to adopt it? This is ten years old.
Comment by Chris Billington (chrisjbillington) - Wednesday, 19 February 2020, 22:30 GMT
If anyone is interested, I've been experimenting with repackaging Arch kernels basically following Eli's outline in his previous comment. The approach works well, as far as I can tell. Things are a little different (simpler, I think!) than when Eli wrote the comment a year ago owing to the fact that the kernel packages no longer own the files in /boot.

You can see my PKGBUILD here: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=linux-versioned-bin

This PKGBUILD is fetching the already-built linux{,-headers,-docs} packages from a mirror or the Arch archive, and repackaging them with some changes. Those changes are:

* Including the full arch package version number in the names of the packages themselves
* Replacing the contents of /usr/lib/modules/<version>/pkgbase with the kernel name and version (in the same format as the name of the directory for the kernel in /usr/lib/modules). This is what mkinitcpio sees and uses as the name of kernel - so that's what shows up in GRUB for example.
* Using the versioned package names instead of pkgbase in the symlinks /usr/src/<pkgname> and /usr/share/doc/<pkgname> in the headers and docs package

There are three dummy packages: linux-versioned-bin, linux-versioned-headers-bin, and linux-versioned-docs-bin, which depend on the latest versioned packages; and three packages at any one time, linuxX.Y.Z-archN-M-bin, linuxX.Y.Z-archN-M-headers-bin, and linuxX.Y.Z-archN-M-docs-bin for the latest kernel, headers, and docs packages themselves.

Since the package names change when the PKGBUILD is updated, when there is a new version the old ones simply vanish from the AUR. They won't be removed locally from your machine of course - up to you to remove orphan packages at some later time.

The linux-versioned-bin package optionally depends on the AUR package grub-hook: https://aur.archlinux.org/packages/grub-hook/ , which is a pacman hook to run grub-mkconfig after kernel changes. If you use grub (and don't roll your own grub.cfg), you'll need to do this one way or another, and with the hook it is automatic, so highly recommended.

The package also conflicts=(linux), so you'll need to remove the main kernel to test it, and switch to dkms versions of any out-of-tree drivers you need.

If this approach were replicated in the official packages, I think this step mentioned by Eli would be the remaining thing to be done, otherwise if I'm understanding correctly the old packages will remain in the repos:

> for bonus points, implement automatic removal of no-longer-existing split pkgnames in dbscripts

I'll also make a non '-bin' package based on [core]/linux - just avoiding local compilation for now since the kernel takes ages.

Feedback appreciated, I'm not responsible if you make your machine unbootable ;).
Comment by DJ Lucas (DJ_Lucas) - Saturday, 01 August 2020, 21:54 GMT
> for bonus points, implement automatic removal of no-longer-existing split pkgnames in dbscripts

I'm confused by this. Just remove the packages and the cpio archives as usual using a default systemd timer (monthly?) that calls pacman. The bin package worked great on real Arch (and only minor tweaks were required locally to adopt) and should also work for community if there is interest in adopting - nothing changes for existing kernel maintainers in core/community. KISS method and I like it. So, for the remaining blocker, what's wrong with adding a simple monthly systemd timer? A user can always create a copy in /etc and make it run more or less frequently. They can do the same with the LTS packages. Add the timer to the grub-hook package (probably rename that as something to do with "kernel-maintenance") and allow for differing retention and initrd naming scenarios (use an array for filenames and include all common names currently in community packages), and enable it by default as part of post-install. For community adoption, your split meta package gains a hard dependency on what is currently named 'grub-hook' and both provides and replaces the equivalent core/community packages. If one of the other Arch consumers adopts it as well, it's a simple change for a cron-monthly job instead of a timer. Also, there is probably no reason to keep all of the fallback images, only the latest. I believe this addresses the remaining concern for users.

AFAICS, although technically addressed in the above, two previous points of contention do actually remain: the removal is not done as a part of regular package maintenance (pacman -Syu), but old packages will be removed, both predictably and consistently; and whomever maintains a like package in community will have to retire the old packages manually. I'm not familiar with how Arch community maintains their repos. Major overkill IMO, but if it is still a point of contention that last part could easily happen with a similar maintenance script that simply runs repo instead of pacman with retention set to 1. Anyway, I'm going to use similar locally. Thanks again.

As far as a prune script, somebody more competent than I can probably use awk for list, but something like the following works:

ret=1

list=`pacman -Qs ^linux[0-9] | grep -vi -e doc -e headers -e "^ " | cut -d " " -f 2 | sort -t. -k 1,1n -k 2,2n -k 3,3n -k 4,4n`

vers=($list)

rems=$(( ${#vers[@]} - ${ret} ))

if [ "${#vers[@]}" -gt "${ret}" ]; then
for pkgver in `echo "${vers[@]}" | tr " " "\n" | head -n "${rems}"`; do
echo "pacman -R linux${pkgver}{,-docs,-headers}-bin"
done
fi
Comment by Chris Billington (chrisjbillington) - Monday, 03 August 2020, 04:47 GMT
The removal Eli is referring to is in Arch's repos, and doesn't refer to automatically uninstalling locally installed kernel packages.

I'd be happy for removal of locally installed kernels to be entirely the user's problem and not automated as part of the packages themselves, I think that would be in line with KISS principles.

The approach I'm taking with the AUR packages linux-versioned-bin [1] and linux-lts-versioned-bin [2] (which it sounds like you're using?) is to have a separate script remove-orphaned-kernels [3] that uninstalls orphaned versioned kernel packages, other than the one currently running. The user could run this script on a timer if they want, or the package could ship a timer, but I like just running it manually every now and again just like I would clear the pacman cache or other occasional system maintenance.

But yes, I think the blocking issue is with removal from the Arch repo database, and I am not familiar enough with how that works to contribute a fix for that.

[1] https://aur.archlinux.org/packages/linux-versioned-bin/
[2] https://aur.archlinux.org/packages/linux-lts-versioned-bin/
[3] https://aur.archlinux.org/packages/remove-orphaned-kernels/
Comment by Fabian (Tids) - Wednesday, 12 August 2020, 09:02 GMT
What I'm missing: An usual ESP is around 300-500MiB in size. So if a kernel update would install new versions, but not uninstall old versions, this would last only a few Kernel updates until ESP runs out of space and so would destroy all the systemd-boot and efistub users upgrade processes. This would also force the user to constantly doing manual interventions. Both of them are a huge no-go for Arch. Please #KISS

I'm, because of this, no fan to keep old kernels around by default.
Comment by Christian Kujau (ckujau) - Wednesday, 12 August 2020, 21:12 GMT
Although this bug report is named "Versioned Kernel installs" I think this is really "just" about keeping the currently running kernel in place when installing the latest kernel. This helps with regressions (the latest kernel won't boot, as the initial report described) and also with currently unloaded modules (once the currently running kernel (modules) are removed, one can no longer load new modules into the running kernel). Also, all of this could be configurable ("keep X kernels around") so space requirements would not be an issue. FWIW, SUSE/zypper has multiversion.kernels in zypp.conf; Fedora/DNF has installonly_limit in dnf.conf; Ubuntu has some (somewhat convoluted) APT hooks in /etc/kernel/postinst.d/apt-auto-removal present to do accomplish that.
Comment by DJ Lucas (DJ_Lucas) - Thursday, 13 August 2020, 00:30 GMT
That's the beauty of this approach. This goes in community, users can choose to install or not, core remains unchanged. It also nicely addresses the other use cases, set the auto-removal to one and run on boot, and it fixes the kennel modules issue. For those who want a couple of versions around to address potential compatibility or need multiple versions for development, it fixes those too.

The only deficiency that I can see is the maintainer load. What I posted earlier addresses that nicely for me in an automated way, but I'm a one man shop and I don't know how the community maintainers manage their repo. This is three full package removals, three new packages, and three package updates with each new kennel version, not just updating existing packages. This probably does not fit well within the normal maintenance routine. Again, I don't know for certain, but I certainly wouldn't find it unreasonable for new package additions and removal of existing packages to require some form of consensus for accountability. This would definitely qualify as an outlier, and would require a new internal process.
Comment by DJ Lucas (DJ_Lucas) - Saturday, 05 September 2020, 22:16 GMT
Alternative to the above locally, I've elected to simplify things based on Chris's stuff above. I figured I'd post here for review in the event that Arch devs are interested in it. I've basically just separated the kernel, so it's down to two repo-add's (two source packages, one with the three updated packages, api-headers, docs, and the linux meta package that depends on the versioned kernel package and the separate versioned kernel itself - no docs or api-headers), and one repo-remove (the old kernel, which personally, I'm keeping three of them in the repo, but this is not necessary for Arch). Please understand that these PKGBUILDs are not intended for Arch Linux (I can't see any obvious reason they wouldn't work I guess but I did remove a feature and a dependency and they are not intended for that purpose). I just posted here to show what I wound up doing to eliminate all of those changes (although I had it automated, and I did just rip off Arch's kernel config, almost verbatim, since I decided to clean up my PKGBUILDs and update that old github repo of the LFS bootstrap with changes necessary for Pacman). Chris, I hadn't thought about it, but you might be able to drop the versioned api-headers and docs, make it a bit easier on your users by doing two PKGBUILDs (one provides linux-${pkgver}). Also, it didn't even occur to me until just now (not my intended use case), but one could use a Pacman hook to keep only the latest two kernels and add a oneshot service with a delay (or depend on user target) to clean-up the old kernel on successful boot if desired. I think something like that as the default would make everybody happy (again, except the devs for the one lingering package) - devs that need multiple kernels can disable the hook/unit. Anyway:

https://github.com/djlucas/LFS-Pacman/tree/master/Main/linux-5.8.5-kernel
https://github.com/djlucas/LFS-Pacman/tree/master/Main/linux-5.8.6-kernel
https://github.com/djlucas/LFS-Pacman/tree/master/Main/linux

Oh yeah, please ignore any meandering comments - LFS's primary goal is user education, so like my posts, they sometimes get long-winded to avoid ambiguity. Also, please don't try to use anything else from that repo on Arch, or at least not as is - it very likely it will break stuff (LFS still uses split /usr, custom CA certificate utilities (though that might actually be worth looking at for the Arch developers given the ease of local modifications I came up with for make-ca/p11-kit and minimum deps), sysvinit not systemd, etc).
Comment by Josef Mutzenbacher (Joe2020) - Thursday, 24 September 2020, 14:48 GMT
To solve "problem #2", that is the problem being unable to load any modules after kernel package upgrade, there is the conflict that it seems to unavoidably produce files untracked by pacman or it requires an ugly clean-up mechanismen.

But IMHO the following "clean-up mechanismen" not to be considered as "ugly", as it is one working automagically:

The "old" modules have some meaning while the corresponding kernel is still up and running, but not beyond. So why not having the "old" modules in a tmpfs volume mounted at /lib/modules? They will be available as long as there is their kernel running having potential interest in them, but they are guaranteed (!) to be gone upon next boot. The point is that the on-disk state is the very same as with the current solution. A common shutdown, a crash or powerloss - it doesn't matter, there is no additional cleanup required, both old booted kernel and its modules will be gone.

What do I miss?
Comment by DJ Lucas (DJ_Lucas) - Friday, 25 September 2020, 23:59 GMT
Assuming nothing custom (NVidia, VirtualBox, VMWare, etc.) you'd be giving up a gig of RAM/swap as well as adding a 500MB file copy in one of the early systemd services (a lot of sit and spin). It's not ideal IMO, but it would work for that specific problem. Memory constrained systems (2GB) are out as well (limit is half RAM by default). I'm not sure why you think untracked files are unavoidable, there are several methods above that avoid it, including this one.
Comment by Josef Mutzenbacher (Joe2020) - Saturday, 26 September 2020, 11:35 GMT
DJ Lucas, you might have misunderstood one aspect of my suggestion:

The TMPFS-solution will only exist after having upgraded your kernel and last till shutdown. That means, it will be created not by an early systemd service all the time, but by pacman after kernel upgrade only. Consider it as a post-install hook of kernel packages. That means btw, that it would be easy to give the user a choice by an EXECUTE_MODULES_PRESEVATION_HOOK FALSE|TRUE option in pacman.conf or whereever. With defaulting to false, nothing would change for most users (reasonable, since most users will reboot immediately after kernel upgrade), but would give those who can't reboot immediately the possibility to alter the behavior.
Comment by Oleksandr Natalenko (post-factum) - Saturday, 26 September 2020, 12:50 GMT
For the sake of statistics:

[~]$ du -hs /usr/lib/modules/`uname -r`
92M /usr/lib/modules/5.8.0-pf5

Also, tmpfs doesn't consume all the reservation. Only what's being really used is… erm… used.
Comment by DJ Lucas (DJ_Lucas) - Saturday, 26 September 2020, 15:04 GMT
Not sure what "pf5" is but my numbers are from the default Arch x86_64 kernel package:
[pkguser@archtest3 ~]$ du -sh /lib/modules/`uname -r`/
442M /lib/modules/5.8.10-arch1-1/
[pkguser@archtest3 ~]$ pacman -Q | grep dkms
[pkguser@archtest3 ~]$

So the solution is effectively something along the lines of:
/usr/share/libalpm/scripts/kernel-pre-upgrade.sh:
TMPDIR=/tmp/modules-`uname -r`
install -vdm750 "${TMPDIR}"
cp -R /usr/lib/modules/`uname -r`/* "${TMPDIR}"

and

/usr/share/libalpm/scripts/kernel-post-upgrade.sh:
TMPDIR=/tmp/modules-`uname -r`
SIZE=$(du -hs /tmp/modules-`uname -r`/ | awk '{print $1}' | tr [:upper:] [:lower:])
# Give a little breathing room.
TSIZE=$(echo ${SIZE} | sed 's/[a-z]//')
TSIZE=$(( ${TSIZE} + 1 ))
SIZE=$(echo -en ${TSIZE}`echo ${SIZE} | grep -o [a-z]`)
mount modules -t tmpfs -o size="${SIZE}" /usr/lib/modules
install -vdm755 /usr/lib/modules/`uname -r`
cp -R "${TMPDIR}"/* /usr/lib/modules/`uname -r`
rm -rf "${TMPDIR}"

I'm not sure how to handle the hooks themselves, but that's ancilary for this part. Is that kind of what you were thinking? It works, the memory used temporarily is almost double the size (as predicted), but frees up as expected. The only caveat I can see is DO NOT do two kernel upgrades without a reboot in between. If you do your updates every night at 3:00 AM and reboot on the weekends, you will create a problem.
Comment by Daan van Rossum (drrossum) - Monday, 28 September 2020, 13:20 GMT
Using 500MB+ of memory for cleaning a directory upon reboot sounds more like a MS windows approach
Comment by Josef Mutzenbacher (Joe2020) - Tuesday, 29 September 2020, 13:00 GMT
> Using 500MB+ of memory for cleaning a directory upon reboot sounds more like a MS windows approach

To me having a broken system after upgrade is much more MS like...
Comment by Daan van Rossum (drrossum) - Tuesday, 29 September 2020, 15:08 GMT
But you're in control. You don't have to upgrade your kernel if you don't want to reboot.
Comment by Josef Mutzenbacher (Joe2020) - Thursday, 01 October 2020, 15:07 GMT
Well, actually it is more like you MUST NOT upgrade if you can't reboot immediately or you will have a broken system.

Look, after an upgrade the kernel image file is gone from disk, but still existent in RAM till reset. Don't you think it has it's logic to treat it's modules in an identical way?

That said I agree that there are very likely much better solutions, I just think my proposal is a good tradeoff for the time being, especially as it can't cause any harm if you don't opt-in to the hook in pacman.conf.
Comment by Daan van Rossum (drrossum) - Thursday, 01 October 2020, 15:15 GMT
You can still upgrade other packages. Put these in your IgnorePkg (pacman.conf), and before you reboot you issue:

pacman -Suy linux linux-headers linux-firmware
Comment by DJ Lucas (DJ_Lucas) - Friday, 02 October 2020, 04:36 GMT
Unrelated to the OT, but regarding the previous post, please do not ever put linux-headers in IgnorePkg. The api headers on disk should always be the ones that were in use when glibc was compiled. In truth, it probably doesn't matter much unless you are going a really long time without updates, but still not great practice.
Comment by Chris Billington (chrisjbillington) - Saturday, 03 October 2020, 02:14 GMT
If all you are looking for is a solution for the time being, the linux-versioned-bin and linux-lts-versioned-bin AUR packages I discussed upthread have served me well for many months. No lost modules after an update, and you just have to run remove-orphaned-kernels (separate AUR package) every now and again to avoid running out of space on your EFI partition.

You can run remove-orphaned-kernels after every update if you like, then you'll have at most one extra kernel installed (the currently running one, since it will never remove that). Mid update (after pacman -Syu but prior to running remove-orphaned-kernels), you'll have at most two extra kernels (the currently running one, and the second most recent one, if different), or three if you have both the lts and regular kernels. So if you run remove-orphaned-kernels after every update, your EFI partition needs to have space for three kernels, or four if you have both lts and regular kernels. With a 500MB EFI partition I haven't had an issue with space yet.
Comment by kiasoc5 (kiasoc5) - Tuesday, 26 April 2022, 00:45 GMT
Another option is systemd-boot with kernel-modules-hook package and mkinitcpio configured to produce UEFI stubs. Then when a new kernel is installed, the old kernel modules are backed up and the new kernel stub doesn't overwrite the old one. The EFI partition still has to be cleaned every now and then. This is the closest vanilla method I could get. Maybe kernel-modules-hook could be patched to officially support multiple kernels/versions.
Comment by Buggy McBugFace (bugbot) - Tuesday, 08 August 2023, 19:11 GMT
This is an automated comment as this bug is open for more then 2 years. Please reply if you still experience this bug otherwise this issue will be closed after 1 month.
Comment by Klaus Frank (agowa338) - Tuesday, 22 August 2023, 13:42 GMT
Also see: https://bugs.archlinux.org/task/79441
Integrating "kernel-modules-hook" may alleviate most of the issues here.
I've put it into a separate feature request, as it is not a versioned kernel install, but only a more streamlined kernel install until rebooted.

Loading...