FS#12384 - Kernel PKGBUILD proposition: Ease custom kernel compilation.

Attached to Project: Arch Linux
Opened by Nicolas Bigaouette (big_gie) - Monday, 08 December 2008, 18:18 GMT
Last edited by Tobias Powalowski (tpowa) - Thursday, 12 March 2009, 17:40 GMT
Task Type Feature Request
Category Kernel
Status Closed
Assigned To Tobias Powalowski (tpowa)
Aaron Griffin (phrakture)
Thomas Bächler (brain0)
Dan McGee (toofishes)
Allan McRae (Allan)
Architecture All
Severity Low
Priority Normal
Reported Version None
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 14
Private No

Details

Description:
I suggested a simple modification to the kernel26's PKGBUILD on the forum[1] and on the mailing list[2] with good feedback from users. Basically, these modifications allows to recompile a different kernel by only changing the pkgname. I am pasting what I have written:

Hi all!

Since a couple of years of using Arch, I often wanted to compile a custom
kernel, as many others did. There is many, many, many! forum threads about
that, a couple of wiki pages too, with questions and howtos.

Many efforts has been put to create new PKGBUILDs[3-4] to ease the effort of
this. Unfortunately, these PKGBUILDs suffer from some drawback:
1) They are placed on a wiki which can be edited by everybody. This mean it
is not a trusted place to store a PKGBUILD, specifically one as vital as the
kernel's.
2) They tend to be forgotten and not updated anymore. This can lead to an
unbootable kernel, rendering all this useless.
3) They do not take advantage of the devs' work: They are the best one to
trust for a quality PKGBUILD. They also have all the feedback from Arch
users, bug reports, etc. They know what has changed in the kernel and what
needs to be adapted for the installation, for example new "provides=()". A
PKGBUILD from a wiki page or a forum post might be good at the time of post,
but be outdated just days latter...
4) They build a kernel which is way different then the stock one, for
example because they don't have the same patchset.

Users most often wants to build a custom kernel just to try it, change some
configurations, add one or two patch, optimize for their own machines, etc.
Maily, they just want to take the existing one, change a couple of things,
and be ready. One also wants to install this kernel side by side with the
stock one, at least until his own kernel is stabilized and running as
wanted. Actually, Arch's stock kernel does not allow that.

What I propose is some simple modifications to the PKGBUILD so users can try
to compile their own kernel based on the original, trusted, working and
approved one. With theese modifications, one only needs to add a "kernel
name" to the pkgname: "pkgname=kernel26-mykernel" for example. Nothing else
is needed to compile a custom kernel. The created package would sit without
problem side by side with the stock kernel. Also, if the pkgname is kept to
"kernel26", then the stock -ARCH kernel will be built.

This approach has many advatanges:
1) The patch is relativelly small: not much is changed in the original
PKGBUILD.
1) Nothing changes for the devs. They continue to provide a quality kernel
to Arch users.
2) Users who wants to compile a custom kernel can just sync abs, change the
pkgname and install a new kernel, in the KISS philosophy. They know that if
they just change the name, they will have the same package so they can
easily test whatever feature/patch/bug solution/optimization they want.
3) Because one only needs to change the pkgname, they use the devs' high
quality, tried and truth PKGBUILD. When a new kernel is out, they can sync
the devs' modifications easily.
4) Bugs affecting the kernel can be easily tested for solutions. Affected
users just need to compile a "-testing" kernel where a patch is applied,
install it and see if it works, without having to risk breaking his system
by replacing -ARCH kernel with an unbootable one.
5) Better feedback to the devs since the users would be using (almost) the
same PKGBUILD as the original one.
6) Simplify a lot all these "custom kernels" threads and wiki posts.

I did "maintained" such a patch for myself because I wanted to keep in sync
with the original PKGBUILD. After posting it on the forum[5], I got some
positive feedback. People reporting back said they liked the simplicity and
the effectiveness of the method. They also positively tested it with
different configuration. The patch should be applied to revision 19747 for
i686 or 19749 for x86_64. It modifies the PKGBUILD and also slightly the
kernel26.install and kernel26.preset file.

Issues:
1) Firmwares are being installed to /lib/firmware by all kernels. Because
of that, a kernel other than stock's "-ARCH" will delete the
${pkgdir}/lib/firmware directory, meaning they will depends on the stock
kernel to provide them.
Possible corrections:
-One could add "kernel26" as a depdency
-Put firmwares in a seperate package and make all kernels depend on it


I am thus reporting this to be include in the official file. I think it
would be a great addition to an already great distro. I am open to
discussion on the subject. I hope I am posting this on the right mailing
list and that it can be included! :)

Sincerely,

big_gie


[1] http://bbs.archlinux.org/viewtopic.php?id=37579
[2] http://archlinux.org/pipermail/arch-general/2008-December/002937.html
[3] http://wiki.archlinux.org/index.php/Kernel_Compilation_with_ABS
[4] http://wiki.archlinux.org/index.php/Custom_Kernel_Compilation_with_ABS
[5] http://bbs.archlinux.org/viewtopic.php?id=37579


Additional info:
* package version(s) 2.6.27.7
This task depends upon

Closed by  Tobias Powalowski (tpowa)
Thursday, 12 March 2009, 17:40 GMT
Reason for closing:  Implemented
Additional comments about closing:  2.6.28.7-2
Comment by Nicolas Bigaouette (big_gie) - Wednesday, 21 January 2009, 18:30 GMT
Updated to 2.6.28.1
Comment by Aaron Griffin (phrakture) - Wednesday, 21 January 2009, 19:11 GMT
I like a few of these changes, but not all. Thomas? Tobias? opinions?
Comment by Nicolas Bigaouette (big_gie) - Wednesday, 21 January 2009, 19:16 GMT
Which one do you not like?
I must say there is one part which is not needed: the test for if the kernel has already been patched. I added this because it speeds up the recompilation. Run "makepkg -e" won't extract the source, so it does not need re-patching (which will fail because of the "|| return 1"). This is not needed for the custom kernel name, but a nice addition IMO, specifically if you build and rebuild and rebuild kernels (which you guys probably do a lot!)
Comment by Aaron Griffin (phrakture) - Wednesday, 21 January 2009, 19:32 GMT
_kernelname=${pkgname:8} - the :8 part is too magic-number-ish
_kernelname=${pkgname#*-} may work better and be more logical (remove everything up to the first dash)

I'd change the -eee comment to -custom or something, to make it more clear

I don't like the _arch_patched check, as you mentioned.

The if around the sed line seems silly, as if kernelname == "" then you will get the same behavior in both cases

And the firmware handling seems questionable. It almost makes me feel like we should split the firmware out into another package or something :S
Comment by Nicolas Bigaouette (big_gie) - Wednesday, 21 January 2009, 20:31 GMT
1) ${pkgname:8}
Yeah to much of magic numbers... What I wanted here is to have a variable contaning "-NAME" or nothing (""). Then if pkgname=kernel26, the stock kernel will be built. But this kernel have "CONFIG_LOCALVERSION=-ARCH", so the modules will be installed to /lib/modules/2.6.28-ARCH/ but with "kernel26.img"...
So because the variable contains "-NAME" or nothing, then it can be addded after "kernel26" where relevant (for example in the mkinitcpio config).
Note that with this logic in mind, I cannot have the variable containing just "NAME" instead of "-NAME": I would need to add "-" between ${_kernelname} and "kernel26" but if ${_kernelname} is empty, I'll end up with "kernel26-".

So, to get back to the magic number. I'd like to find an "automatic" way. ${pkgname#*-} would return "NAME" (and not "-NAME"). Again I can't simply add "-" in from of ${pkgname#*-} because it would result in "kernel26-" for the stock kernel. Also, if "pkgname=kernel26", then ${pkgname#*-} return "kernel26" and not an empty string... So the best bet might be "${pkgname#kernel26}":
$ pkgname=kernel26
$ echo ${pkgname#kernel26}

$ pkgname=kernel26-custom
$ echo ${pkgname#kernel26}
-custom
$ pkgname=kernel26-eee
$ echo ${pkgname#kernel26}

$ pkgname=kernel26-e
$ echo ${pkgname#kernel26}

$
But then, its driving me crazy: How come I cannot match "kernel26" when I have "-*[e]" as the name??

2) -eee
Of course... done.

3) _arch_patched
How come you don't like it? Its faster if you recompile often. Its just to ease the maintenance...
I'll remove it anyway for this.

4) sed line
Wow your right, really useless ;) Removed!

5) Firmware
I think I never had the conflicting problem before 2.6.27 or something... It's probably included by default since .27. So our choices are:
-Keep the test to remove the firmware: hackish and not nice...
-Create a new package which will contain these firmwares, and make the kernel's PKGBUILD depend on it.
I would prefer another package... Because even the wiki page about custom kernel compilation does not talk about that. All custom kernels (installed side by side) will have conflict about these firmwares.

There is also another positive aspect of putting the firmwares in a separate packages. I remember reading something on the incompatible license of some firmwares. Correct me if I'm wrong, I would say the vanilla kernel does not have such problematic firmware included, but if it does, then one could make sure he has a clean system by not installing then if not necessary. That would make it an optinal dependency. Or it could be a real dependency if If there is no license problem and these firmwares are vital to any working systems.

Comment by Nicolas Bigaouette (big_gie) - Wednesday, 21 January 2009, 20:35 GMT
Corrections from previous post.
TODO:
-Fix firmware issue
Comment by Aaron Griffin (phrakture) - Wednesday, 21 January 2009, 20:41 GMT
Re: #1
-e is a flag to echo, so it doesn't output it. I ran into it too.
This syntax will do what you want: ${pkgname/*-/-}
echo " ${pkgname/*-/-} " # note the spaces because of the -e thing here. this is JUST for echo

Other than that, I'm waiting for opinions from Thomas or Tpowa here 8)
Comment by Nicolas Bigaouette (big_gie) - Wednesday, 21 January 2009, 20:51 GMT
Haaaaaaa...... thanx ;)
${pkgname/*-/-} will match everything at the left of the _last_ "-", meaning if one have "kernel26-foo-bar", you'll end up with "-bar" only...
Comment by Aaron Griffin (phrakture) - Wednesday, 21 January 2009, 21:16 GMT
ick! this works:
$ expr kernel-foo-bar : '[^-]*\(-.*\)'

I imagine a echo and sed might be a tad clearer to read...
$ echo kernel-foo-bar | sed 's/[^-]*\(-.*\)/\1/'

same regex in both cases
Comment by Nicolas Bigaouette (big_gie) - Wednesday, 21 January 2009, 21:25 GMT
Ok ;) But is it really simpler?
_kernelname=${pkgname#kernel26}
or
_kernelname=`echo ${pkgname} | sed 's/[^-]*\(-.*\)/\1/'`
?
Comment by Aaron Griffin (phrakture) - Wednesday, 21 January 2009, 21:29 GMT
Oh, yours is better with just removing the "kernel26" part. Good idea
Comment by Nicolas Bigaouette (big_gie) - Thursday, 05 February 2009, 04:07 GMT
Sync with $Id: PKGBUILD 25277 2009-01-25 10:01:21Z tpowa $
2.6.28.2-1
Comment by Nicolas Bigaouette (big_gie) - Thursday, 05 February 2009, 04:12 GMT
Any comment from Thomas or Tpowa?
Comment by Aaron Griffin (phrakture) - Thursday, 05 February 2009, 17:43 GMT
I think these changes would be nice to add, but I'd like to hear opinions from others. Tagging some people for comments
Comment by Nicolas Bigaouette (big_gie) - Thursday, 05 February 2009, 17:46 GMT
Thanx ;)
Comment by Petr Mrázek (Peterix) - Friday, 06 February 2009, 00:13 GMT
I've already used this to build a custom 2.6.29-rc3 kernel and It's a great improvement over the stock PKGBUILD. There's still room for improvement though :)
I changed this part:
# load configuration
yes "" | make config
# build!
####################
# stop here
# this is useful to configure the kernel
#msg "Stopping build"
#return 1
####################
To:
####################
# stop here, this is useful to configure the kernel
msg "Stopping build, configure kernel in separate terminal and press key to continue"
read stuff
####################
This allows me to run make $favorite_.config_tweaker in a second terminal and continue building when it's ready.

One great thing would be to have a simple switch for disabling mkinitcpio completely somewhere in the PKGBUILD header (for static kernels). And ability to easily use a custom mkinitcpio${pkgname#kernel26}.conf along with the preset :)
Comment by Aaron Griffin (phrakture) - Friday, 06 February 2009, 00:17 GMT
What's wrong with replacing the "yes|make config" part with a commented "make menuconfig"?
It makes more sense to do something like that in the PKGBUILD instead of stopping like that
Comment by Petr Mrázek (Peterix) - Friday, 06 February 2009, 00:32 GMT
Last time I did this I ended up having xconfig, menuconfig and kate open with the same .config file to get what I wanted faster... not a single one of those tools could be labeled 'the best'. Obviously YMMV.
Comment by Nicolas Bigaouette (big_gie) - Friday, 06 February 2009, 00:54 GMT
What I do is simply replace the config file with my own. It wont pass md5sum, but its not an issue since the goal is to have something custom...

So first I let the extraction go, then when compilation starts I cancel that. I go in the source/linux-... dir, run make menuconfig myself, configure it the way I want, and then copy the .config back so it overwrite the origianl config (the one side-by-side with the PKGBUILD).

One I have a config which I like, I just keep it and use that one.

I even move the original config to config.arch, and put mine as config.mine, and create a symbolic link to the one I want to use. But then this is my workflow and does not influence the main topic... ;)
Comment by Aaron Griffin (phrakture) - Friday, 06 February 2009, 17:19 GMT
No no, what I meant was this:

----------
# Configure the kernel. Replace the line below with one of your choice.
# make menuconfig # CLI menu for configuration
# make xconfig # X-based configuration
# ... or manually edit .config
yes "" | make config
----------
Comment by Nicolas Bigaouette (big_gie) - Wednesday, 11 February 2009, 16:12 GMT
To correct the firmware issue, I created another PKGBUILD which I'm attaching here. It is basically the kernel26 original package but stiped down. I kept the -ARCH patch set.

Because the firmwares depends on the modules being compiled or not (I think), all modules should be compiled to create the firmwares. This is why I "make allmodconfig" without any config file.

This is a proof of concept. A kernel package could depend on this one.

Also note that I kept the "if source is already patched, skip the patching" test to speed up compilation. It could be removed.
   PKGBUILD (1.3 KiB)
Comment by Tobias Powalowski (tpowa) - Friday, 13 February 2009, 09:28 GMT
Thomas shall we adress this for .29 kernel series?
Comment by Nicolas Bigaouette (big_gie) - Monday, 23 February 2009, 18:45 GMT
Anyone?
Comment by (MajorTom) - Thursday, 12 March 2009, 06:16 GMT
Well, it looks like 2.6.28.7-2 finally makes it easy to build a custom kernel package.
Thank you Nicolas for initiating this change and thanks to the whole team for making this happen.

Loading...