Arch Linux

His patches contain no explanations/commit messages. Already 0001-p1.patch will break entirely though.

Please read the page first : http://students.mimuw.edu.pl/~kk219459/mkinitcpio/
patch 1 and 2 accomplish this :
# When appending files/dirs/... to the result list, don't grep to check if they are already there. At the end sort and remove the duplicates.

It would also be nice if you could give a detailed explanation of why it breaks, and why it still worked for several users who tested these patches.
And if it is truly broken, I would like to know if you see any other ways to kill these bottlenecks.

The best would be to contact directly the author of these patches about the potential problems, but I can forward the information if needed.

The files are extracted in the order they are added in the file list. Directories must exist so that we can extract files for example. And files added later overwrite the ones added earlier. Coincidentally, "file" is lexicographically bigger than "dir", however, "dev" is smaller than "dir" and thus alone should lead to breakage.

I am not relying on coincidences, I once issued a "sort" on the file list myself to work around a problem, and it all stopped working entirely. The order the files are added in does matter.

I see, it's a bit magic, but it does work.

$ sort -u filelist
dir /bin 755 0 0
dir /dev 755 0 0
dir /etc 755 0 0
dir /etc/modprobe.d 755 0 0
dir /etc/udev 755 0 0
dir /hooks 755 0 0
dir /lib 755 0 0
dir /lib/udev 755 0 0
dir /lib/udev/rules.d 755 0 0
dir /proc 755 0 0
dir /sbin 755 0 0
dir /sys 755 0 0
dir /usr 755 0 0
dir /usr/bin 755 0 0
dir /usr/lib 755 0 0
dir /usr/sbin 755 0 0
file /bin/busybox /lib/initcpio/busybox 755 0 0
file /config /etc/mkinitcpio.conf 644 0 0
file /etc/modprobe.d/usb-load-ehci-first.conf /etc/modprobe.d/usb-load-ehci-first.conf 644 0 0
file /etc/udev/udev.conf /etc/udev/udev.conf 644 0 0
file /hooks/udev /lib/initcpio/hooks/udev 644 0 0
file /init_functions /lib/initcpio/init_functions 755 0 0
file /init /lib/initcpio/init 755 0 0
file /lib/ld-2.11.1.so /lib/ld-2.11.1.so 755 0 0
file /lib/libblkid.so.1.1.0 /lib/libblkid.so.1.1.0 755 0 0
file /lib/libc-2.11.1.so /lib/libc-2.11.1.so 755 0 0
file /lib/libuuid.so.1.3.0 /lib/libuuid.so.1.3.0 755 0 0
file /lib/udev/firmware /lib/udev/firmware 755 0 0
file /lib/udev/load-modules.sh /lib/initcpio/udev/load-modules.sh 755 0 0
file /lib/udev/rules.d/50-firmware.rules /lib/udev/rules.d/50-firmware.rules 644 0 0
file /lib/udev/rules.d/50-udev-default.rules /lib/udev/rules.d/50-udev-default.rules 644 0 0
file /lib/udev/rules.d/60-persistent-storage.rules /lib/udev/rules.d/60-persistent-storage.rules 644 0 0
file /lib/udev/rules.d/80-drivers.rules /lib/udev/rules.d/80-drivers.rules 644 0 0
file /sbin/blkid /sbin/blkid 755 0 0
file /sbin/modprobe /sbin/modprobe 755 0 0
file /sbin/udevadm /sbin/udevadm 755 0 0
file /sbin/udevd /sbin/udevd 755 0 0
nod /dev/console 644 0 0 c 5 1
nod /dev/mem 644 0 0 c 1 1
nod /dev/null 644 0 0 c 1 3
nod /dev/zero 644 0 0 c 1 5
slink /lib/ld-linux.so.2 /lib/ld-2.11.1.so 777 0 0
slink /lib/libblkid.so.1 /lib/libblkid.so.1.1.0 777 0 0
slink /lib/libc.so.6 /lib/libc-2.11.1.so 777 0 0
slink /lib/libuuid.so.1 /lib/libuuid.so.1.3.0 777 0 0

I'm confused. I am fairly certain I tried it in the past and it failed hilariously. But there's another problem:

Our add_* functions do explicit checks only for the filename inside the image, not the whole line. So:

file /foo /bar1 644 0 0
file /foo /bar2 644 0 0

can never happen, as add_file greps for "^file /foo" only. With the "don't grep and sort later" method, two files will be added, only one if which will be visible after extraction.

A better method would be to not store all this in a file, but rather in memory and write it to a file just to generate the image.

Here is Krzysztof answer, which also looks correct to me :

First problem gets solved by -k 2,2, second has different solutions:
1. non-deterministic version, fastest:
sort -k 2,2 -u ${FILELIST}.tmp > ${FILELIST}
2. first file wins, same results as with grepping:
sort -s -k 2,2 -u ${FILELIST}.tmp > ${FILELIST}
3. last file wins, probably most reasonable:
tac ${FILELIST}.tmp | sort -s -k 2,2 -u > ${FILELIST}

Since this is still open, I figured I'd comment. The real win here is to not use gen_init_cpio at all. Create a tree of files in a temp directory and pipe to cpio, i.e.

cd $buildroot
find . -print0 | cpio -0oH newc | $compress > image.out

It's a whole lot cheaper to do a shell test for file/dir existance than it is to grep the file, or hope that sort doesn't munge your input list. Order no longer matters because find pipes the proper hierarchy to cpio.

There are two gotchas to this method:
1) You're using disk space in /tmp, which isn't much but still needs to be mentioned. Shouldn't be more than 5-6mb for a full (non-autodetect) image.
2) You lose the ability to call add_device as an unprivileged user. The only installer to currently use add_device is base. However, /init does not need these devices prior to mounting /dev. In the case of udev being mounted on tmpfs /dev, /init can (and does) create these nodes.

So, I went and did this. Patches have already been submitted for review on arch-projects, and some other efficiencies have already been merged.

https://github.com/falconindy/mkinitcpio/commits/working

short version: great success.

mkinitcpio 0.6.14-1 is still slower compared to geninit.

sudo mkinitcpio -p kernel26 23.04s user 8.34s system 69% cpu 45.366 total
sudo geninit -p kernel26 13.97s user 3.72s system 69% cpu 25.462 total

//offtopic: geninit has nice coloured output.

And? Perhaps you should look at my working branch of mkinitcpio.

Yes. falconindy/working branch (pkg created from PKGBUILD within the repo)

sudo mkinitcpio -p kernel26 14.74s user 2.45s system 69% cpu 24.654 total

and with coloured output too.

EDIT: mkinitcpio being faster than geninit was due to my config file being saved as /etc/mkinitcpio.conf.pacsave instead of writing the new file as /etc/mkinitcpio.conf.pacnew .

The falconindy/working branch and geninit, are they identical?

They're completely separate projects. I'm slowly merging what I learned from writing geninit into mkinitcpio...

For better performance, wait for mkinitcpio 0.7.