FS#23968 - add support for compression/decompression via lrzip to makepkg/pacman

Attached to Project: Pacman
Opened by John (graysky) - Thursday, 28 April 2011, 07:07 GMT
Last edited by Allan McRae (Allan) - Wednesday, 13 February 2013, 04:02 GMT
Task Type Feature Request
Category Backend/Core
Status Closed
Assigned To Allan McRae (Allan)
Architecture All
Severity Low
Priority Normal
Reported Version 3.5.2
Due in Version 4.1.0
Due Date Undecided
Percent Complete 100%
Votes 4
Private No

Details

Summary and Info: See the graphs CK put up on his blog:
http://ck-hack.blogspot.com/2011/04/lrzip-0603.html

For compressing the kernel26 source tree (v2.6.38) on a quad core Intel, lrzip did it approx 3 times faster than tar/gzip and was more or less the same filesize (2 % difference). Why is this value-added/why should you spend your time doing it? With more and more desktop/laptops running multicore processors, the speed benefits of a compression/decompression program that can use them are obvious.
This task depends upon

Closed by  Allan McRae (Allan)
Wednesday, 13 February 2013, 04:02 GMT
Reason for closing:  Implemented
Additional comments about closing:  git commit 572b1a5d
Comment by Allan McRae (Allan) - Sunday, 05 June 2011, 10:00 GMT
Freshmeat page for lrzip: http://freshmeat.net/projects/lrzip
Comment by Kyle Keen (keenerd) - Saturday, 13 August 2011, 19:56 GMT
Here is the easy half - makepkg support. I would rather not think about the other half, linking lrz into Pacman.

edit: There is no .so to link against, pacman would need a dirty dirty system() call to unextract.
Comment by Allan McRae (Allan) - Saturday, 13 August 2011, 21:39 GMT
I was think that was should add extracting .lrz source files to makepkg and that is it. Maybe allowing source packages to be crated using it is also OK... To make actual packages with that compression really requires support to be added to libarchive to be of any use.
Comment by John (graysky) - Thursday, 08 March 2012, 20:30 GMT
Allan - CK has just released v0.610 wherein we now have an liblrzip library!

"The major feature addition in this latest version is largely thanks to the work of Michael Blumenkrantz in the form of a liblrzip library! This means you can now link in lrzip compression and decompression into any other application. There is a range of APIs available to use this capability from the simplest gzip equivalent lrzip_compress and lrzip_decompress functions to ones that expose all the low level features knobs and dials in lrzip. There are also a couple of demo examples of using these functions included in the source code, but in their simplest form they should be very easy to use. I look forward to seeing package managers, archivers and maybe even git and so on having lrzip support in the future ;) "

https://aur.archlinux.org/packages.php?ID=7418
Comment by Dan McGee (toofishes) - Thursday, 08 March 2012, 20:40 GMT
GPL is a bummer, that is going to be a very very hard sell to the libarchive guys.

http://code.google.com/p/libarchive/issues/list is where you will need to go if you want us to think about supporting this.
Comment by John (graysky) - Thursday, 08 March 2012, 21:03 GMT
@Dan - what is the preferred licence for inclusion in libarchive?
Comment by Karol Błażewicz (karol) - Thursday, 08 March 2012, 21:05 GMT Comment by John (graysky) - Friday, 09 March 2012, 20:21 GMT Comment by John (graysky) - Sunday, 11 March 2012, 22:54 GMT
Anyone following the suggestion in the upstream report? Possible for pacman to use lrzip externally rather than in libarchive?
Comment by Allan McRae (Allan) - Sunday, 11 March 2012, 23:03 GMT
Implementing compression directly using lrzip in makepkg is the easy part. But there is not point until decompression is implemented in libarchive. I highly doubt we will ever call an external decompression program in pacman.
Comment by John (graysky) - Monday, 12 March 2012, 19:53 GMT
EDIT: I put together a few benchmarks comparing compression metrics of xz to lrzip as it relates to compressing the linux-3.2.11 source tree. Using default settings (-L 7) the data illustrate my original, "3x faster and only 2 % larger than xz." Also included are the compression metrics using the zpaq compression mode (-z) while give a compression time 2x faster and 19 % smaller than xz. Really bright future for Arch packages if folks want to switch from xz to lrzip for them :)

http://repo-ck.com/bench/lrzip.pdf
Comment by John (graysky) - Sunday, 18 March 2012, 10:43 GMT
Pretty great news on getting lrzip's libs into libarchive. I contacted CK who worked with Michael Blumenkrantz (author of liblrzip) to do this. The full libraries that are GPL licensed have been accepted which means that we can work on full lrzip support with libarchive.

https://github.com/libarchive/libarchive/pull/7
Comment by Corey Richardson (cmr) - Wednesday, 06 February 2013, 03:57 GMT
https://github.com/libarchive/libarchive/pull/28

I think that finishes it up? Not sure.
Comment by Allan McRae (Allan) - Wednesday, 06 February 2013, 04:02 GMT
I'm guessing libarchive-3.1 might be in a usable state by pacman-4.1...

To implement this we need the following patches for makepkg:
1) create_package() -> add lrzip to compression schemes
2) extract_sources() -> do we want to add a lrzip source type (non-tar)
3) create_srcpackage() -> add lrzip - does bsdtar have a flag to compress with lrzip?

pacman will handle lrzip packages automatically when linked against libarchive-4.1
Comment by John (graysky) - Saturday, 09 February 2013, 12:07 GMT
Thanks Allan. Don't forget the patch in https://bugs.archlinux.org/task/31782 to enable makepkg compression.
Comment by Allan McRae (Allan) - Saturday, 09 February 2013, 12:46 GMT
That patch is quite outdated...
Comment by John (graysky) - Saturday, 09 February 2013, 14:30 GMT
Ack, I'll leave developing to the developers on this one. Just trying to be helpful.

Loading...