Arch Linux

Please read this before reporting a bug:
https://wiki.archlinux.org/title/Bug_reporting_guidelines

Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!
Tasklist

FS#39529 - [pacman] parallel xz compression / change default compressor for package generation to pixz

Attached to Project: Arch Linux
Opened by Olaf Leidinger (leidola) - Tuesday, 18 March 2014, 17:06 GMT
Last edited by Allan McRae (Allan) - Friday, 12 June 2015, 03:40 GMT
Task Type Feature Request
Category Packages: Core
Status Closed
Assigned To Dan McGee (toofishes)
Allan McRae (Allan)
Dave Reisner (falconindy)
Architecture All
Severity Very Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 3
Private No

Details

Description:

For binary packages from aur, the most time consuming task for me is compressing the package via XZ. I recently checked for a parallel version in the same manner as pbzip2 and I found pixz. As for pbzip2, the generated packages are compatible with the single-core implementations.

I can generate xz packages via pixz by setting

COMPRESSXZ=(pixz )

in makepkg.conf. I think this would probably benefit the build-servers, too.

pixz is currently in AUR only, the code is available via:

https://github.com/vasi/pixz


This task depends upon

Closed by  Allan McRae (Allan)
Friday, 12 June 2015, 03:40 GMT
Reason for closing:  Won't implement
Comment by Dave Reisner (falconindy) - Tuesday, 18 March 2014, 17:24 GMT
from xz(1):

-T threads, --threads=threads
Specify the number of worker threads to use. The actual number of threads can be less than threads if using more threads would exceed the memory usage limit.

Feel free to do as you wish with your config. I don't believe there's any justification here to set a multi-threaded default and make assumptions about the machines that build packages.
Comment by Olaf Leidinger (leidola) - Tuesday, 18 March 2014, 17:37 GMT
The manpage says further on:

"Multithreaded compression and decompression are not implemented yet, so this option has no effect for now."

Comment by Allan McRae (Allan) - Tuesday, 18 March 2014, 22:26 GMT
Yes - xz requires the upstream developmental release.

Anyway, this has been discussed by the developers previously and there is no interesting in doing anything beyond waiting for a new xz release. Feel free to edit your config files how you want...
Comment by UnicornDarkness (Xorg) - Thursday, 05 February 2015, 06:26 GMT
  • Field changed: Percent Complete (100% → 0%)
Since xz 5.2.0, parallel xz compression is implemented (see changelog [1]).

We can enable it in makepkg.conf like this:
COMPRESSXZ=(xz -T0 -c -z -)
Tested, it works.


[1] http://git.tukaani.org/?p=xz.git;a=blob;f=NEWS;hb=HEAD
Comment by Andreas Radke (AndyRTR) - Thursday, 05 February 2015, 06:28 GMT
Now that we have xz 5.2.0 we should discuss this further.

From discussion in IRC I remember that creating delta packages holds back this change in default behavior.
Maybe this can be fixed on the side where the delta pkg creator reads its config.
Comment by Allan McRae (Allan) - Thursday, 05 February 2015, 06:54 GMT
Delta packages work in weird ways... xdetla essentially uncompresses the two tarballs and creates the delta. To reconstruct a package from a delta, you need to uncompress the old package, apply the delta and re-compress the package. This means that the compression needs to be repeatable on different machines for the package signature to validate. With xz, that can only happen when compressing without threads.

Loading...