FS#25882 - Use a parallel compression for packages.

Attached to Project: Arch Linux
Opened by Mark (markg85) - Saturday, 03 September 2011, 19:43 GMT
Last edited by Dan McGee (toofishes) - Thursday, 08 September 2011, 18:30 GMT
Task Type Feature Request
Category Arch Projects
Status Closed
Assigned To No-one
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Hi,

I've recompiled Qt a few times on a 6 core machine and it kinda pains me to see that the compilation process is using all cores just fine, but making the .xz package is only using 1 core at 100%. Compressing Qt is no easy task and i even bet it right now takes even longer to compress Qt then to build it!

Anyway, please consider implementing a parallel xz compression (p7zip can do it) or use for example pbzip2. Or anthing else i don't know.

Just my opinion.

Kind regards,
Mark
This task depends upon

Closed by  Dan McGee (toofishes)
Thursday, 08 September 2011, 18:30 GMT
Reason for closing:  Deferred
Additional comments about closing:  Waiting for upstream programs to implement multithreading.
Comment by Mark (markg85) - Saturday, 03 September 2011, 19:56 GMT
Actually make all of pacman and makepkg parallel please. I tried makepkg with just replacing bzip2 with pbzip2 and bunzip2 with pbunzip2 and that made makepkg a lot faster for me! Just minutes/seconds instead of 15 minutes (for Qt!). However, pacman itself is also doing something that just occupies 1 core at 100%. That should be multicore as well if possible.
Comment by Ionut Biru (wonder) - Saturday, 03 September 2011, 20:05 GMT
what's up with this bug? xz doesn't support parallel compression and once it will, makepkg will inherit as well
Comment by Pierre Schmitz (Pierre) - Saturday, 03 September 2011, 20:34 GMT
Xz 5.1 will have multi threading support. Bzip2 is quite an inefficient format in respect to performance/compression ratio. I don't really see the need to work on any intermediate solution. We'll just wait until these features are provided by upstream.

If you compile packages for you own use you might want to set pkgext to e.g. pkg.tar.gz which is a lot faster. You can even override this within the PKGBUILD.
Comment by Mark (markg85) - Saturday, 03 September 2011, 20:41 GMT
Ohh nice! So i guess this just someday becomes parallel (xz) without doing anything besides updating the xz package?
Comment by Karol Błażewicz (karol) - Saturday, 03 September 2011, 21:09 GMT
See also  FS#23968 .

Loading...