FS#59081 - Add support zstd to makepkg/pacman

Attached to Project: Pacman
Opened by whoami (atommixz) - Wednesday, 20 June 2018, 07:53 GMT
Last edited by Eli Schwartz (eschwartz) - Friday, 11 October 2019, 19:41 GMT
Task Type Feature Request
Category makepkg
Status Closed
Assigned To Eli Schwartz (eschwartz)
Architecture All
Severity Low
Priority Normal
Reported Version 5.1.0
Due in Version 5.2.0
Due Date Undecided
Percent Complete 100%
Votes 4
Private No

Details

Description:
757M pycharm-professional-2018.1.4-1-x86_64.pkg.tar
338M pycharm-professional-2018.1.4-1-x86_64.pkg.tar.zst
309M pycharm-professional-2018.1.4-1-x86_64.pkg.tar.xz

# compress
$ time zstd -T0 pycharm-professional-2018.1.4-1-x86_64.pkg.tar
real 0m2,941s
$ time xz -T0 -1 pycharm-professional-2018.1.4-1-x86_64.pkg.tar
real 0m31,260s

# decompress
$ time zstd -d pycharm-professional-2018.1.4-1-x86_64.pkg.tar.zst
real 0m1,869s
$ time xz -d pycharm-professional-2018.1.4-1-x86_64.pkg.tar.xz
real 0m18,694s


Additional info:
Wants COMPRESSZST=(zstd -q -c -z -T0 -)


Steps to reproduce:
This task depends upon

Closed by  Eli Schwartz (eschwartz)
Friday, 11 October 2019, 19:41 GMT
Reason for closing:  Implemented
Additional comments about closing:  https://git.archlinux.org/pacman.git/com mit/?id=bae74c8e9e69b4f4e1a235eef21b9b27 fb14aff0
Comment by whoami (atommixz) - Wednesday, 20 June 2018, 08:09 GMT
$ time xz -T0 pycharm-professional-2018.1.4-1-x86_64.pkg.tar
real 1m29,134s

258M pycharm-professional-2018.1.4-1-x86_64.pkg.tar.xz
Comment by Allan McRae (Allan) - Wednesday, 20 June 2018, 10:09 GMT
Support for zstd is not in a released version of libarchive yet.
Comment by Eli Schwartz (eschwartz) - Wednesday, 20 June 2018, 17:24 GMT
https://lists.archlinux.org/pipermail/pacman-dev/2017-October/022186.html

As Allan said, we're still waiting on libarchive to release a version with zst support, at which point pacman will automatically be able to use packages created with zst compression.

At that point, we will merge my patch to add zstd support in makepkg itself, but until then we're not going to provide makepkg support that breaks pacman.
Comment by Adam Fontenot (amfontenot) - Wednesday, 05 September 2018, 02:59 GMT
I'm commenting here so that those subscribed to this bug get pinged. libarchive 3.3.3 is (finally!) released, so this can move forward.

After some time for this to mature, I would love to see a discussion about moving the distribution to zstd for packages. For many of us, the time to decompress packages is the majority of the time it takes to upgrade, and zstd provides the best download + decompression time total for a very wide range of bandwidths. For example zstd -19 is within the margin of error or wins between 2.5 and 55 MiB/s on the Mozilla dataset here: https://quixdb.github.io/squash-benchmark/unstable/#transfer-plus-processing
Comment by Eli Schwartz (eschwartz) - Thursday, 20 September 2018, 06:07 GMT
This has now landed in https://git.archlinux.org/pacman.git/commit/?id=bae74c8e9e69b4f4e1a235eef21b9b27fb14aff0
repo-add is likewise covered by https://git.archlinux.org/pacman.git/commit/?id=7afe51171fe063bf3031cc68fc8c7ac914a01de2

From pacman's side of things, this is done. If anyone wants to (in future, whenever as Adam says it has matured a bit?) discuss the best decompression scheme for packages on a repository level there are other places to discuss this, and indeed it's likely to be quite the discussion.
It's probably best handled via the arch-general mailing list.

Loading...