Arch Linux

i vote for that change gz -> bz2.

especially for the x86_64 port: our binaries are somewhat bigger than on i686 and cpu power is not a real problem. but bandwidth still is.

This is a much larger issue than a simple change, from gz->bz2. This one requires lots of discussion and things like that. It has been discussed a lot before. The basic issue is that there is a trade-off. Binaries won't compress all that much more, percentage-wise, and you're increasing cpu load and time while decreasing bandwidth and disk space. It's the typical "speed/space" tradeoff that every computer science student discusses in some class.

The fact is, it's been decided and there are pros and cons for each side.

I will leave this open, in order to reference at a later date.

I leave you with one interesting tid-bit:
Try out a bz2 database with pacman3. I _think_ it should work out of the box, assuming you change the DB_EXT definition to "db.bz2"

bz2 should be default. arch is i686 so cpu doesn't matter

Saying CPU doesn't matter due to the i686 requirement doesn't make sense. Even on a dual core machine like I am sitting at now, extracting a 1MB bz2 file takes a good 2 seconds longer than the same data gzipped. That is highly significant.

Again. This matter has come up 4 million times. It is over and done with. Arch will use gzip. Sorry.

dude you must be kidding me. 2 secs? how much faster did you get the smaller package (let's say it is an Xorg package?). How much less is the bw that is the cost of running archlinux.org?

I don't care for neither gzip or bzip2, but if you can't see why it will benefit, then I can't force you of course..

Of course bzip2 is slower when compressing/decompressing.
But using bzip2 will result in short _total_ installing time, if only you don't install from cache or local mirror. That's because _downloading_ smaller files will take less time. And for most of us the difference will be much more than seconds.
This is my point.
Anyway I cannot insist on something. It depends on your choice, Aaron.

Of course delta sync downloads would be even better, but they are harder to implement and maintain.

About decompression speed again:
I suggest to add 'NOCOMPRESS" option to makepkg and tar-only packages support to pacman.
This will be useful for large packages that contain mostly hardly compressable data, like large binary games with already compressed gamedata files into "packs".

i'm still voting to switch to bzip2 compression just for a faster download time. at least for cpu-powerfull ports like x86_64. how about making use of DualCore power some of our cpus now have.

http://compression.ca/pbzip2/ - the project has reached 1.0 state. maybe we could even use it as a replacement for bzip2 in current/base after some testing.

I'm going to say the same thing I did with the xdelta guys. I don't really consider this all that important. If you do, however, you are welcome to provide a patch and I will see what I can do about integrating it. As it stands, using gzip works, and has been working for some time, so there's not much reason to change it. I have many other features/fixes I want to work on before something like this.

(Side note: please do not 'replace' bzip2 with something like the above. If anything, provide it as an option in extra or something using the standard provides/replaces way of doing things. A full replacement is probably not a good idea)

I would be another vote against this change- given the speed of my connection the unzipping portion takes much longer than the download itself.

Instead of adding to the flames, I'll present some real numbers. This is on an exceptionally large file, the kernel source tarball, so repeating this experiment on something smaller (say our kernel binary package) might be more useful. Realize that you only need to calculate the download time and speed for either tar.gz or bz2; you can use math to figure out the other one given the sizes of the two packages.

File: linux-2.6.21.1.*
tar: 243 MB
bz2: 42 MB
gz: 54 MB

bz2:
download: 9.4 sec
unzip: 38.6 sec
zip: 136.95 sec (wow!)

gz:
download: 11.7 sec
unzip: 4.8 sec
zip: 24.4 sec

Notice how much faster the gzip is for me- it takes 1/3 of the time of the bzip download/unzip process. The best thing I can produce here is another number- the speed of the connection, given my machines unzip times, where the two would break even. This test was done with a 4.5 MB/sec download rate, which is pretty fast. Doing some linear equations, I can tell you that my connection would have be be slower than 327 KB/sec to prefer bzip. That is an awfully slow connection for those with broadband, isn't it?

(If you want the answer I found above, find the intersection of these two equations:
y1 = 38.6 + (42984 / x)
y2 = 4.8 + (54029 / x)
The numbers should be fairly obvious where they come from, they are times and sizes from above.)

Well, libarchive can handle .tar.gz and .tar.bz2 transparently; so this is your choice which compression do you prefer.
So pacman -U foo.tar.bz2 will also work, if .PKGINFO and such things exist in the archive. And in sync repos we use the %FILENAME% field to determine filename, so this is quite flexible.
The same true for repo uncompressing (in ArchLinux we use db.tar.gz DBEXT which could be a bit misleading in case of compression change ;-), simple .db would be transparent).

So if you implement compression type option to makepkg, this task can be closed, since this was a distro specific discussion :-P