FS#65197 - Increase database size limit
Attached to Project:
Pacman
Opened by Matt McDonald (gardotd426) - Saturday, 18 January 2020, 19:46 GMT
Last edited by Andrew Gregory (andrewgregory) - Thursday, 02 July 2020, 20:06 GMT
Opened by Matt McDonald (gardotd426) - Saturday, 18 January 2020, 19:46 GMT
Last edited by Andrew Gregory (andrewgregory) - Thursday, 02 July 2020, 20:06 GMT
|
Details
Description: After adding the chaotic-aur repository to
/etc/pacman.conf, and receiving and signing the keys, pacman
-Fy refuses to get the file list:
sudo pacman -Fy --verbose --verbose [sudo] password for matt: Root : / Conf File : /etc/pacman.conf DB Path : /var/lib/pacman/ Cache Dirs: /var/cache/pacman/pkg/ Hook Dirs : /usr/share/libalpm/hooks/ /etc/pacman.d/hooks/ Lock File : /var/lib/pacman/db.lck Log File : /var/log/pacman.log GPG Dir : /etc/pacman.d/gnupg/ Targets : None :: Synchronizing package databases... core is up to date extra is up to date community is up to date multilib is up to date valveaur is up to date error: failed retrieving file 'chaotic-aur.files' from lonewolf-builder.duckdns.org : Maximum file size exceeded error: failed retrieving file 'chaotic-aur.files' from chaotic.bangl.de : Maximum file size exceeded error: failed retrieving file 'chaotic-aur.files' from chaotic.bangl.de : Maximum file size exceeded error: failed retrieving file 'chaotic-aur.files' from repo.kitsuna.net : Maximum file size exceeded error: failed to update chaotic-aur (download library error) error: failed to synchronize all databases I opened an issue on the chaotic-aur github page, and pedrohlc, the maintainer, thought it might be because he needed rebuild the repo, but after doing that he discovered that he ran into the same issue, and suggested that this must be a problem with Arch and that we need to file a bug report. I have a Manjaro installation on this same PC, and the repo works just fine in Manjaro with zero issues. After he rebuilt the repo, I tried again just in case, and the issue persists. If I download chaotic-aur.files myself and place it in /var/lib/pacman/sync, then it all works fine. Since the actual process of adding the repo to pacman.conf and receiving and signing the keys (as instructed on the arch wiki) works fine in Manjaro, this is upstream from that, and since Arch is responsible for pacman bugs and there's nothing upstream of Arch for pacman, this bug would fall under Arch's responsibility as per the Arch wiki. I'm more than happy to provide any needed info from either my Arch or Manjaro (in case you need to compare for some reason) installations. From what I read on the how to file a bug report article on the wiki, I figured this should be categorized as a Medium-level bug (non-essential broken function), however I apologize if I categorized it wrong. Additional info: * package version(s) 5.2.1-4 * config and/or log files etc. will attach /etc/pacman.conf Steps to reproduce: Add the following to /etc/pacman.conf: [chaotic-aur] Server = http://lonewolf-builder.duckdns.org/$repo/x86_64 Server = http://chaotic.bangl.de/chaotic-aur/x86_64 Server = http://chaotic.bangl.de/$repo/x86_64 Server = https://repo.kitsuna.net/x86_64 Run the following: sudo pacman-key --keyserver keys.mozilla.org -r 3056513887B78AEB sudo pacman-key --lsign-key 3056513887B78AEB Run: sudo pacman -Syy After which, run: sudo pacman -Fy ..to get the package database. At that point, it will fail with the errors listed above. If, however, you add chaotic-aur.files to /var/lib/pacman/sync after it fails, and run it again, it will succeed. Here is the link to the github issue page, if anyone needs it. Also, pedrohlc said that arch's own repo-add tool generated the db files in the first place. Anyhow, here's the github, and I'm more than happy to help however I can https://github.com/PedroHLC/chaotic-aur/issues/36 |
This task depends upon
Closed by Andrew Gregory (andrewgregory)
Thursday, 02 July 2020, 20:06 GMT
Reason for closing: Fixed
Additional comments about closing: Commit 2856a7dea3c0d4584e126b5ca5957e13e23f83d1
Thursday, 02 July 2020, 20:06 GMT
Reason for closing: Fixed
Additional comments about closing: Commit 2856a7dea3c0d4584e126b5ca5957e13e23f83d1
https://gitlab.manjaro.org/packages/core/pacman/commit/cc8253a7bd6c230c533db448cc1daad66e96d498
repo... splitting it to reduce its size, changing the database file compression... Maybe check the chat logs of that day.
Having some limit also prevents a rogue repo database download continuing until the (probably root) partition is full.
I think you mean this predates pacman's second coming of the -F flag. Dan and I introduced the limit after we switched from libfetch to libcurl. You can see the rationale at the time in 6dc71926f9b16e. This has aged decently well, aside from the claim that 25MiB is double the size of all the repos including files (currently weighing in at 39MB including staging and testing repos).
I'm not sure where you're getting the idea that DB bloat comes from the addition of signatures. Even with all packages being signed, the difference in repo size between db and files is generally (at least in current Arch repos) about 2-5x larger[0]. I have a hard time believing that a fixed addition of <500 bytes is responsible for exceeding the DB size limit.
Generally, I'm in favor of keeping the limit and this might actually be a case where it makes sense to make the unknown package/db upper limit a pacman.conf knob. Distros should have an understanding of how big their repos are in order to set sane defaults in the distributed config, and users can bump the limit as they see fit (for cases like custom repos). Alternatively, we can extend the logic in be_sync.c to set a payload max_size based on the db extension, but that's probably just going to bite us again O(years) from now.
[0] for f in /var/lib/pacman/sync/*.db; do awk -v d="$(wc -c <"$f")" -v f="$(wc -c <"${f%.db}.files")" 'BEGIN { inc=(f-d)/d; if (inc) printf "%.2f%%\n", inc }'; done
But also regarding another thing Eli mentioned, I spoke with pedrohlc and I tested using zstd myself and when I untar-ed the original chaotic-aur.files database and then re-archived it with zstd, it did show a 40 percent decrease in total size, but pedrohlc said he couldn't figure out how to change the compression level with repo-add, and that's how he's making the database archives. And the 40 percent decrease requires the ZSTD_CLEVEL=19 envvar, I'm not sure if repo-add respects that or not.