FS#23407 - Allow soft failures on Server URLs

Attached to Project: Pacman
Opened by Dan McGee (toofishes) - Wednesday, 23 March 2011, 18:33 GMT
Last edited by Allan McRae (Allan) - Sunday, 29 June 2014, 07:27 GMT
Task Type Bug Report
Category Backend/Core
Status Assigned
Assigned To Dave Reisner (falconindy)
Architecture All
Severity Low
Priority Normal
Reported Version 3.5.1
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 11
Private No

Details

It would be awesome to have something like this; I've thought it but haven't done it yet and now I'm seeing a forum post (https://bbs.archlinux.org/viewtopic.php?id=115582) as well about this.

Currently:
$ pacSyu
:: Synchronizing package databases...
testing 45.8K 247.5K/s 00:00:00 [##################################################] 100%
error: failed retrieving file 'core.db' from localhost:8954 : HTTP response code said error
core is up to date
extra is up to date
community-testing is up to date
multilib is up to date
community is up to date

Sidenote: It should have errored on extra/multilib/community as well, maybe those just got covered up somehow? and the error stinks, it returned a 404, we can tell the user that. Also, when I wasn't even running my localhost:8954 listening script, I got no errors at all, so not sure what is up with the new curl code.

Now that I've been totally sidetracked, Here is the proposal, with plenty of leeway for a different name:

[repo]
CacheServer = http://localhost:8954/$repo/
Server = http://example.com/$repo/

Where we would treat CacheServer entries as different than Server entries- if they returned a 404 or weren't available, we would simply ignore the failure and move on to the next URL in the list. Additionally, I think it would make sense to skip CacheServer when downloading repository DBs or repository signatures.
This task depends upon

Comment by Dan McGee (toofishes) - Wednesday, 23 March 2011, 18:35 GMT
My working branch currently contains some work that was directed toward this eventual goal- cleaning up the add server API a bit so we could denote the difference between these server types.
Comment by Dan McGee (toofishes) - Friday, 25 March 2011, 21:15 GMT
why isn't this dun yet
Comment by Xilon (Xilon) - Sunday, 10 April 2011, 11:20 GMT
> we would simply ignore the failure and move on to the next URL in the list

I thought something similar was already done for the Server entries. I guess it only tries subsequent Server entries if there's a transient (timeout) error? Any reason why Server doesn't behave that way?

I don't really see the benefit of distinguishing these (at least not compared to the added complexity). The only difference I can think of is that there's an implicit, dynamic, limit of retries (i.e. the amount of CacheServer entries). With Server it would try each server in the list, possibly up to some maximum.
Comment by Dan McGee (toofishes) - Sunday, 10 April 2011, 17:27 GMT
I should have bee more explicit about the "silent" part. Add a non-existent URL to your mirrorlist and notice that although it will still work, we won't exactly be quiet about it.
Comment by Allan McRae (Allan) - Sunday, 29 June 2014, 07:27 GMT
  • Field changed: Due in Version (4.2.0 → Undecided)
Removing 4.2 target as not much has been done about this currently
Comment by flacs (Tilka) - Tuesday, 13 April 2021, 17:56 GMT
Pinging this 10 year old task to see if anyone is interested in implementing it^^
Comment by Christian Hesse (eworm) - Monday, 07 June 2021, 15:11 GMT
With pacman 6.0.0 we have a new feature "server error limit". This is a good thing in general, but breaks where download errors are expected - for example with caching servers.
Thus I would like to see this implemented asap...

Looking at the code servers are stored in a list... So no way to add an extra attribute/flag whatever to mark the server as cache server.
To those familiar with the code: How to store this information? Or do we add a new list "alpm_list_t *cacheservers" in "struct __alpm_db_t"? How to handle the order then?
Comment by Christian Hesse (eworm) - Monday, 24 January 2022, 19:28 GMT
(Re-)Started the discussion here...
https://lists.archlinux.org/pipermail/pacman-dev/2022-January/025492.html

... and sent my current implementation here...
https://lists.archlinux.org/pipermail/pacman-dev/2022-January/025505.html

Anybody wants to give it a try? Add this in your pacman.conf:

[cache-servers]
Server = http://pkgbuild.com/~eworm/$repo/
Comment by Mayeul C. (MayeulC) - Thursday, 03 November 2022, 11:04 GMT
This would really be appreciated. One of the locations I often visit really has a spotty internet connection (our DSL link went from 8M to often broken and <1M when it's up), and running `pacman -Syu` on a single computer can take... days, sometimes (at 50 KB/s).
Visiting with a freshly upgraded laptop and pacredir enabled used to speed this process up tremendously, as there are multiple computers running Arch on the LAN.

At some point, I think I'll also experiment with torrent or ipfs-based caching, which this would make easier.

I would therefore like to see the patch merged, I can help to test if necessary.

Loading...