FS#23353 : Privoxy with keep-alive-timeout plus Polipo breaks builtin downloader

Historical bug tracker for the Pacman package manager.

The pacman bug tracker has moved to gitlab:
https://gitlab.archlinux.org/pacman/pacman/-/issues

This tracker remains open for interaction with historical bugs during the transition period. Any new bugs reports will be closed without further action.

FS#23353 - Privoxy with keep-alive-timeout plus Polipo breaks builtin downloader

Attached to Project: Pacman
Opened by Taylan Ulrich B. (taylanub) - Saturday, 19 March 2011, 18:25 GMT
Last edited by Dave Reisner (falconindy) - Tuesday, 21 February 2012, 20:37 GMT

Task Type	Bug Report
Category	Backend/Core
Status	Closed
Assigned To	Dave Reisner (falconindy)
Architecture	All
Severity	Medium
Priority	Normal
Reported Version	3.5.0
Due in Version	Undecided
Due Date	Undecided
Percent Complete
Votes	2 Taylan Ulrich B. (taylanub) (2011-03-25) yaser (G_P) (2011-03-25)
Private	No

Details

I have both privoxy and polipo running. Setting privoxy to use keep-alive-timeout and to forward to polipo, then using it as the http_proxy (i.e. pacman->privoxy->polipo->server), the pacman downloader fails somewhat randomly:
In the case of a package file, it always seems to fail the first time and succeed with subsequent trials; when doing a -Syy it seems like the first repo always fails, and the rest is somewhat random.
(Polipo's caching must be related.)

Disabling privoxy's keep-alive-timeout, everything works.

With other proxying combinations (only polipo; only privoxy; polipo->privoxy and not the other way around), everything works.

When using wget or curl (via XferCommand), everything works. (Which is why i report this as a pacman bug; one could suspect privoxy/polipo.)

Transcripts:

<start>
Targets (1): vi-050325-4 [0.12 MB]

Total Download Size: 0.12 MB

Proceed with download? [Y/n]
:: Retrieving packages from core...
error: failed retrieving file 'vi-050325-4-i686.pkg.tar.xz' from repo.parabolagnulinux.org : Broken pipe
error: failed retrieving file 'vi-050325-4-i686.pkg.tar.xz' from ftp.inf.utfsm.cl : Bad Request
warning: failed to retrieve some files from core
error: failed to commit transaction (Bad Request)
Errors occurred, no packages were upgraded.
<end>

(A subsequent download trial of vi will now succeed normally.)

<start>
:: Synchronizing package databases...
error: failed retrieving file 'testing.db' from repo.parabolagnulinux.org : Broken pipe
error: failed retrieving file 'testing.db' from ftp.inf.utfsm.cl : Bad Request
error: failed to update testing (Bad Request)
error: failed retrieving file 'core.db' from repo.parabolagnulinux.org : Broken pipe
error: failed retrieving file 'core.db' from ftp.inf.utfsm.cl : Bad Request
error: failed to update core (Bad Request)
... (all the same)
<end>

Second trial:

<start>
:: Synchronizing package databases...
error: failed retrieving file 'testing.db' from repo.parabolagnulinux.org : Broken pipe
error: failed retrieving file 'testing.db' from ftp.inf.utfsm.cl : Bad Request
error: failed to update testing (Bad Request)
core 51.5K 3.3M/s 00:00:00 [###########################################] 100%
extra 675.2K 21.2M/s 00:00:00 [###########################################] 100%
error: failed retrieving file 'community-testing.db' from repo.parabolagnulinux.org : Broken pipe
error: failed retrieving file 'community-testing.db' from ftp.inf.utfsm.cl : Bad Request
error: failed to update community-testing (Bad Request)
community 604.5K 21.7M/s 00:00:00 [###########################################] 100%
<end>

(The first always fails, the others are random...)

Race conditions are probably involved.

Note that FTP also fails.

This task depends upon

Closed by Dave Reisner (falconindy)
Tuesday, 21 February 2012, 20:37 GMT
Reason for closing: No response

Comments (6)
Related Tasks (0/0)

Comment by Taylan Ulrich B. (taylanub) - Saturday, 19 March 2011, 18:30 GMT

Forgot to say that other servers are the same. It's not a server specific problem. And this is definitely since version 3.5.
(And i should've used --noprogress for those pacman transcripts...)

Comment by Dan McGee (toofishes) - Sunday, 20 March 2011, 17:09 GMT

Well wget and curl as external downloaders are not going to be sending the Keep-Alive: HTTP header- that is my first guess, and we enabled keep-alive connections in pacman 3.5.0 via libfetch. If you can test with another program through this same set of programs but sending the keep alive header that would be useful- perhaps using ab, which has the option to send the header?

Comment by yaser (G_P) - Friday, 25 March 2011, 15:17 GMT

the same

I use cntlm and let http_proxy to use it
this start to happen when i upgrade from pacman 3.4.3 to 3.5.1
When letting http_proxy doesn't use cntlm. the error message,"Broken pipe", will not appear.

# pacman -Syy
:: Synchronizing package databases...
error: failed retrieving file 'core.db' from archlinux.limun.org : Broken pipe
error: failed retrieving file 'core.db' from mirror.c9h.de : Broken pipe
error: failed retrieving file 'core.db' from ftp5.gwdg.de : Broken pipe
error: failed retrieving file 'core.db' from ftp.halifax.rwth-aachen.de : Broken pipe
error: failed retrieving file 'core.db' from ftp.hosteurope.de : Broken pipe
error: failed retrieving file 'core.db' from ftp-stud.hs-esslingen.de : Broken pipe
error: failed retrieving file 'core.db' from mirror.selfnet.de : Broken pipe
error: failed retrieving file 'core.db' from ftp.spline.inf.fu-berlin.de : Broken pipe
error: failed retrieving file 'core.db' from ftp.tu-chemnitz.de : Broken pipe
error: failed retrieving file 'core.db' from ftp.uni-kl.de : Broken pipe
error: failed retrieving file 'core.db' from archlinux.supsec.org : Broken pipe
error: failed retrieving file 'core.db' from cake.lib.fit.edu : Connection timed out
error: failed retrieving file 'core.db' from mirrors.cat.pdx.edu : Service Unavailable
error: failed retrieving file 'core.db' from mirror.ece.vt.edu : Broken pipe
error: failed retrieving file 'core.db' from www.gtlib.gatech.edu : Broken pipe
error: failed retrieving file 'core.db' from mirrors.hosef.org : Broken pipe
error: failed retrieving file 'core.db' from hpc.arc.georgetown.edu : Broken pipe
error: failed retrieving file 'core.db' from distro.ibiblio.org : Broken pipe
error: failed retrieving file 'core.db' from lug.mtu.edu : Broken pipe
error: failed retrieving file 'core.db' from mirrors.xmission.com : Broken pipe
error: failed retrieving file 'core.db' from mirror.mocker.org : Broken pipe
error: failed retrieving file 'core.db' from ftp.osuosl.org : Broken pipe
error: failed retrieving file 'core.db' from mirror.rit.edu : Broken pipe
error: failed retrieving file 'core.db' from mirrors.rutgers.edu : Broken pipe
error: failed retrieving file 'core.db' from mirror.yellowfiber.net : Broken pipe
error: failed to update core (Broken pipe)
error: failed retrieving file 'extra.db' from archlinux.limun.org : Broken pipe
error: failed retrieving file 'extra.db' from mirror.c9h.de : Broken pipe
error: failed retrieving file 'extra.db' from ftp5.gwdg.de : Broken pipe
error: failed retrieving file 'extra.db' from ftp.halifax.rwth-aachen.de : Broken pipe
error: failed retrieving file 'extra.db' from ftp.hosteurope.de : Broken pipe
error: failed retrieving file 'extra.db' from ftp-stud.hs-esslingen.de : Broken pipe
error: failed retrieving file 'extra.db' from mirror.selfnet.de : Connection timed out
error: failed retrieving file 'extra.db' from ftp.spline.inf.fu-berlin.de : Broken pipe
error: failed retrieving file 'extra.db' from ftp.tu-chemnitz.de : Broken pipe
error: failed retrieving file 'extra.db' from ftp.uni-kl.de : Broken pipe
error: failed retrieving file 'extra.db' from archlinux.supsec.org : Broken pipe
error: failed retrieving file 'extra.db' from cake.lib.fit.edu : Connection timed out
error: failed retrieving file 'extra.db' from mirrors.cat.pdx.edu : Service Unavailable
error: failed retrieving file 'extra.db' from mirror.ece.vt.edu : Broken pipe
error: failed retrieving file 'extra.db' from www.gtlib.gatech.edu : Connection timed out
error: failed retrieving file 'extra.db' from mirrors.hosef.org : Connection timed out
error: failed retrieving file 'extra.db' from hpc.arc.georgetown.edu : Broken pipe
error: failed retrieving file 'extra.db' from distro.ibiblio.org : Broken pipe
extra 468.6K 107.4K/s 00:00:04 [####################################################################] 100%
community 426.0K 55.7K/s 00:00:08 [####################################################################] 100%
multilib 24.3K 16.4K/s 00:00:01 [####################################################################] 100%

Comment by Dave Reisner (falconindy) - Thursday, 22 September 2011, 17:39 GMT

Please retest this on pacman 4.0.0 rc2:

http://mailman.archlinux.org/pipermail/pacman-dev/2011-September/014415.html

Comment by Allan McRae (Allan) - Monday, 20 February 2012, 06:09 GMT

@taylanub: does this work with pacman-4?

Comment by Taylan Ulrich B. (taylanub) - Tuesday, 21 February 2012, 20:17 GMT

@Allan: I don't have an ArchLinux system nor much time these days; I can't give feedback about that, sorry.

	Tasks related to this task (0)

Duplicate tasks of this task (0)

Arch Linux

Pacman

FS#23353 - Privoxy with keep-alive-timeout plus Polipo breaks builtin downloader

Details

Loading...