FS#50368 - [pacman] skips mirrors if using a HTTP anti virus proxy

Attached to Project: Pacman
Opened by Martin (monojp) - Friday, 12 August 2016, 08:40 GMT
Last edited by Eli Schwartz (eschwartz) - Tuesday, 26 November 2019, 05:27 GMT
Task Type Bug Report
Category General
Status Closed
Assigned To Christian Hesse (eworm)
Architecture All
Severity Low
Priority Normal
Reported Version 5.0.1
Due in Version 5.1.0
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Summary and Info:
We're using a HTTP anti virus proxy which basically intercepts all connections and scans the server responses for viruses. Because of https://bugs.archlinux.org/task/15369 or https://git.archlinux.org/pacman.git/commit/?id=8807cac we have the problem that pacman skips all mirrors (error: failed retrieving file <file> from <mirror> : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds) if downloading huge files because the proxy doesn't return anything while scanning so it may take longer than 10 seconds.

Steps to Reproduce:
# Use a HTTP anti virus proxy
# Download big files via pacman which cause the proxy to scan longer than 10 seconds

Possible Fixes:
# Disable timeout / skipping if there is only one mirror configured
# Maybe even better: Make the timeout configurable. This would suit all users I guess and the current behaviour could be the default one
This task depends upon

Closed by  Eli Schwartz (eschwartz)
Tuesday, 26 November 2019, 05:27 GMT
Reason for closing:  Implemented
Additional comments about closing:  https://git.archlinux.org/pacman.git/com mit/?id=c635f185ba86967cd8de9c31890b57e5 58f65e9b
Comment by Christian Hesse (eworm) - Friday, 12 August 2016, 08:47 GMT
I am suffering the same here... Found a simple workaround, though: Just use https mirrors. The gateway does not break encrypted traffic and passes through the data without delay, making pacman happy again.
Comment by Martin (monojp) - Friday, 12 August 2016, 08:59 GMT
@eworm: Thx for the info. Unfortunately I was using the same workaround for years, but recently our proxy also started intercepting SSL/TLS connections via a certificate that had to be trusted by the clients (that's man in the middle I know, but we were forced that configuration so I can't change anything about it). My current workaround is a patch and a custom PKGBUILD which removes the cURL config responsible for the limits but I would love to see a manual override for that in pacman.
Comment by Christian Hesse (eworm) - Friday, 12 August 2016, 11:09 GMT
Ok, man in the middle breaks this...

I had this idea a long time ago... Finally implemented it. Does the attached patch help?
   0001-libalpm-dload-handle-sec... (2.8 KiB)
Comment by Martin (monojp) - Friday, 12 August 2016, 11:33 GMT
Unfortunately it doesn't. I double-checked if it was patched correctly, but I get the 10 seconds error again.
I was able to debug it a little with cURL, a mirror and the virtualbox package, it seems that I get the 200 OK within ~8 seconds but then it takes almost ~35 seconds for the body. I guess pacman lowers the timeout after I get the 200 OK and then falls into the timeout again?

$ curl -v https://mirror.aardvark.it/archlinux/community/os/x86_64/virtualbox-5.1.2-2-x86_64.pkg.tar.xz
* Trying 188.40.42.195...
* Connected to mirror.aardvark.it (188.40.42.195) port 443 (#0)
* ALPN, offering http/1.1
* <SSL INFO>
> GET /archlinux/community/os/x86_64/virtualbox-5.1.2-2-x86_64.pkg.tar.xz HTTP/1.1
> Host: mirror.aardvark.it
> User-Agent: curl/7.50.1
> Accept: */*
>

~ 8 seconds wait time

< HTTP/1.1 200 OK
< ...

~ 35 seconds wait time until the binary data begins getting printed
Comment by Christian Hesse (eworm) - Friday, 12 August 2016, 11:51 GMT
Pacman waits up to 60 seconds, then lowers the low speed time after data has been received. Some internal curl values are not recalcutated, though - which triggers the error condition.
Comment by Allan McRae (Allan) - Friday, 12 August 2016, 11:56 GMT
  • Assignment removed
Increasing the timeout is not a solution. We have packages that are 900MB in our repos. Can you guarantee that package is downloaded in 60 seconds?
Comment by Martin (monojp) - Friday, 12 August 2016, 12:05 GMT
Would a config entry or command line parameter be a better solution? I am available for further testing, but I can guess that a package with multiple hundred megabytes won't be finished in < 60s with the proxy scanning it.
Comment by Allan McRae (Allan) - Friday, 12 August 2016, 12:11 GMT
I'll accept a "--stupid-proxy" flag.
Comment by Christian Hesse (eworm) - Friday, 12 August 2016, 12:19 GMT
Just contacted the curl people. Currently curl does not recalculate internal values when options are set and/or altered at runtime. The the behaviour is completely undefined.

"--stupid-proxy" flag is a good idea... :D Give me a moment for that.
Comment by Christian Hesse (eworm) - Friday, 12 August 2016, 14:37 GMT
Ok, next try... Does that help?

You have to add command line option '--stupid-proxy' or add 'StupidProxy' to pacman.conf.
   0001-Introduce-a-stupid-proxy... (6.5 KiB)
Comment by Martin (monojp) - Friday, 12 August 2016, 14:49 GMT
Yep, this works perfectly! Tried it with the command line option and independently the config option set.
Comment by Eli Schwartz (eschwartz) - Tuesday, 26 November 2019, 05:26 GMT
This was implemented a long time ago in https://git.archlinux.org/pacman.git/commit/?id=c635f185ba86967cd8de9c31890b57e558f65e9b

Loading...