FS#28259 - https://aur.archlinux.org/ no longer loads

Attached to Project: AUR web interface
Opened by cfr (cfr42) - Sunday, 05 February 2012, 02:57 GMT
Last edited by Lukas Fleischer (lfleischer) - Thursday, 01 November 2012, 00:03 GMT
Task Type Bug Report
Category Backend
Status Closed
Assigned To Lukas Fleischer (lfleischer)
Architecture All
Severity High
Priority Normal
Reported Version 1.9.0
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 10
Private No

Details

For the past few days, I've gotten errors anytime I tried to access, view or search https://aur.archlinux.org/. I discovered this because it breaks aurget but the problem is much more general e.g. Firefox is unable to load the page.

This is specific to this site as far as I can tell. I can access AUR over http. I can also load other SSL pages just fine e.g. https://bugs.archlinux.org/newtask/proj2. I've been through my logs and can't even see an acknowledgement of the error. (I tried running aurget and looking for something matching the time.)

With aurget, I get the following error from curl:

curl: (35) Unknown SSL protocol error in connection to aur.archlinux.org:443

With Firefox, I get an error printed in Welsh. I use Welsh because I'm trying to learn it which isn't maximally useful in this case, but my Welsh can manage most of this and I've a dictionary to hand so here's an attempted translation:

Cafodd y cysylltiad ei darfu
- The connection was "scattered". (That's what my dictionary says for 'tarfu' which is mutated to 'darfu' here.) I guess "interrupted" or "broken" might be about right.)

Cafodd cysylltiad â aur.archlinux.org ei darfu wrth i'r dudalen lwytho.
- The connection with aur.archlinux.org was interrupted/scattered as the page loaded.

Efallai bod y wefan yn brysur neu nad yw ar gael dros dro. Ceisiwch eto ymhen ychydig.
- Perhaps the site is busy or not available temporarily. Try again in a little while.

Os nad ydych yn gallu llwytho unrhyw dudalennau, gwiriwch gysylltiad rhwydwaith eich cyfrifiadur.
- If you cannot load any pages, verify your computer's network connection.

Os yw eich cyfrifiadur neu rwydwaith wedi ei ddiogelu gan fur cadarn neu ddirprwy, gwnewch yn siŵr fod gan Firefox hawl i fynediad i'r we.
- If your computer or network is protected by a firewall or proxy, make sure that Firefox has a right to connect to the web. ('mur cadarn' isn't the term I know for 'firewall' but it literally means 'strong wall' so I think that must be what's meant; 'dirprwy' is another new word which literally means 'delegate' or 'proxy' according to the dictionary so I think it must be 'proxy'.)
This task depends upon

Closed by  Lukas Fleischer (lfleischer)
Thursday, 01 November 2012, 00:03 GMT
Reason for closing:  Fixed
Additional comments about closing:  The AUR has been moved to a new server which fixed the issue. Please reopen if the problem persists.
Comment by cfr (cfr42) - Sunday, 05 February 2012, 03:05 GMT
I just discovered that using the web console rather than the error console in Firefox gives me something, though not much:

[03:03:22.541] GET https://aur.archlinux.org/ [undefined 20561ms]
Comment by Andreas (misc) - Tuesday, 07 February 2012, 20:10 GMT
curl: same error

aria2: "[SocketCore.cc:915] errorCode=1 SSL initialization failed: The TLS connection was non-properly terminated."

wget: "... connected. Unable to establish SSL connection."

Firefox (-nightly): "The connection was interrupted"

Chromium (-dev): "Error 107 (net::ERR_SSL_PROTOCOL_ERROR)"

(Testing repos enabled)
Comment by cfr (cfr42) - Tuesday, 07 February 2012, 22:58 GMT
To clarify: I do NOT have testing repos enabled. Everything available from the official repositories is up to date. (I can't say about AUR right now for reasons connected with this bug...)
Comment by Szunti (Szunti) - Wednesday, 08 February 2012, 16:28 GMT
Same for me without testing repos.
Comment by Ionut Biru (wonder) - Wednesday, 08 February 2012, 16:43 GMT
i don't know why for you guys is not working. maybe something is performing a man in the middle attack?
Comment by Szunti (Szunti) - Wednesday, 08 February 2012, 18:30 GMT
With Win Xp, and on my other pc with Ubuntu, it loads.
Comment by Ionut Biru (wonder) - Wednesday, 08 February 2012, 18:38 GMT
have you tried to reinstall nss, ca-certificates and openssl?
Comment by Szunti (Szunti) - Wednesday, 08 February 2012, 18:51 GMT
Thanks for suggesting packages.
I tried downgrading ca-certificates, openssl, the kernel, glibc (although only one at a time) without succes. Then reinstalled those packages, still without success. Firefox depends only on nss, which as i found don't depend on openssl, so I don't understand how can both fail.
Comment by Szunti (Szunti) - Wednesday, 08 February 2012, 19:24 GMT
Real mystery for me. Now I tried my Arch pendrive with the image I used to install. And 'curl https://aur.archlinux.org'; gives this same error, although I had no problem till a couple of days ago: I used cower, which I can't now.

Then I tried an Ubuntu 11.10 live cd and curl worked.
Comment by Andreas (misc) - Wednesday, 08 February 2012, 19:30 GMT
Tried with ca-certificates reinstalled, openssl (latest snapshot) and nss recompiled; still the same.
Comment by Pierre Schmitz (Pierre) - Wednesday, 08 February 2012, 20:46 GMT
Does it work now? Does http://www.archlinux.de/ work for you? (It uses a similar setup)
Comment by Andreas (misc) - Wednesday, 08 February 2012, 21:02 GMT
Still does not work for me. All other sites are accessible via https, incl. Arch Wiki, BBS and .de — just not the AUR.
Comment by Pierre Schmitz (Pierre) - Wednesday, 08 February 2012, 21:03 GMT
Might be some kind of routing problem then.
Comment by cfr (cfr42) - Wednesday, 08 February 2012, 21:49 GMT
I've discovered it depends where I am. I only usually access AUR from home but today I tried loading the page at work and, mystery of mysteries, it loaded.

I'm back home now and I'm back to the same error.

I thought maybe it was interface related because I used a wired connection to check on campus and almost always use wireless at home. But that's not it. I get the same error with a wired connection at home as I do with wireless.

As others have said, the problem is peculiar to AUR over a secure link. (At least, I've yet to find the same problem with other sites.) I can get AUR insecurely. I can get regular arch, bugs etc. securely. I can get other SSL sites and non-SSL sites.

http://www.archlinux.de/ is fine. So is https://www.archlinux.de/, https://www.archlinux.de/?page=Packages etc. But, then, these are OK on the main site as well. It is only AUR over SSL which is affected.

Is this man-in-the-middle attack plausible? I don't know much about how they work but...
Comment by cfr (cfr42) - Wednesday, 08 February 2012, 21:59 GMT
I just established that another machine on my home network can load the page fine in firefox. That machine is running Linux Mint 12. The machines share an external IP address.

So on the one hand, how can it be my machine when it works on campus?

On the other hand, how can it be anything but my machine when a different machine on the same LAN works?
Comment by Lukas Fleischer (lfleischer) - Wednesday, 08 February 2012, 23:14 GMT
I updated sigurd on 2012-02-01, so this might be related. I just checked our lighty logs and I see *lots* of "(connections.c.1717) SSL (error): 5 -1 104 Connection reset by peer" entries there (new errors appear every 5 seconds or so, so you're not alone :) ). We should probably check if I missed some PHP config changes (didn't merge the "php.ini.pacnew" yet)...
Comment by Cam Webb (camwebb) - Thursday, 09 February 2012, 13:55 GMT
I also can't get to https://aur.archlinux.org/ . Using chromium started from the command line this is the error report: [11363:11379:51734948930:ERROR:ssl_client_socket_nss.cc(1536)] handshake with server aur.archlinux.org:443 failed; NSS error code -5938, net_error -107
Comment by Szunti (Szunti) - Thursday, 09 February 2012, 22:57 GMT
I found that wicd causes this for me.

How about you? Do you use wicd? Does switching to NetworkManager helps like it did for me?
Comment by Szunti (Szunti) - Friday, 10 February 2012, 00:31 GMT
The problem seems to be the mtu. NetworkManager leaves it at 1500, while dhcpcd and wicd made it smaller.

After
$> sudo ip link set wlan0 mtu 1500
aur works again.

If it makes sense for anyone, pls explain me why do you need to increase the mtu.

Comment by Cam Webb (camwebb) - Friday, 10 February 2012, 06:44 GMT
I can confirm that I can access https AUR with a 3g modem w ppp0, but not with wicd. And that:

# ip link set wlan0 mtu 1500

fixes it (thanks Vandrus!). Also curious why this works.

Comment by cfr (cfr42) - Friday, 10 February 2012, 14:47 GMT
I use wicd. But if wicd is the problem, why can I access the site from campus? I can't actually try the fix now because I'm on campus so I don't see the problem but I'll try later from home.

Also, from home, the machine with Mint is also using wicd and that works OK. (NetworkManager caused problems so is disabled.)
Comment by cfr (cfr42) - Friday, 10 February 2012, 21:44 GMT
I can, however, confirm that the fix does work for me too. What exactly does it do? I read the man pages but they don't seem to say what MTU is?

I don't get it, though. Why doesn't my machine need the same fix on campus?

But thanks very much for figuring a work around! Presumably this won't be necessary once the server stuff is looked at and fixed but it is great for now.
Comment by Szunti (Szunti) - Friday, 10 February 2012, 22:04 GMT
I was wrong first. It's not wicd that cause the problem, it's the mtu(http://en.wikipedia.org/wiki/Maximum_transmission_unit) that was decreased. Dhcpcd makes the mtu change for me. Wicd just uses dhcpd. Most probably you don't use dhcp to get the ip on campus. I at least have static ip on mine.

I don't know if it's the servers fault. I hope someone who knows these things better can figure out what behaves incorrectly.
Comment by cfr (cfr42) - Friday, 10 February 2012, 22:11 GMT
Edit: pipped to the post.

I don't know if this is really a wicd thing or not. wicd uses dhcpcd and dhcpcd seems to be set to respect the network MTU setting. However, netcfg and networkmanager also depend on dhcpcd so I'm not sure if they override that setting or if something else is going on.

But if that's right, it would explain some things e.g. why it might work for the same machine on different networks (home network might set MTU lower than campus? don't know why...) It would also explain why some people might not ever see the issue even if using wicd. But I'm not convinced I understand why it works on Mint on my home network unless Mint sets up dhcpcd differently?

I don't have a static ip on campus - I'm using dhcpd there to get an ip from the dhcp server. But if the network has a different default MTU setting, dhcpcd would set the setting to that and it might be OK.
Comment by cfr (cfr42) - Saturday, 11 February 2012, 02:50 GMT
Does anybody know how to make it stick? Mine gets reset... (I don't mean on reboot or anything just in the background somehow.)
Comment by Andreas (misc) - Saturday, 11 February 2012, 16:32 GMT
Just to add further confirmation: Of the three networks which I've tested, the failure indeed only occurs on the one in which wicd & dhcpcd have to care for the outside IP (etc.) themselves.

I'm not sure if this behavior started with the latest dhcpcd update or the one of wicd a few days prior to it.
Comment by Lukas Fleischer (lfleischer) - Saturday, 11 February 2012, 17:08 GMT
MTU is the maximum size of a packet (in bytes) that can be transferred in one frame (without fragmentation). My guess is that setting this to a very low value might cause problems with the protocol overhead of SSL... I'm not sure if this a client-side only issue - especially since most of you said that it works with other websites. I'll have a look at our server config. There's some other minor things to fix anyway (e.g. some PHP memory issues that are probably completely unrelated, though).
Comment by cfr (cfr42) - Saturday, 11 February 2012, 23:49 GMT
I'm not sure what you mean by their taking care of the outside IP etc. themselves. On campus, I'm also using dhcpd and wicd to connect and they have to get an ip address etc. just as they do on my home LAN although the authentication set up is different in the case of wireless. So I don't think that's enough to create the problem. It has to be something more than that.

Is the server config for ssl AUR different from the config for other archlinux.org sites over ssl e.g. bugs.archlinux.org etc.? Because we are all happily (well, maybe not quite that) posting on https://bugs.archlinux.org/...
Comment by cfr (cfr42) - Sunday, 12 February 2012, 15:51 GMT
Looking at the logs, when I connect on campus, dhcpcd itself "restores" MTU to 1500. At home, dhcpcd sets it to 576. Is there something I should change about my home LAN? Or is this an ISP thing? (But then it works OK for the computer here running Mint and using wicd etc....)
Comment by Szunti (Szunti) - Sunday, 12 February 2012, 20:32 GMT
First you should try to find the mtu setting in your router.

In my case that didn't help. Mtu on my router's admin page is set to 1500, still the dhcp server in my router sends 576 (I can see that in wireshark).

The example config on dhcpcd's webpage (http://roy.marples.name/projects/dhcpcd/wiki/DhcpcdConfig) tells that there are many routers with this problem.
The solution is to comment out the

#option interface_mtu

line in /etc/dhcpcd.conf. Which means that dhcpcd will no longer ask the dhcp server for the mtu. I used ping as described on http://muzso.hu/2009/05/17/how-to-determine-the-proper-mtu-size-with-icmp-pings to get the mtu that my ISP use. That was 1500 in my case.

If it's not 1500 you can set the mtu in dhcpcd.conf by changing the option interface_mtu line to:

static interface_mtu=1480



Comment by cfr (cfr42) - Sunday, 12 February 2012, 22:59 GMT
So I commented out the option line in dhcpcd.conf and I used ip link set to restore the 1500 mtu setting temporarily.

I tried to use the ping method to establish the appropriate mtu. The result I get is 548. That's smaller than dhcpcd is setting it anyway. In any case, I'm not sure I should set the static mtu line since I move between different networks which I think have different mtus...

But I'm not quite sure what I'm meant to do to the router so I didn't do anything which may be the problem...

Hopefully something will get changed in the config for aur in the meantime and I won't have to figure this out!
Comment by cfr (cfr42) - Monday, 13 February 2012, 01:14 GMT
Well some googling suggests that:
1) my ISP recommends an MTU of 1500;
2) my router does not alter the packets in any way so that it is up to the computers on the LAN to set the MTU.

But if that's so, I don't see where the 576 figure I'm seeing is coming from...
Comment by Igor Saric (karabaja4) - Monday, 05 March 2012, 22:03 GMT
I have this problem too and messing with MTU didn't help :(
Comment by Phillip Wood (phil) - Tuesday, 06 March 2012, 16:06 GMT
This problem started for me last night. I am unable to access https://aur.archlinux.org but can access it under http. I am able to access all the other archlinux sites using https (and all other https sites that I have tried) Chromium says
Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error.
When I try to access the site. ip -d link gives
4: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
so my mtu is 1500. I am unable to access the site using safari on my ipod touch as well so I don't think it is a configuration problem with my archlinux box.
Comment by Phillip Wood (phil) - Tuesday, 06 March 2012, 16:23 GMT
I have attached the output of 'tcpdump -i wlan0 host aur.archlinux.org -w aur.dump' while trying to connect to the aur with wget in the hope that it will help in diagnosing the problem.

wget https://aur.archlinux.org
--2012-03-06 16:16:05-- https://aur.archlinux.org/
Resolving aur.archlinux.org... 208.92.232.29
Connecting to aur.archlinux.org|208.92.232.29|:443... connected.
Unable to establish SSL connection.

Let me know if any other tests would be useful
   aur.dump (7.5 KiB)
Comment by Lukas Fleischer (lfleischer) - Tuesday, 06 March 2012, 22:31 GMT
Ok, this seems to be a real issue. I didn't have time to investigate this thoroughly but this is probably related to one of our recent server updates and/or to the NIC replacement. Maybe we broke PMTU discovery at some point...
Comment by Phillip Wood (phil) - Wednesday, 07 March 2012, 10:24 GMT
cryptocrack, thanks for looking into this. The problem only occurs if I try to connect over my home broadband, I just ssh'd into a server with a direct internet connection and wget https://aur.archlinux.org worked fine, so it sounds like PMTU discovery may be to blame.
Comment by bohoomil (bohoomil) - Friday, 09 March 2012, 01:14 GMT
My quick 2 cents: using OpenVPN tunneling = no AUR at all (while other https' work fine, including those Arch related). Clean, plain, direct connection = AUR is accessible. The same behaviour can be observed in two different distros, and in both I use a very basic wifi scheme: wpa_supplicant + iproute2 + dhcpcd + a simple autostart daemon.
Comment by Lukas Fleischer (lfleischer) - Friday, 09 March 2012, 12:27 GMT
Seems like updating and rebooting sigurd fixed this. I blame $random_package. Can anyone else, who hasn't been able to access the AUR using HTTPs before, confirm that it is working again? :)
Comment by Noel Maersk (veox) - Friday, 09 March 2012, 12:56 GMT
Didn't work before, works now. Thanks!
Comment by Phillip Wood (phil) - Friday, 09 March 2012, 12:57 GMT
It works for me now :-) Thanks
Comment by Andreas (misc) - Friday, 09 March 2012, 13:13 GMT
Nope, still not working for me without MTU manually set to 1500.
Comment by Igor Saric (karabaja4) - Friday, 09 March 2012, 15:42 GMT
Works again for me too. Thanks.
Comment by Szunti (Szunti) - Friday, 09 March 2012, 17:01 GMT
Same as for Andreas: still not working without MTU set to 1500.
Comment by Alexander F. Rødseth (xyproto) - Friday, 09 March 2012, 19:29 GMT
MTU is 576 here and https://aur.archlinux.org doesn't work.

Could this option in /etc/dhcpcd.conf have anything to do with it?

# Respect the network MTU.
option interface_mtu

When setting MTU to 1500 it works again.
Comment by Kohei Yoshitomi (anergy) - Saturday, 10 March 2012, 12:31 GMT
Neither when this problem had occured to me and when it has been fixed,
but I now can access aur.archlinux.org with https. Thanks a lot!

BTW, when I was unable to connect, I tried some online proxy and got worked.
e.g
$ https_proxy="210006020247.ctinets.com:3128" curl "https://aur.archlinux.org"
This is dangerous as unknown proxy server snoops your request so do not try...
Comment by SanskritFritz (SanskritFritz) - Friday, 08 June 2012, 12:37 GMT
For me this workaround works:
ip link set eth0 mtu 1500
cower, burp, aurphan are usable again. The problem first occured at 2012.06.07 here.
UPDATE: on the second thought, I switched ISP some days ago, that is most probably the reason.
Comment by Dan Seminara (semi225599) - Sunday, 17 June 2012, 03:35 GMT
For me even trying to access it from a Windows machine doesn't work. It's not an MTU problem, or if it was in March, it isn't now. My router and OS tell me the MTU is 1500 and that doesn't fix it. If it's the ISP is everyone that uses Comcast having the problem? I have LogMeIn Hamachi running (VPN), I will try disabling that and rebooting to see if that works.

To reiterate the problem, https://aur.archlinux.org is completely inaccessible, but the http site is fine.
Comment by Dan Seminara (semi225599) - Saturday, 30 June 2012, 20:28 GMT
When trying to access from Google Chrome on Windows, I get the following error:

SSL connection error
Unable to make a secure connection to the server. This may be a problem with the server, or it may be requiring a client authentication certificate that you don't have.
Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error.
Comment by rob.til.freedman (rtfreedman) - Friday, 17 August 2012, 12:58 GMT
I can't access https://aur.archlinux.org - http:// is no problem - for several days.
Midori says 'SSL handshake failed', tried Firefox and also packer
Comment by Gaurish Sharma (gary4gar) - Friday, 31 August 2012, 08:53 GMT
I still can't open aur site, get the error "SSL protocol error". Attaching TCPdump & curl out, so it helps you guys to debug.

Link: https://gist.github.com/3550261

Comment by rob.til.freedman (rtfreedman) - Friday, 31 August 2012, 13:22 GMT
This was solved by setting MTU >=1492
Look at the output of ifconfig
Comment by Lukas Fleischer (lfleischer) - Sunday, 28 October 2012, 10:28 GMT
Status? Does this still occur?
Comment by Gaurish Sharma (gary4gar) - Sunday, 28 October 2012, 10:54 GMT
No, The problem got solved without using any workaround, it works fine now. https://gist.github.com/3968306

So what was the problem?

Loading...