FS#50090 - [openvpn] version 2.3.11-2 fails to connect with long passwords (>128 chars)

Attached to Project: Arch Linux
Opened by Giancarlo Razzolini (grazzolini) - Monday, 18 July 2016, 18:22 GMT
Last edited by Christian Hesse (eworm) - Friday, 23 September 2016, 13:26 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Christian Hesse (eworm)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No


Version 2.3.11-2 fails to establish any connection. The connections hang on the TLS negotiation phase, and fails after 60 seconds, the default timeout. Downgrading the package solve the issue. The only difference on the package is the enabling of pkcs11:


Additional info:
* version 2.3.11-2

Steps to reproduce:
Try to establish any connection.
This task depends upon

Closed by  Christian Hesse (eworm)
Friday, 23 September 2016, 13:26 GMT
Reason for closing:  Upstream
Additional comments about closing:  Upstream (and remote ISP) issue.
Comment by Gerardo Exequiel Pozzi (djgera) - Monday, 18 July 2016, 19:20 GMT
Works fine here.

Logs please...
Comment by Giancarlo Razzolini (grazzolini) - Monday, 18 July 2016, 19:42 GMT
I'm using private internet access (PIA). OpenVPN never completes a handshake, doesn't matter which configuration I'm using or server I'm connecting to. I'll try later today to establish a connection to a server I have control over. But a packet capture with tcpdump doesn't shed much more light into it. It shows my machine(s) starting a connection to the server and, after some point, only packets sent from it and no replies from the server. If you need more verbosity, please let me know.
Comment by Giancarlo Razzolini (grazzolini) - Monday, 18 July 2016, 20:15 GMT
I just compiled the package from abs, to discard any issue with the build process. The same issue is present. If I remove the --enable-pkcs11 from the PKGBUILD and rebuild it, the issue goes away. I fail to see why enabling PKCS#11 would trigger this behavior, unless there is something wrong with OpenSSL. I will try to diagnose it further later.
Comment by Giancarlo Razzolini (grazzolini) - Tuesday, 19 July 2016, 14:37 GMT
I can confirm that this behavior only seems to happen with PIA. I successfully connected to servers that I control. But I managed to discard a machine specific issue, because this happened on another machine I own, with a different hardware.
Comment by Bernardas Ališauskas (granitosaurus) - Tuesday, 19 July 2016, 16:26 GMT
I'm having the same issue with AirVpn and their generated profiles.
edit: I've tried "If I remove the --enable-pkcs11 from the PKGBUILD and rebuild it, the issue goes away." but mine kept failing anyway. However the issue doesn't seem to be present on some AirVpn servers, I suspect the issue might be on their side.
Comment by Christian Hesse (eworm) - Tuesday, 19 July 2016, 20:48 GMT
I can not reproduce here with my servers...
And nobody has access to a server where connection failed, no?

Can you please increase verbosity to... let's try 4... and create two logs, one with successful connect, one with error?
Comment by Giancarlo Razzolini (grazzolini) - Tuesday, 19 July 2016, 21:46 GMT
I increased it to 6, because it shows the UDP READ's and WRITE's (there is no major difference between a verb 3 and 4). I included the configuration I'm using, the CA and CRL files and there are 2 logs, one that shows a working connection and the other that doesn't. The one that doesn't is openvpn installed from the repos. The one that does work is openvpn compiled without --enable-pkcs11 and without the depend on pkcs11-helper. I used the PKGBUILD from abs to do that.

I tried playing with link-mtu, tun-mtu and mssfix parameters, but it doesn't seem to be a PMTU issue. And, it happens with all PIA servers I have tested so far.
Comment by Christian Hesse (eworm) - Wednesday, 20 July 2016, 06:33 GMT
The server just stops responding... Wired.
Comment by Christian Hesse (eworm) - Wednesday, 20 July 2016, 10:08 GMT Comment by Giancarlo Razzolini (grazzolini) - Wednesday, 20 July 2016, 16:55 GMT
I saw that they closed it as notabug. Indeed, it seems to be just with PIA servers that this is happening. I find it strange that more people didn't reported this yet. I also discarded that this would be an ISP issue (I use the same ISP in both my home and work places). I couldn't connect using another ISP, or even a cellular data connection.

I will open a ticket support with them. They recently (very recently) beefed up their security after their russian servers were seized. They changed their CA's and CRL's, and also implemented stronger ciphers and certificates. It is possible that in this revamp, they changed something that interacts badly with PKCS#11.
Comment by Christian Hesse (eworm) - Wednesday, 17 August 2016, 16:37 GMT
Any news from privateinternetaccess.com service?
Comment by Giancarlo Razzolini (grazzolini) - Wednesday, 17 August 2016, 16:57 GMT
I have opened a ticket with them, they escalated it to their development team, but I did not heard from them after that. Either they silently fixed it or didn't looked further. I guess you can close this ticket. When the next openvpn version lands, I will test again. It is working without pkcs#11 just fine.
Comment by Giancarlo Razzolini (grazzolini) - Thursday, 25 August 2016, 01:28 GMT
Time to re-open that upstream ticket. It turns out I have a very long password (256 chars) and it was the culprit. If I use a smaller sized password it works. I didn't looked at PKCS#11's OpenVPN relevant code yet, but I will look at it tomorrow, perhaps I can understand why just enabling it, interacts badly with long sized passwords. If you guys want I can re-open it myself, since it doesn't seem to be arch related.
Comment by Giancarlo Razzolini (grazzolini) - Thursday, 25 August 2016, 13:43 GMT
This also affects version 2.3.12-1 that was published yesterday. Also, I don't know the exact size of password that make this issue to manifest.
Comment by Christian Hesse (eworm) - Wednesday, 21 September 2016, 21:37 GMT
In fact PKCS#11 does change the supported password length in openvpn. Without PKCS#11 it supports 128 chars, with PKCS#11 it supports 4096 chars.

So when you tested with a shorter password... You changed it on server and client side?

I suppose PIA runs a version with PKCS#11 support disabled, so only passwords up to 128 chars are used. To verify: Set a password with 256 chars on server and client side, try to connect... Should fail. Do not change the password on server but truncate it to 128 chars on client side. Does it connect?
Comment by Giancarlo Razzolini (grazzolini) - Thursday, 22 September 2016, 05:03 GMT
If I don't change it on the server (it still is 256 chars), but I truncate it on the client side to exactly 128 chars, it does indeed work. This indicates that when PKCS#11 is not enabled, then openvpn does the truncating of the password to 128 chars. But when PKCS#11 is enabled, it doesn't. So, this is more complicated than a simple bug, because it might be intended behavior. So, PIA should either use OpenVPN server's with PKCS#11 enabled at compile time, or they should limit the OpenVPN password to 128 chars. Nevertheless, OpenVPN probably still has a bug, because the connection simply hangs.
Comment by Christian Hesse (eworm) - Thursday, 22 September 2016, 08:06 GMT
Thanks for verification!

I reopened the upstream bug report. Let's see what happens there...

Is the PIA support ticket still open? You could tell them to limit password length to 128 chars (and give an error otherwise).
Comment by Giancarlo Razzolini (grazzolini) - Thursday, 22 September 2016, 11:15 GMT
I've seen OpenVPN developers effectively solved the issue by raising the password length, regardless of PKCS#11. The ticket I had with PIA was closed a long time ago, but I can see it re-opened. This one was hard to diagnose, but I'm glad we found it.
Comment by Maxime Poulin (Max-P) - Thursday, 22 September 2016, 18:28 GMT
I'm certainly glad you found it as well, because I was scratching my head quite a bit on the other end trying to reproduce despite both of us practically having pacstrapped an identical VM with different results haha :) Sent the upstream bug report to the ops, hopefully they can deploy soon-ish. That sure was an interesting one!
Comment by Christian Hesse (eworm) - Friday, 23 September 2016, 13:25 GMT
Upstream is aware of the issue and there is nothing we can do about it. In fact we are perfectly fine.
Wait for upstream to fix and blame PIA (and others) to deploy a working version.
Closing for now.