FS#66067 - [openssl] 1.1.1.f openvpn connections using ssl to fail to verify Cert.

Attached to Project: Arch Linux
Opened by Harold Drost (beren) - Wednesday, 01 April 2020, 16:40 GMT
Last edited by freswa (frederik) - Thursday, 02 April 2020, 12:31 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Pierre Schmitz (Pierre)
Architecture x86_64
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:

This morning I was suddenly no longer able to use our vpn which had been working fine. After long investigation it turned out that openssl was upgraded from 1.1.1.e to 1.1.1.f and when rolling back things worked again. I looked at the release notes for the changes they amount to hardly any change. After logging the bug upstream I had a response suggesting the issue was unlikely with them and to check with downstream (hence this report)

Additional info:
* package version(s)

```
moiraine$>openvpn --version
OpenVPN 2.4.8 [git:makepkg/3976acda9bf10b5e+] x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jan 3 2020
library versions: OpenSSL 1.1.1f 31 Mar 2020, LZO 2.10
Originally developed by James Yonan
Copyright (C) 2002-2018 OpenVPN Inc <sales@openvpn.net>
Compile time defines: enable_async_push=no enable_comp_stub=no enable_crypto=yes enable_crypto_ofb_cfb=yes enable_debug=yes enable_def_auth=yes enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=yes enable_fragment=yes enable_iproute2=yes enable_libtool_lock=yes enable_lz4=yes enable_lzo=yes enable_management=yes enable_multihome=yes enable_pam_dlopen=no enable_pedantic=no enable_pf=yes enable_pkcs11=yes enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=no enable_server=yes enable_shared=yes enable_shared_with_static_runtimes=no enable_small=no enable_static=yes enable_strict=no enable_strict_options=no enable_systemd=yes enable_werror=no enable_win32_dll=yes enable_x509_alt_username=yes with_aix_soname=aix with_crypto_library=openssl with_gnu_ld=yes with_mem_check=no with_sysroot=no
```

* config and/or log files etc.
Can't share config without permission from company. I can probably share some lines from the config if you tell me which ones your interrested in.

Log entries of failing to verify SSL which dissapear when downgrading to 1.1.1.e:

```
Apr 01 13:05:57 moiraine nm-openvpn[17321]: VERIFY ERROR: depth=0, error=unable to get local issuer certificate: O=**********, CN=*********************
Apr 01 13:05:57 moiraine nm-openvpn[17321]: OpenSSL: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
Apr 01 13:05:57 moiraine nm-openvpn[17321]: TLS_ERROR: BIO read tls_read_plaintext error
Apr 01 13:05:57 moiraine nm-openvpn[17321]: TLS Error: TLS object -> incoming plaintext read error
Apr 01 13:05:57 moiraine nm-openvpn[17321]: TLS Error: TLS handshake failed
```

release notes:

```
Major changes between OpenSSL 1.1.1e and OpenSSL 1.1.1f [31 Mar 2020]
Revert the unexpected EOF reporting via SSL_ERROR_SSL
```

working version:

```
moiraine$>openvpn --version
OpenVPN 2.4.8 [git:makepkg/3976acda9bf10b5e+] x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jan 3 2020
library versions: OpenSSL 1.1.1e 17 Mar 2020, LZO 2.10
Originally developed by James Yonan
Copyright (C) 2002-2018 OpenVPN Inc <sales@openvpn.net>
Compile time defines: enable_async_push=no enable_comp_stub=no enable_crypto=yes enable_crypto_ofb_cfb=yes enable_debug=yes enable_def_auth=yes enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=yes enable_fragment=yes enable_iproute2=yes enable_libtool_lock=yes enable_lz4=yes enable_lzo=yes enable_management=yes enable_multihome=yes enable_pam_dlopen=no enable_pedantic=no enable_pf=yes enable_pkcs11=yes enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=no enable_server=yes enable_shared=yes enable_shared_with_static_runtimes=no enable_small=no enable_static=yes enable_strict=no enable_strict_options=no enable_systemd=yes enable_werror=no enable_win32_dll=yes enable_x509_alt_username=yes with_aix_soname=aix with_crypto_library=openssl with_gnu_ld=yes with_mem_check=no with_sysroot=no
```

* link to upstream bug report, if any

https://github.com/openssl/openssl/issues/11456

Steps to reproduce:

> sudo pacman -S openvpn
> sudo openvpn <your openvpn config using openssl>

Fails to verify and tries to reconnect until timed out.

> sudo pacman -U /var/cache/pacman/pkg/openssl-1.1.1.e-1-x86_64.pkg.tar.zst
> sudo openvpn <same openvpn config using openssl>

Succesfully verifies and establishes the vpn.
This task depends upon

Closed by  freswa (frederik)
Thursday, 02 April 2020, 12:31 GMT
Reason for closing:  Upstream
Additional comments about closing:  upstream changed the certificate validation behaviour
Comment by loqs (loqs) - Wednesday, 01 April 2020, 17:51 GMT
Can you please bisect between the two releases and find the causal commit?
Edit:
Build the attached src archive in a clean chroot it should not have the issue.
Change the tag to OpenSSL_1_1_1f build it again in a clean chroot it should have the issue.

cd src/openssl/
git bisect start
git bisect good OpenSSL_1_1_1e
git bisect bad OpenSSL_1_1_1f
Bisecting: 6 revisions left to test after this (roughly 3 steps)
[673692b8d62c8014b70c609caf69a251608303a9] Coverity: fix two minor NPD issues. Found by Coverity.

So update the source line to source=("git+https://github.com/openssl/openssl.git#commit=673692b8d62c8014b70c609caf69a251608303a9"
Again clean chroot build, test pass the result to git to get the next commit to check. Repeat until git has found the cause.
See also https://wiki.archlinux.org/index.php/Bisecting_bugs_with_Git
Comment by Jonas Witschel (diabonas) - Wednesday, 01 April 2020, 20:04 GMT
I think I know where the issue is, it's probably a combination of a broken self-signed certificate served by your VPN provider and an (unintended?) behavioural change to certificate verification in OpenSSL 1.1.1f: https://github.com/openssl/openssl/issues/11456#issuecomment-607460998
Comment by Harold Drost (beren) - Thursday, 02 April 2020, 09:00 GMT
Thanks both for the comments, I'll check with the serverside and see if they can resolve the cert issue.

Loading...