FS#61871 - [nginx][openssl] nginx failed to start

Attached to Project: Arch Linux
Opened by Vladimir (_v_l) - Wednesday, 27 February 2019, 00:10 GMT
Last edited by Giancarlo Razzolini (grazzolini) - Monday, 20 January 2020, 00:27 GMT
Task Type Bug Report
Category Packages: Testing
Status Closed
Assigned To Bartłomiej Piotrowski (Barthalion)
Giancarlo Razzolini (grazzolini)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description: after updating to openssl-1.1.1b nginx failed to start (checked with nginx-mainline 1.15.8 and 1.15.9), with openssl-1.1.1a with the same configuration nginx starts fine.


Additional info:
* nginx-mainline 1.15.8/1.15.9, openssl-1.1.1a/openssl-1.1.1b
* part of nginx configuration (works fine with openssl-1.1.1a):

ssl_protocols TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ecdh_curve prime256v1:secp384r1;
ssl_ciphers TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384;


* link to upstream bug report, if any

Steps to reproduce:

- set up nginx to use TLS 1.3 using above configuration (may be that will require to generate a new certificate, not sure);
- ensure that system has nginx-mainline (tested only with it) version either 1.15.8 or 1.15.9 and openssl-1.1.1a; this combination should work;
- update openssl package to ver. 1.1.1b, reload systemd daemon and restart nginx-mainline; see that nginx failed to start and 'status' shows

systemd[1]: Starting A high performance web server and a reverse proxy server...
nginx[9256]: 2019/02/27 08:00:25 [emerg] 9256#9256: SSL_CTX_set_cipher_list("TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384") failed (SSL: error:1410D0B9:SSL routines:SSL_CTX_set_cipher_list:no cipher match)
systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
systemd[1]: nginx.service: Failed with result 'exit-code'.
systemd[1]: Failed to start A high performance web server and a reverse proxy server.
systemd[1]: nginx.service: Consumed 0 CPU time.

The openssl-1.1.1b has these ciphers but still nginx fails to start.

P.S. Not sure if this is a upstream bug or distribution bug.

---
Vladimir Lomov
This task depends upon

Closed by  Giancarlo Razzolini (grazzolini)
Monday, 20 January 2020, 00:27 GMT
Reason for closing:  Fixed
Comment by Vladimir (_v_l) - Wednesday, 27 February 2019, 00:50 GMT
Seems that openssl-1.1.1b breaks the "compatability" with openssl-1.1.1a intensionally. From https://www.openssl.org/news/changelog.html#x1

Change the info callback signals for the start and end of a post-handshake
message exchange in TLSv1.3. In 1.1.1/1.1.1a we used SSL_CB_HANDSHAKE_START
and SSL_CB_HANDSHAKE_DONE. Experience has shown that many applications get
confused by this and assume that a TLSv1.2 renegotiation has started. This
can break KeyUpdate handling. Instead we no longer signal the start and end
of a post handshake message exchange (although the messages themselves are
still signalled). This could break some applications that were expecting
the old signals. However without this KeyUpdate is not usable for many
applications.
[Matt Caswell]

There is a chance that the problem not with openssl but with nginx. I reported the problem to nginx developers. I'll post news as soon as the ticket will be active.
Comment by loqs (loqs) - Wednesday, 27 February 2019, 00:54 GMT
Looking at Failed (SSL: error:1410D0B9:SSL routines:SSL_CTX_set_cipher_list:no cipher match) the change to SSL_CTX_set_cipher_list in
https://github.com/openssl/openssl/commits/OpenSSL_1_1_1b but not in https://github.com/openssl/openssl/commits/OpenSSL_1_1_1a
I would have the thought the issue would be https://github.com/openssl/openssl/commit/1c31fe7eb093a8f07d32e910a46616209883cf84
Comment by loqs (loqs) - Wednesday, 27 February 2019, 01:43 GMT
https://github.com/nginx/nginx/blob/release-1.15.9/src/event/ngx_event_openssl.c#L824
see man SSL_CTX_set_cipher_list
SSL_CTX_set_cipher_list() sets the list of available ciphers (TLSv1.2
and below) for ctx using the control string str. The format of the
string is described in ciphers(1). The list of ciphers is inherited by
all ssl objects created from ctx. This function does not impact TLSv1.3
ciphersuites. Use SSL_CTX_set_ciphersuites() to configure those.
https://github.com/openssl/openssl/commit/1c31fe7eb093a8f07d32e910a46616209883cf84 stops TLSv1.3 ciphers from being valid for SSL_CTX_set_cipher_list which is consistent with the man page but not backwards compatible.
nginx does not support SSL_CTX_set_ciphersuites
Comment by loqs (loqs) - Wednesday, 27 February 2019, 12:44 GMT
https://trac.nginx.org/nginx/ticket/1734 conclusion the configuration is invalid.
Comment by Vladimir (_v_l) - Wednesday, 27 February 2019, 13:43 GMT
2loqs:
If I understand the answer on the ticket correctly, the nginx won't work with openssl-1.1.1b and TLSv1.3. I tested nginx with openssl-1.1.1b without setting explicitly the ciphers and it failed to start. So basically, since they (nginx developers) won't implement (in any near future I think) the new openssl interface then nginx+TLSv1.3 will be broken.
Comment by loqs (loqs) - Wednesday, 27 February 2019, 17:04 GMT
This is with ssl_protocols TLSv1.3 so TLSv1.2 and below are disabled?
The NGX_DEFAULT_CIPHERS is set to HIGH:!aNULL:!MD5
What is the error message?
Did you try https://trac.nginx.org/nginx/ticket/1529#comment:12 ?
Comment by Vladimir (_v_l) - Thursday, 28 February 2019, 00:31 GMT
> This is with ssl_protocols TLSv1.3 so TLSv1.2 and below are disabled?
Exactly.

> The NGX_DEFAULT_CIPHERS is set to HIGH:!aNULL:!MD5
> What is the error message?

When I commented the ssl_ciphers but keep TLSv1.3 setting the error message was the same.

> Did you try https://trac.nginx.org/nginx/ticket/1529#comment:12 ?

I did, but it was so I didn't try it. Now I tried it:
- created the /etc/openssl/openssl.conf file with the following content:
openssl_conf = default_conf

[default_conf]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
Ciphersuites = TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384
- changed a bit the nginx configuration:
ssl_protocols TLSv1.3;
# ssl_prefer_server_ciphers on;
ssl_ecdh_curve prime256v1:secp384r1;
# ssl_ciphers ECDH+ECDSA:+AES256;
# ssl_ciphers TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384;
- updated openssl to 1.1.1b
- and restarted nginx ... and it works!

I'm fine if this configuration will work as intended forever. Now I must say that nginx+openssl-1.1.1b+TLSv1.3 works perfectly.

P.S. Seems that one previous error was due to my mistake, I overlook another settings with ciphers.
Comment by Vladimir (_v_l) - Thursday, 28 February 2019, 03:05 GMT
Seems I did not understand what I should do.

I made file '/etc/openssl/openssl.conf' but according to documentation this is wrong directory, the correct one is '/etc/ssl'. But this directory already has 'openssl.conf' file and I'm not sure how to add above settings to it. If I simply add the lines above to the end of '/etc/ssl/openssl.conf', nothing is changed, nginx will use 'TLS_AES_256_GCM_SHA384' cipher instead of 'TLS_CHACHA20_POLY1305_SHA256', I checked that with 'openssl s_client'.

According to config(5) an application should somehow to read the new config but where it will search for the config file? I'm a bit lost here.
Comment by Vladimir (_v_l) - Monday, 04 March 2019, 13:56 GMT
Ok, if I get thing right then I need to a copy of nginx.service file in /etc/systemd/system and I have to add Environment setting:
Environment=OPENSSL_CONF=/etc/nginx/openssl.conf

Of course before that one needs to stop nginx, disable the service, then copy the nginx.service, modify the file, enable and start the service. With with advice the problem is solved.
Comment by Giancarlo Razzolini (grazzolini) - Friday, 17 January 2020, 14:37 GMT
Does this issue still happens with the latest nginx and openssl versions?
Comment by Vladimir (_v_l) - Saturday, 18 January 2020, 01:42 GMT
@grazzolini,
No, now everything works fine (actually, I don't remember when I resolved the issue, may be on April or May and since then nginx works fine with TLS1.3, so I almost forgot about the ticket, sorry).
Comment by Giancarlo Razzolini (grazzolini) - Monday, 20 January 2020, 00:27 GMT
No problem. I'm closing this one. This issue got lost in the torrent of bugs, but good to know it's fixed now.

Loading...