FS#73673 - [dnscrypt-proxy] Repeatedly crashing with status=31/SYS

Attached to Project: Community Packages
Opened by Balázs Glávits (BalazsGlavits) - Monday, 07 February 2022, 23:17 GMT
Last edited by David Runge (dvzrv) - Wednesday, 16 February 2022, 17:17 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To David Runge (dvzrv)
Architecture x86_64
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:

This issue is a program crash concerning the Community repo package dnscrypt-proxy 2.1.1-1. Systemd is version 250.3-2.
Probably it's the same problem as in task 66147.

Steps to reproduce:

- Enable query logging by setting query_log.file = '/var/log/dnscrypt-proxy/query.log' in /etc/dnscrypt-proxy/dnscrypt-proxy.toml.
- Start dnscrypt-proxy.service and use it to resolve domain names for an extended duration of time. Alternatively, append some junk data to query.log then restart the service.
- When query.log eventually reaches at least 10 MB in size and another DNS query is received by the service, log file rotation will be triggered (as specified by the log_files_max_size key in dnscrypt-proxy.toml).
- dnscrypt-proxy.service will fail due to the main process /usr/bin/dnscrypt-proxy exiting with status SIGSYS (31) "Bad system call".
- Restarting dnscrypt-proxy.service in this state will soon result in failure for the exact same reason upon receiving another query.

Evidence:

I've rebuilt the package with options=(debug !strip). I've triggered the described crash with this debug binary and saved the core dump. Opening the core dump with the Go debugger Delve shows the stack trace found in the attachment dnscrypt-proxy-stacktrace.txt. The sample commands and outputs in the attachment dnscrypt-proxy-permissions.txt show the state of the /var/log/dnscrypt-proxy directory.

Conclusion:

The program attempts to make a fchownat syscall during its log rotation procedure. (Curiously, changing from one uid/gid to the exact same uid/gid. Is this an upstream bug?) However the system call filter in line 32 and 33 of /usr/lib/systemd/system/dnscrypt-proxy.service prevents it from doing so, resulting in an unhandled error state.

I believe that this system call filter is erroneously specified. In line 32 fchownat is added to the whitelist with the @chown system call set but then it's added to the blacklist in line 33 with the @privileged system call set, therefore effectively disallowing it.

In order to allow @system-service, disallow @privileged, and allow @chown all at the same time, three directives should be specified strictly in this order.

Workaround:

Create the service drop-in /etc/systemd/system/dnscrypt-proxy.service.d/override.conf with the following contents:
[Service]
# Clear the old syscall filter. Allow everything:
SystemCallFilter=
# Allow @chown, @privileged, @resources, and other reasonable ones:
SystemCallFilter=@system-service
# Allow only the other reasonable ones:
SystemCallFilter=~@resources @privileged
# Allow @chown, and the other reasonable ones:
SystemCallFilter=@chown
This task depends upon

Closed by  David Runge (dvzrv)
Wednesday, 16 February 2022, 17:17 GMT
Reason for closing:  Fixed
Additional comments about closing:  Fixed with dnscrypt-proxy 2.1.1-2
Comment by David Runge (dvzrv) - Wednesday, 16 February 2022, 08:12 GMT
@BalazsGlavits: Many thanks for this very detailed investigation and suggested fix!

I will look into this in a pkgrel bump and also add the debug package.
Comment by David Runge (dvzrv) - Wednesday, 16 February 2022, 08:33 GMT
@BalazsGlavits: Please see whether 2.1.1-2 in [community-testing] is the appropriate fix for this.
Comment by Balázs Glávits (BalazsGlavits) - Wednesday, 16 February 2022, 16:45 GMT
@dvzrv: Ok I tested 2.1.1-2 by disabling my workaround drop-in, then filling up the log file with megabytes of junk data. The log rotation worked every time. There were no crashes at all.
Comment by David Runge (dvzrv) - Wednesday, 16 February 2022, 17:16 GMT
@BalazsGlavits: Thanks for testing! :)

FWIW, the debug package could not yet be added as for go packages the packages still contain non-compliant directories and files. I will add it as soon as there is a fix released for that with pacman though.

Loading...