Arch Linux

Please read this before reporting a bug:
https://wiki.archlinux.org/index.php/Reporting_Bug_Guidelines

Do NOT report bugs when a package is just outdated, or it is in Unsupported. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!
Tasklist

FS#68480 - [openvpn] running unprivileged with iproute no longer working

Attached to Project: Arch Linux
Opened by tom (archtom) - Friday, 30 October 2020, 16:05 GMT
Last edited by Christian Hesse (eworm) - Friday, 06 November 2020, 07:03 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To Christian Hesse (eworm)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 4
Private No

Details

Description:

After updating to openvpn 2.5.0 using
iproute /usr/local/sbin/unpriv-ip
as server config option is no longer working.

The wiki
https://community.openvpn.net/openvpn/wiki/HOWTO#UnprivilegedmodeLinuxonly
still lists it as official solution to run unpriviliged but states that the package has to be build with the --enable-iproute2 option.
Also the archwiki still lists this as solution.

According to
https://github.com/archlinux/svntogit-packages/commit/e3e9b45fb62e267bea0fa7bc6077e2c162749e2f#diff-3e341d2d9c67be01819b25b25d5e53ea3cdf3a38d28846cda85a195eb9b7203a
the package is currently build without the needed option.

The service fails to start when using iproute /usr/local/sbin/unpriv-ip in the server config file.

Please re-build the package with the needed option --enable-iproute2.

Thanks

Additional info:
* package version(s)
openvpn 2.5.0-1
This task depends upon

Closed by  Christian Hesse (eworm)
Friday, 06 November 2020, 07:03 GMT
Reason for closing:  Fixed
Additional comments about closing:  openvpn 2.5.0-2
2020-11-10: A request to re-open the task has been made. Reason for request: Not really working. See: https://bbs.archlinux.org/viewtopic.php?id=260567 https://bbs.archlinux.org/viewtopic.php?id=260625
Comment by Boris "Hundi Wam Wam" Digital (boris64) - Friday, 30 October 2020, 17:14 GMT
Nice catch, so that's why all my vpns died a sudden and painful death today.
Comment by Christian Hesse (eworm) - Friday, 30 October 2020, 17:29 GMT
OpenVPN introduced a netlink interface that should supersede the method of running ip command from iproute2:
https://github.com/OpenVPN/openvpn/commit/c6542257019ba66a98b817d3b851621dd553f80a

Both services, openvpn-server@.service and openvpn-client@.service, have CAP_NET_ADMIN in CapabilityBoundingSet. So wondering why this does not work for you. Do you use modified services?
Comment by Christian Hesse (eworm) - Friday, 30 October 2020, 17:38 GMT
Also please give error messages to see what goes wrong.
Comment by Boris "Hundi Wam Wam" Digital (boris64) - Friday, 30 October 2020, 17:43 GMT
Hey Christian,

if you have the "iproute"-option set in the openvpn config, the service simply fails w/
"openvpn[117530]: Options error: Unrecognized option or missing or extra parameter(s) in whateverunameit.conf:32: iproute (2.5.0)"

Commenting the option and running as user seems to work so far afaik, but i haven't been able to test the unprivileged reconnect yet...
Comment by tom (archtom) - Friday, 30 October 2020, 20:53 GMT
@Christian: Thanks for looking into this. I´m getting the same Options error message that Boris gets.

What is the recommended way of running unprivileged without using iproute /usr/local/sbin/unpriv-ip?
Am I supposed to use these?
user openvpn
group openvpn
;iproute /usr/local/sbin/unpriv-ip

If I use it like this the service starts but I don't know if using CAP_NET_ADMIN helps with the problem mentioned in the openvpn and archwiki:
[...] It can also happen, however, that the OpenVPN server pushes updates to routes at runtime of the tunnel. A client with dropped privileges will be unable to perform the update and exit with an error. [...]
Using iproute /usr/local/sbin/unpriv-ip with the recommended unpriv-ip script helped working around that.

Thanks for further help and recommendations in advance.
Comment by Christian Hesse (eworm) - Friday, 30 October 2020, 21:01 GMT
With the new netlink interface the workaround is no longer required.
Even openvpn with dropped privileges can perform network changes if required.

If you can prove opposite please do. ;)
Comment by tom (archtom) - Friday, 30 October 2020, 21:22 GMT
Sorry I'm not an openvpn expert and I can't prove the opposite, but perhaps someone else can ;)

Only for avoiding misunderstandings. Are these the correct settings in version 2.5.0?

My openvpn user and group were created like this:
[[ $(getent group openvpn) ]] || groupadd openvpn
[[ $(getent passwd openvpn) ]] || useradd -d '/etc/openvpn' -c '' -M -g openvpn -G openvpn -s /usr/bin/nologin openvpn
passwd -l openvpn 2>&1 | grep -v "Signal 11"
# setting values according to nobody user
# -E -1 has to be set, if -E 0 is used, openvpn commands with unpriv-ip do no longer work as sudo complains about an expired account
chage -E -1 -I -1 -m -1 -M -1 -W -1 openvpn

and there is an entry in "/etc/sudoers":
%openvpn ALL = (root) NOPASSWD: /usr/bin/ip

1 I guess the sudoers entry is longer needed, right?

2 /usr/local/sbin/unpriv-ip can be deleted?

3 Shall the server config file entries be
user openvpn
group openvpn
;iproute /usr/local/sbin/unpriv-ip

or
user nobody
group nobody
;iproute /usr/local/sbin/unpriv-ip

or anything else?

Thanks for the help
Comment by ketsui (ketsui) - Tuesday, 03 November 2020, 07:26 GMT
I'm not sure if these warnings are bogus but after OpenVPN drops privileges, if I stop openvpn-client@.service I get these warnings in my journal:

SIGTERM received, sending exit notification to peer
Stopping OpenVPN tunnel for protonvpn...
net_route_v4_del: 89.38.99.188/32 via 192.168.0.1 dev [NULL] table 0 metric -1
sitnl_send: rtnl: generic error (-1): Operation not permitted
ERROR: Linux route delete command failed
net_route_v4_del: 0.0.0.0/1 via 10.24.0.1 dev [NULL] table 0 metric -1
sitnl_send: rtnl: generic error (-1): Operation not permitted
ERROR: Linux route delete command failed
net_route_v4_del: 128.0.0.0/1 via 10.24.0.1 dev [NULL] table 0 metric -1

I made sure that the unit file is reverted after removing the iproute2 workaround.
Comment by ketsui (ketsui) - Tuesday, 03 November 2020, 12:13 GMT
>Both services, openvpn-server@.service and openvpn-client@.service, have CAP_NET_ADMIN in CapabilityBoundingSet.
According to this blog[0], the effective set is what determines whether or not the kernel allows a process to do a system call:

$ grep Cap /proc/$(pidof openvpn)/status
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 00000000000470c2
CapAmb: 0000000000000000

So the process actually has no effective capabilities, running an strace also shows[1] that OpenVPN never actually sets its capabilities either with capset(2) or cap_set_proc(3) so the kernel denies its attempt to delete routes after it drops root privileges.

[0] https://blog.container-solutions.com/linux-capabilities-why-they-exist-and-how-they-work (The effective set is the set that is checked by the kernel to allow or disallow calls.)
[1] http://ix.io/2CS5
Comment by Christian Hesse (eworm) - Tuesday, 03 November 2020, 12:47 GMT
I've expected this to be too easy - and it does not work as delivered by upstream out of the box.

The process has the capabilities when running as root (at startup), but capabilities are lost when privileges are dropped by switching user.

The netlink interface is really nice and I would like to keep it instead of reverting to using the ip command from iproute2.

So what we need is a dedicated user, then we can add that into the systemd unit files (alternatively set DynamicUser=yes) and change CapabilityBoundingSet to AmbientCapabilities. The process is then started as unprivileged user but is granted extra capabilities. This works just fine for starting and stopping the service, probably the same for intermittent actions.
The downside: Scripts (up, tls-verify, ipchange, client-connect, route-up, route-pre-down, client-disconnect, down, learn-address, auth-user-pass-verify) are run with that user as well and need a workaround if elevated permissions are required. (Scripts run after dropping privileges required a workaround before already...)

Not really sure how to proceed from here... Wondering if the latter will show any more breakage.
Comment by Christian Hesse (eworm) - Tuesday, 03 November 2020, 12:58 GMT
Anybody wants to give this test build a try?
https://pkgbuild.com/~eworm/openvpn-2.5.0-1.1-x86_64.pkg.tar.zst
Comment by ketsui (ketsui) - Tuesday, 03 November 2020, 13:28 GMT
I ran into a few issues with that version, first the process failed to read my credential file (auth-user-pass cred) that is owned by the user openvpn probably because of the DynamicUser? After I removed that from my .conf file instead of asking for my credentials on the terminal OpenVPN just exits after complaining about read-only filesystem so it doesn't work either. Also instead of replacing CapabilityBoundingSet with AmbientCapabilities I suggest you keep them both because CapabilityBoundingSet is supposed to be a safeguard preventing the process from gaining unnecessary capabilities.

What worked for me in the end is removing both user openvpn and group openvpn from my .conf file so OpenVPN doesn't call setuid(2) and adding AmbientCapabilities, User=openvpn and Group=openvpn to the .service file, now the routes are properly deleted when I stop OpenVPN.

SIGTERM received, sending exit notification to peer
Stopping OpenVPN tunnel for protonvpn...
net_route_v4_del: 190.2.132.214/32 via 192.168.0.1 dev [NULL] table 0 metric -1
net_route_v4_del: 0.0.0.0/1 via 10.20.0.1 dev [NULL] table 0 metric -1
net_route_v4_del: 128.0.0.0/1 via 10.20.0.1 dev [NULL] table 0 metric -1
Closing TUN/TAP interface
net_addr_v4_del: 10.20.0.56 dev tun0
SIGTERM[soft,exit-with-notification] received, process exiting
openvpn-client@protonvpn.service: Succeeded.
Stopped OpenVPN tunnel for protonvpn.
Comment by Christian Hesse (eworm) - Wednesday, 04 November 2020, 09:43 GMT
Probably it's better not to use dynamic users...

https://pkgbuild.com/~eworm/openvpn-2.5.0-1.2-x86_64.pkg.tar.zst

Switched to creating user with sysusers, added upgrade notice.
Comment by Christian Hesse (eworm) - Thursday, 05 November 2020, 20:52 GMT
No more comments... Is anybody happy with that build?
Comment by ketsui (ketsui) - Friday, 06 November 2020, 03:02 GMT
Works for me, this line is a bit vague though:
"The configuration should no longer drop privileges and running scripts may require a workaround to elevate privileges."
I suggest:
"The 'user' and 'group' directives in OpenVPN's configuration file are no longer needed and scripts that require elevated privileges may need a workaround."

Loading...