FS#62248 - [openssh] Using socket activation can result in denial of service

Attached to Project: Arch Linux
Opened by Vladimir Panteleev (CyberShadow) - Saturday, 06 April 2019, 00:06 GMT
Last edited by Gaetan Bisson (vesath) - Friday, 20 September 2019, 23:51 GMT
Task Type Feature Request
Category Packages: Core
Status Closed
Assigned To Gaetan Bisson (vesath)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 2
Private No

Details

Too many connections to an sshd server, configured using socket activation (sshd.socket as per https://wiki.archlinux.org/index.php/OpenSSH#Daemon_management), can cause the socket to be disabled permanently ("sshd.socket: Trigger limit hit, refusing further activation.").

Excerpt from the journal for an example occurrence attached.

sshd.socket and sshd@.service seem to be supplied by Arch, so this would be a distribution-specific issue.
https://git.archlinux.org/svntogit/packages.git/tree/trunk?h=packages/openssh

Observed with openssh 7.9p1-1. No pertinent configuration changes, other than enabling sshd.socket.
This task depends upon

Closed by  Gaetan Bisson (vesath)
Friday, 20 September 2019, 23:51 GMT
Reason for closing:  Implemented
Additional comments about closing:  openssh-8.0p1-3 in [testing]
Comment by Zhuoyun Wei (wzyboy) - Sunday, 14 April 2019, 08:58 GMT
I encountered something similar. An EC2 instance running Arch Linux had been refusing SSH connections since a few hours ago. I had to reboot it. Having examed the log (journalctl -b -1), it seemed that someone was trying to brute-force the instance and triggered the permanent deactivation of sshd.socket, thus refusing incoming connections.

systemd[1]: sshd.socket: Trigger limit hit, refusing further activation.
systemd[1]: sshd.socket: Failed with result 'trigger-limit-hit'.

A workaround is to enable and start sshd.service (not sshd.socket).
Comment by Gaetan Bisson (vesath) - Monday, 09 September 2019, 08:40 GMT
I've always disliked socket activation. The OpenSSH daemon is serious and robust enough to bind to a port and not be run through inetd. Let me however give other devs who used socket activation in the past the chance to reply to this bug report. If nobody chimes in to support socket activation in the coming days, I'll remove it from our package.
Comment by Christian Hesse (eworm) - Monday, 09 September 2019, 08:51 GMT
I use socket activation for sshd. My systems have extra security, though: My netfilter firewall limits ssh access.

table inet filter {
chain input {
...
tcp dport ssh flow table ssh { ip saddr limit rate 5/minute } accept
...
}
}

Opening for unlimited brute force may be a bad idea anyway. :-p

If you remove the support... Will you provide an automated update path? Just removing the socket unit may introduce even more havoc, given users do not read upgrade messages.
Comment by Jan Alexander Steffens (heftig) - Monday, 09 September 2019, 08:52 GMT
Yeah, I'm not sure it has much value. sshd needs root perms anyway, so doesn't really benefit from systemd managing the socket.

I think you should port existing setups in post_upgrade so people don't suddenly lose ssh after a reboot, *and* throw out a message so users with weird setups that alter sshd.socket can react.
Comment by Gaetan Bisson (vesath) - Monday, 16 September 2019, 21:49 GMT
Could anyone interested look at what I've just commited to SVN?

I've done some testing and experienced no problem with the automated migration (from sshd.socket to sshd.service), but I'd value input to confirm this and/or suggestions on how to improve it. Cheers.
Comment by Gaetan Bisson (vesath) - Monday, 16 September 2019, 21:52 GMT
There is the proposed updated package (see the install file in particular): https://git.archlinux.org/svntogit/packages.git/tree/trunk?h=packages/openssh
Comment by Christian Hesse (eworm) - Monday, 16 September 2019, 22:58 GMT
Your condition on whether or not sshd socket is enabled could be:

systemctl is-enabled -q sshd.socket

And disable/reload/enable:

systemctl disable --now sshd.socket
systemctl daemon-reload
systemctl enable --now sshd.service

Not sure what the check for /etc/systemd/system/sshd\@.service is supposed to do. Can you elaborate?
How about checking for config in /etc/systemd/system/sshd.socket.d/?
Comment by Gaetan Bisson (vesath) - Monday, 16 September 2019, 23:05 GMT
Thanks eworm.
I wanted to check if the user was using a custom sshd@.service file placed under /etc/systemd/system which would shadow the one we ship under /usr/lib/systemd/system.
I have no idea what's supposed to be under /etc/systemd/system/sshd.socket.d could you explain?
Cheers.
Comment by Gaetan Bisson (vesath) - Monday, 16 September 2019, 23:08 GMT
(That's also why I checked the target of the sockets.target.wants/sshd.socket link, to make sure the user was using the stock one we ship and not a custom one.)
Comment by Christian Hesse (eworm) - Tuesday, 17 September 2019, 00:01 GMT
The directory can hold configuration override snippets, so placing /etc/systemd/system/sshd.socket.d/port.conf with this content:

[Socket]
ListenStream=
ListenStream=2222

... makes systemd listen on port 2222 only. We would have to check for: /etc/systemd/system/sshd{,@}.{service,socket}{,.d/}
Additionally changes in /etc/ssh/sshd_config could make the system unreachable as well.

How about reverting the changes, keeping sshd.socket, but modifying it? From systemd.socket(5):

> TriggerLimitIntervalSec=, TriggerLimitBurst=
> Configures a limit on how often this socket unit my be activated within a specific time interval. [...] Set either to 0 to disable any form of trigger rate limiting. [...]

So adding TriggerLimitIntervalSec=0 and/or TriggerLimitBurst=0 to sshd.socket should be sufficient to fix this.
Comment by Gaetan Bisson (vesath) - Tuesday, 17 September 2019, 07:27 GMT
We could do that but I liked the idea of sunsetting sshd.socket.

Another option would be to look for any symlink to /usr/lib/systemd/system/sshd.socket under /etc/systemd and replace them by the actual content of that file. Also do the same for sshd@.service. Then we can remove the /usr/lib files from our package. This way, people who were using socket activation keep a working configuration but it becomes unsupported by our package.

No worries: if we can't all agree on what a good solution is, I won't change anything.
Comment by Christian Hesse (eworm) - Tuesday, 17 September 2019, 07:40 GMT
Some more thoughts on this... Of course spawning a new process is more expensive than forking, so this is still more prone to being exploited for denial of service by high load on the system. The socket will not go to failed state and stay available, though.

How about disabling the rate limiting for the socket, then adding something like this in install script if socket is enabled:

post_upgrade() {
if (( $(vercmp $2 8.0p1-3) < 0 )); then
if systemctl is-enabled -q sshd.socket; then
echo '==> To mitigate denial of service rate limiting in sshd.socket has been'
echo '==> disabled. It's still advised to migrate to sshd.service and/or limit'
echo '==> rate with firewall.'
fi
fi
}
Comment by Christian Hesse (eworm) - Tuesday, 17 September 2019, 07:51 GMT
Ha, just had a very similar idea to yours. :D

pre_upgrade() {
if (( $(vercmp $2 8.0p1-3) < 0 )); then
if systemctl is-enabled -q sshd.socket; then
if [[ ! -e /etc/systemd/system/sshd.socket ]]; then
echo '==> The package drops sshd.socket, the unit is move to /etc/systemd/system/.'
echo '==> Please consider migrating to sshd.service to mitigate denial of service.'
install -m0644 /usr/lib/systemd/system/sshd.socket /etc/systemd/system/sshd.socket
systemctl reenable sshd.socket
fi
fi
fi
}

(Not tested...)
Comment by Vladimir Panteleev (CyberShadow) - Tuesday, 17 September 2019, 13:45 GMT
> My netfilter firewall limits ssh access.

If I understand it correctly, I would not recommend this approach as a mitigation to this problem:

1. Rapid connection attempts from different IPs (DDoS) will still trigger this problem. Most bots which scan / brute-force SSH operate from large botnets. A per-IP rate limit will thus be completely ineffectual in this situation.
2. A per-IP rate limit affects valid use cases. For example, when using git over ssh, each fetch and push operation will initiate a new connection. Some remote filesystem software (e.g. TRAMP) can also spawn multiple connections.

Edit: I pointed this out because of the suggestion to show the message "It's still advised to migrate to sshd.service and/or limit rate with firewall" to users. Suggesting an incomplete solution to users and making them fill in the blanks might be suboptimal.
Comment by Christian Hesse (eworm) - Tuesday, 17 September 2019, 14:09 GMT
> 1. Rapid connection attempts from different IPs (DDoS) will still trigger this problem. Most bots which scan / brute-force SSH operate from large botnets. A per-IP rate limit will thus be completely ineffectual in this situation.

There's not the one and only solution available. Firewalls can help a lot, but let's not discuss this in detail and focus on the specific topic.

> 2. A per-IP rate limit affects valid use cases. For example, when using git over ssh, each fetch and push operation will initiate a new connection. Some remote filesystem software (e.g. TRAMP) can also spawn multiple connections.

That's why I use connection/session sharing. ;) See 'ControlMaster' in ssh_config(5). But again: Let's focus on the topic.
Comment by Gaetan Bisson (vesath) - Tuesday, 17 September 2019, 23:36 GMT
Here's a new proposal: the install file simply copies the sshd.socket and sshd\@.service files to /etc/systemd/system if it determines they are being used. Could you please go over it and let me know if I missed something? I'd like to push this to [testing] in the next few days. Thanks again.

https://git.archlinux.org/svntogit/packages.git/tree/trunk/install?h=packages/openssh

Loading...