Arch Linux

I encountered something similar. An EC2 instance running Arch Linux had been refusing SSH connections since a few hours ago. I had to reboot it. Having examed the log (journalctl -b -1), it seemed that someone was trying to brute-force the instance and triggered the permanent deactivation of sshd.socket, thus refusing incoming connections.

systemd[1]: sshd.socket: Trigger limit hit, refusing further activation.
systemd[1]: sshd.socket: Failed with result 'trigger-limit-hit'.

A workaround is to enable and start sshd.service (not sshd.socket).

I've always disliked socket activation. The OpenSSH daemon is serious and robust enough to bind to a port and not be run through inetd. Let me however give other devs who used socket activation in the past the chance to reply to this bug report. If nobody chimes in to support socket activation in the coming days, I'll remove it from our package.

I use socket activation for sshd. My systems have extra security, though: My netfilter firewall limits ssh access.

table inet filter {
chain input {
...
tcp dport ssh flow table ssh { ip saddr limit rate 5/minute } accept
...
}
}

Opening for unlimited brute force may be a bad idea anyway. :-p

If you remove the support... Will you provide an automated update path? Just removing the socket unit may introduce even more havoc, given users do not read upgrade messages.

Yeah, I'm not sure it has much value. sshd needs root perms anyway, so doesn't really benefit from systemd managing the socket.

I think you should port existing setups in post_upgrade so people don't suddenly lose ssh after a reboot, *and* throw out a message so users with weird setups that alter sshd.socket can react.

Could anyone interested look at what I've just commited to SVN?

I've done some testing and experienced no problem with the automated migration (from sshd.socket to sshd.service), but I'd value input to confirm this and/or suggestions on how to improve it. Cheers.

There is the proposed updated package (see the install file in particular): https://git.archlinux.org/svntogit/packages.git/tree/trunk?h=packages/openssh

Your condition on whether or not sshd socket is enabled could be:

systemctl is-enabled -q sshd.socket

And disable/reload/enable:

systemctl disable --now sshd.socket
systemctl daemon-reload
systemctl enable --now sshd.service

Not sure what the check for /etc/systemd/system/sshd\@.service is supposed to do. Can you elaborate?
How about checking for config in /etc/systemd/system/sshd.socket.d/?

Thanks eworm.
I wanted to check if the user was using a custom sshd@.service file placed under /etc/systemd/system which would shadow the one we ship under /usr/lib/systemd/system.
I have no idea what's supposed to be under /etc/systemd/system/sshd.socket.d could you explain?
Cheers.

(That's also why I checked the target of the sockets.target.wants/sshd.socket link, to make sure the user was using the stock one we ship and not a custom one.)

The directory can hold configuration override snippets, so placing /etc/systemd/system/sshd.socket.d/port.conf with this content:

[Socket]
ListenStream=
ListenStream=2222

... makes systemd listen on port 2222 only. We would have to check for: /etc/systemd/system/sshd{,@}.{service,socket}{,.d/}
Additionally changes in /etc/ssh/sshd_config could make the system unreachable as well.

How about reverting the changes, keeping sshd.socket, but modifying it? From systemd.socket(5):

> TriggerLimitIntervalSec=, TriggerLimitBurst=
> Configures a limit on how often this socket unit my be activated within a specific time interval. [...] Set either to 0 to disable any form of trigger rate limiting. [...]

So adding TriggerLimitIntervalSec=0 and/or TriggerLimitBurst=0 to sshd.socket should be sufficient to fix this.

We could do that but I liked the idea of sunsetting sshd.socket.

Another option would be to look for any symlink to /usr/lib/systemd/system/sshd.socket under /etc/systemd and replace them by the actual content of that file. Also do the same for sshd@.service. Then we can remove the /usr/lib files from our package. This way, people who were using socket activation keep a working configuration but it becomes unsupported by our package.

No worries: if we can't all agree on what a good solution is, I won't change anything.

Some more thoughts on this... Of course spawning a new process is more expensive than forking, so this is still more prone to being exploited for denial of service by high load on the system. The socket will not go to failed state and stay available, though.

How about disabling the rate limiting for the socket, then adding something like this in install script if socket is enabled:

post_upgrade() {
if (( $(vercmp $2 8.0p1-3) < 0 )); then
if systemctl is-enabled -q sshd.socket; then
echo '==> To mitigate denial of service rate limiting in sshd.socket has been'
echo '==> disabled. It's still advised to migrate to sshd.service and/or limit'
echo '==> rate with firewall.'
fi
fi
}

Ha, just had a very similar idea to yours. :D

pre_upgrade() {
if (( $(vercmp $2 8.0p1-3) < 0 )); then
if systemctl is-enabled -q sshd.socket; then
if [[ ! -e /etc/systemd/system/sshd.socket ]]; then
echo '==> The package drops sshd.socket, the unit is move to /etc/systemd/system/.'
echo '==> Please consider migrating to sshd.service to mitigate denial of service.'
install -m0644 /usr/lib/systemd/system/sshd.socket /etc/systemd/system/sshd.socket
systemctl reenable sshd.socket
fi
fi
fi
}

(Not tested...)

> My netfilter firewall limits ssh access.

If I understand it correctly, I would not recommend this approach as a mitigation to this problem:

1. Rapid connection attempts from different IPs (DDoS) will still trigger this problem. Most bots which scan / brute-force SSH operate from large botnets. A per-IP rate limit will thus be completely ineffectual in this situation.
2. A per-IP rate limit affects valid use cases. For example, when using git over ssh, each fetch and push operation will initiate a new connection. Some remote filesystem software (e.g. TRAMP) can also spawn multiple connections.

Edit: I pointed this out because of the suggestion to show the message "It's still advised to migrate to sshd.service and/or limit rate with firewall" to users. Suggesting an incomplete solution to users and making them fill in the blanks might be suboptimal.

> 1. Rapid connection attempts from different IPs (DDoS) will still trigger this problem. Most bots which scan / brute-force SSH operate from large botnets. A per-IP rate limit will thus be completely ineffectual in this situation.

There's not the one and only solution available. Firewalls can help a lot, but let's not discuss this in detail and focus on the specific topic.

> 2. A per-IP rate limit affects valid use cases. For example, when using git over ssh, each fetch and push operation will initiate a new connection. Some remote filesystem software (e.g. TRAMP) can also spawn multiple connections.

That's why I use connection/session sharing. ;) See 'ControlMaster' in ssh_config(5). But again: Let's focus on the topic.

Here's a new proposal: the install file simply copies the sshd.socket and sshd\@.service files to /etc/systemd/system if it determines they are being used. Could you please go over it and let me know if I missed something? I'd like to push this to [testing] in the next few days. Thanks again.

https://git.archlinux.org/svntogit/packages.git/tree/trunk/install?h=packages/openssh