FS#65517 - [openssh] upgrade to 8.2p1-1 breaks logins

Attached to Project: Arch Linux
Opened by Erich Eckner (deepthought) - Sunday, 16 February 2020, 08:00 GMT
Last edited by Gaetan Bisson (vesath) - Monday, 17 February 2020, 01:34 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Gaetan Bisson (vesath)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 11
Private No

Details

Description:

I recently upgraded all packages, amongst which:
upgraded openssh (8.1p1-2 -> 8.2p1-1)

This broke remote logins via ssh:
kex_exchange_identification: read: Connection reset by peer

(This happened to me with the same version on archlinux, archlinux32 and archlinuxarm)
Downgrading openssh to any earlier verions fixed it.

Additional info:
* package version(s)
openssh 8.2p1-1

Steps to reproduce:
> pacman -Syu
# from remote
> ssh the-other.host.name
(there's probably more to this, but IDK)
One thing, that all machines have in common, is that I mainly authenticate via key.
This task depends upon

Closed by  Gaetan Bisson (vesath)
Monday, 17 February 2020, 01:34 GMT
Reason for closing:  Fixed
Additional comments about closing:  openssh-8.2p1-3 in [core]
Comment by Erik Lauritsen (eriklauritsen) - Sunday, 16 February 2020, 08:20 GMT
I can confirm this.

I have just upgraded a remote host and cannot login any longer.

kex_exchange_identification: read: Connection reset by peer
Comment by Erich Eckner (deepthought) - Sunday, 16 February 2020, 08:31 GMT
my private key is ssh-rsa - possibly the reason?
https://www.openssh.com/txt/release-8.2
Comment by Erik Lauritsen (eriklauritsen) - Sunday, 16 February 2020, 08:32 GMT
I think this upgrade should have been tested thoroughly before being implemented as upstream has made quite a few changes: https://www.openssh.com/releasenotes.html

I suspect a lot of people will be logged out of their remote boxes.
Comment by Erik Lauritsen (eriklauritsen) - Sunday, 16 February 2020, 08:35 GMT
The problem is not only related to key logins.
Comment by Erik Lauritsen (eriklauritsen) - Sunday, 16 February 2020, 08:38 GMT
I have logged in to an OpenBSD box and then from there tried to login to the broken remote box running Arch, without a key, but with password:

kex_exchange_identification: read: Connection reset by peer

I'm gonna try have the remote Arch box rebooted once I get a chance.
Comment by Andreas Baumann (andreas_baumann) - Sunday, 16 February 2020, 08:38 GMT
Ditto here:

The 8.2 sshd servers tell me:

Feb 16 08:44:33 server sshd[3104261]: fatal: recv_rexec_state: buffer error: incomplete message

Comment by Erich Eckner (deepthought) - Sunday, 16 February 2020, 08:39 GMT
eriklauritsen: Do you *have* a key on the openbsd box? That will get send along and if does not match the prerequisites regarding algorithms, it will fail, too. Move it out of the way for a test.
Comment by Erich Eckner (deepthought) - Sunday, 16 February 2020, 08:45 GMT
restarting sshd solves the issue for me
Comment by Erik Lauritsen (eriklauritsen) - Sunday, 16 February 2020, 08:50 GMT
@Erich Eckner, no there is no key on the OpenBSD box. I suspect the server needs to be restarted.

Upgrading this package should make people aware with a message.
Comment by Andreas Baumann (andreas_baumann) - Sunday, 16 February 2020, 08:55 GMT
restarting sshd seems to help
Comment by Andreas Baumann (andreas_baumann) - Sunday, 16 February 2020, 08:56 GMT
Also, the keygen script on the server still generates ssh_host_rsa_key and ssh_host_rsa_key.pub and hands them out
to old 8.1 ssh clients.

/etc/ssh/ssh_host_rsa* should maybe be deleted or people should be made aware that they have to be removed
(and eventually HostKey to be adapted in /etc/ssh/sshd_config). OTOH, there might be reasons to keep the rsa
keys for other distros with older ssh clients to still be able to log in).
Comment by Erik Lauritsen (eriklauritsen) - Sunday, 16 February 2020, 08:57 GMT
Yes. Restarting sshd solves it.

I just tried this on another box in the building with physical access, same problem.

Rebooted the box manually solved the problem.
Comment by Andreas Baumann (andreas_baumann) - Sunday, 16 February 2020, 09:35 GMT
Aha. ssh-rsa is still accepted for the time being. But it's good to change it anyways to
a safer one. So it's also consistent to keep the rsa server keys..
Comment by B B (0xbb) - Sunday, 16 February 2020, 11:36 GMT
I can confirm that restarting sshd is required after the update.
Comment by kyak (kyak) - Sunday, 16 February 2020, 11:37 GMT
Same problem. Works fine after restarting the remote server. This should not have happened - not without the news on the front page.
Comment by Serge (SR-G) - Sunday, 16 February 2020, 13:32 GMT
Same problem ... this is the worst problem i've seen in a while.

I've chosen to downgrade to previous version (pacman -U /var/cache/pacman/pkg/openssh-8.1p1-4-x86_64.pkg.tar.zst) (which solved temporarily the issue, without additional restart), as it is still quite not clear :
- if the problem is about RSA keys (i'm also using RSA keys)
- or if a restart of SSHD is enough
- or if a restart of the server is mandatory (which is not possible in my situation)
Comment by Erik Lauritsen (eriklauritsen) - Sunday, 16 February 2020, 15:45 GMT
I too am having difficulty figuring out what the exact problem is. I have been going over the upstream changelog, but it seems like a restart of sshd is enough.

We need this issue to go on the frontpage of Arch ASAP, lots of people risk being logged out of remote systems.
Comment by kyak (kyak) - Sunday, 16 February 2020, 16:00 GMT
Some more info. The following can be seen in server logs (before sshd had been restarted) when the client tries to login (and fails):

sshd[23423]: fatal: recv_rexec_state: buffer error: incomplete message
Comment by Joerg Gollnick (wurzelbenutzer) - Sunday, 16 February 2020, 20:32 GMT
same issue here need to restart remote server.
Comment by Chih-Hsuan Yen (yan12125) - Monday, 17 February 2020, 00:43 GMT
If I understand sshd sources correctly, this is the key commit: https://github.com/openssh/openssh-portable/commit/c2bd7f74b0e0f3a3ee9d19ac549e6ba89013abaf. Internal protocol between parent and child sshd is changed, so the parent sshd (version 8.1 before restarting sshd) cannot run child sshd (version 8.2 after pacman -Syu). I believe restarting sshd is needed and sufficient.

The linked commit above is about the new "Include" feature, not related to deprecated keys, so login via a password is also broken.
Comment by Gaetan Bisson (vesath) - Monday, 17 February 2020, 01:34 GMT
You appear to be correct, Chih-Hsuan.

I deeply regret that this issue had not been identified while openssh-8.2p1-1 was in [testing] or even earlier. I have just pushed a new openssh-8.2p1-3 package which will automatically restart existing sshd.service. I will also post a news announcement just in case.

My apologies for any inconvenience.

Loading...