FS#22897 - [openssh] SSH is falling to connect to some server Read from socket failed: Connection reset by peer

Attached to Project: Arch Linux
Opened by Ronan Arraes Jardim Chagas (Ronis_BR) - Monday, 14 February 2011, 22:22 GMT
Last edited by Gaetan Bisson (vesath) - Saturday, 09 April 2011, 10:46 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Gaetan Bisson (vesath)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 3
Private No

Details

Description:

It seems that Archlinux is having problem to connect to some servers using ssh.
I have at my internal network three devices:

1) A bleeding edge Arch Linux;
2) An one year old Arch Linux;
3) An android cellphone.

When I try to connect to my remote server, 1) hangs and 2) and 3) connects perfectly.
Btw, I can connect 1) to a local ssh server.

The server log isn't showing anything about the access and the client debug log is:

OpenSSH_5.8p1, OpenSSL 1.0.0d 8 Feb 2011
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to ronan-ita [161.24.19.187] port 22.
debug1: Connection established.
debug1: identity file /home/ronan/.ssh/id_rsa type -1
debug1: identity file /home/ronan/.ssh/id_rsa-cert type -1
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/ronan/.ssh/id_dsa" as a RSA1 public key
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'Proc-Type:'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'DEK-Info:'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /home/ronan/.ssh/id_dsa type 2
debug1: identity file /home/ronan/.ssh/id_dsa-cert type -1
debug1: identity file /home/ronan/.ssh/id_ecdsa type -1
debug1: identity file /home/ronan/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.6
debug1: match: OpenSSH_5.6 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "ronan-ita" from file "/home/ronan/.ssh/known_hosts"
debug3: load_hostkeys: loaded 0 keys
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none
debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 zlib@openssh.com
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 zlib@openssh.com
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
Read from socket failed: Connection reset by peer


Additional info:
* OpenSSH_5.8p1, OpenSSL 1.0.0d 8 Feb 2011
* Kernel 2.6.37-ARCH
* At #archlinux channel on freenode, bencahill and fsckd verified that the connection to my server works on Ubuntu and Debian. Also, bencahill and sledgehammer verified that bleeding edge Archlinux hangs when try to connect to my server, as I saw here.
* soupcan is also having the same problem: https://bbs.archlinux.org/viewtopic.php?pid=892227#p892227


Steps to reproduce:
* Update Archlinux
* Try to connect to an external ssh server (not sure if the problem is with all ssh servers)
This task depends upon

Closed by  Gaetan Bisson (vesath)
Saturday, 09 April 2011, 10:46 GMT
Reason for closing:  Upstream
Additional comments about closing:  See last comment.
Comment by Gaetan Bisson (vesath) - Monday, 14 February 2011, 23:23 GMT
What SSH software is your mysterious server running?
Here, for instance, I have no problem connecting to RHEL OpenSSH_4.3p2, FreeBSD OpenSSH_4.5p1, Fedora OpenSSH_4.7p1, Debian OpenSSH_5.1p1, etc.

On the client side, does it solve your problem to downgrade openssh to 5.7p1? 5.6p1?
It would really help to isolate the problem if we knew whether *on the same machine* 5.8p1 fails while another version works.
Comment by Gaetan Bisson (vesath) - Monday, 14 February 2011, 23:33 GMT
Other ideas:

Are you using the default /etc/ssh_config? Do you have a ~/.ssh/config?

Maybe also try downgrading openssl.
Comment by Ronan Arraes Jardim Chagas (Ronis_BR) - Monday, 14 February 2011, 23:34 GMT
The server is SSH-2.0-OpenSSH_5.6.
With OpenSSH 5.7 I also see the problem, but it is gone with OpenSSH 5.6p1-2
Comment by Ronan Arraes Jardim Chagas (Ronis_BR) - Monday, 14 February 2011, 23:35 GMT
I didn't need to downgrade OpenSSL and I'm using default ssh_config and I don't have ~/.ssh/config
By now, I'll use OpenSSH 5.6p1 :) Thanks for the advice!
Comment by Gaetan Bisson (vesath) - Monday, 14 February 2011, 23:43 GMT
Using 5.6p1 is okay temporarily but I'd like to get to the bottom of this. :)

Can you be more specific about the SSH server? What is its distribution and package version?
Comment by Ronan Arraes Jardim Chagas (Ronis_BR) - Monday, 14 February 2011, 23:49 GMT
The SSH server is an Archlinux that hasn't be updated for a while. So, it still has OpenSSH 5.6p1.
Comment by Gaetan Bisson (vesath) - Monday, 14 February 2011, 23:58 GMT
Here I have no problem connecting to a 5.6p1 server using a 5.8p1 client, so your setup must have some specificity...
Is the server's /etc/sshd_config the default one? Can you do a "ls -la /etc/ssh/" on the server so that I can see which host keys it has?
Comment by Ronan Arraes Jardim Chagas (Ronis_BR) - Tuesday, 15 February 2011, 00:06 GMT
ls /etc/ssh/
moduli ssh_host_dsa_key ssh_host_key.pub
ssh_config ssh_host_dsa_key.pub ssh_host_rsa_key
sshd_config ssh_host_key ssh_host_rsa_key.pub

and the sshd_config is the default one

At the client, I have already tried with default ssh_config.
Comment by Gaetan Bisson (vesath) - Tuesday, 15 February 2011, 10:29 GMT
So I did a binary search and the code from

export CVSROOT=anoncvs@anoncvs.mindrot.org:/cvs
export CVS_RSH=/usr/bin/ssh
cvs get -D 2010-09-01 openssh

produces a client which cannot connect to your server (the client initiates key-exchange and gets no response from the server), while replacing 2010-09-01 by 2010-08-31 does.

Now this could be an upstream bug but there is certainly something weird about your server (since I can connect to other 5.6p1 sshd alright).
- Can you try restarting sshd?
- Can you try (re)installing openssh-5.6p1-2 and then restarting sshd?
- Can you try (re)installing openssh-5.6p1-1 and then restarting sshd?
Comment by Ronan Arraes Jardim Chagas (Ronis_BR) - Tuesday, 15 February 2011, 10:50 GMT
Well, I have already done all these steps without any success...
Now, I'll try to update the server to bleeding edge ArchLinux to see if my 5.8 client can connect to 5.8 server.
Comment by Gaetan Bisson (vesath) - Tuesday, 15 February 2011, 10:59 GMT
Regardless of this issue, I'd like to recall that it's better practice to update Arch machines at least once a month. :)

Let me know if your issue disappears with 5.8p1; if it does and the other person who had a server with this issue does not give us insight by then, I will close this bug since we won't have any way of getting a diagnosis.
Comment by Ronan Arraes Jardim Chagas (Ronis_BR) - Tuesday, 15 February 2011, 15:15 GMT
Hi fellow,

I have upgraded server ssh to 5.8p1 (default configuration) and the problem still exists.
I can not log with ssh client 5.8p1 but I can with ssh 5.6p1 client.
Comment by Leonid Isaev (lisaev) - Wednesday, 16 February 2011, 20:32 GMT
Ronan, there is definitely something wrong with your config, because I have pretty stable (like 12 hrs long stable) ssh tunnels between two up-to-date arch installations. Can you run "sshd -d" on the server and "ssh -vv" on the cliend and collect both outputs? Also, when you reinstalled ssh, did you check that /etc/ssh actually gets removed?
Comment by Petar Anastasov (parky6) - Thursday, 17 February 2011, 14:52 GMT

Hi,

I have a problem connecting to HP iLO with current version: openssh 5.8p1.
I have compiled from source 5.6p1 5.7p1 and 5.8p1 with no any options passed to ./configure but prefix. The result is attached

Comment by Gaetan Bisson (vesath) - Thursday, 17 February 2011, 15:20 GMT
We already established that there are certain servers to which 5.6p1 clients can connect perfectly well while 5.8p1 cannot.
What we now need to find out is **what makes these servers special** since most servers out there obviously do not have this issue.
(Roman's server is affected, he upgraded sshd from 5.6p1 to 5.8p1 and the issue remains, so this is most likely related to the configuration of sshd or its interaction with another package.)

Can someone with access to an affected server do what Leonid suggested ("sshd -d" to get debug output from the server)?
Comment by Ronan Arraes Jardim Chagas (Ronis_BR) - Thursday, 17 February 2011, 16:31 GMT
Hi Leonid,

Yes I can, but just on Monday, when I'll have physical access to the computer.
Thanks!
Comment by Petar Anastasov (parky6) - Tuesday, 22 February 2011, 09:26 GMT
Hi all,

I found a solution and the reason is new default HostKeyAlgorithms, introduced in 5.7. When I remove elliptic curve cryptography modes from the list, I'm able to connect again

in my ~/.ssh/config :

HostKeyAlgorithms ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,ssh-dss

You can see the dafault order in ssh_config manual page
Comment by Gaetan Bisson (vesath) - Tuesday, 22 February 2011, 10:28 GMT
Thanks Petar for this workaround.
However, lots of us use the default HostKeyAlgorithms of 5.8p1 and can connect perfectly well to various kinds of SSH servers, so we need to dig deeper if we want to find the underlying cause to this problem.

Edit: For other workarounds and debugging information, see: http://lists.mindrot.org/pipermail/openssh-unix-dev/2011-February/thread.html#29361
Comment by Ronan Arraes Jardim Chagas (Ronis_BR) - Tuesday, 22 February 2011, 12:26 GMT
Hi Petar,

OpenSSH 5.8 is working here with your suggestion!
I'll use it as a workaround until we figured out why this problem is affecting just some servers.

I have a interesting thing here:

If I connect by VNC on server and bind that ports:

ssh -R 2222:localhost:22 <USER>@<MY IP> -p <MY LAPTOP SSH SERVER PORT>

than I'm able to connect to my server from my laptop by:

ssh localhost -p 2222

using OpenSSH 5.8p1.

Can this bring new information?
Comment by Gaetan Bisson (vesath) - Tuesday, 22 February 2011, 13:09 GMT
Ronan, could you post this information as a reply on the thread that I mentioned? I am sure it would help upstream in their diagnosis.
Comment by Leonid Isaev (lisaev) - Tuesday, 22 February 2011, 16:16 GMT
> I found a solution and the reason is new default HostKeyAlgorithms, introduced in 5.7. When I remove elliptic curve
> cryptography modes from the list, I'm able to connect again

Are you saying that the relevant files are /etc/ssh/{ssh_host_ecdsa_key,ssh_host_ecdsa_key.pub}? Can you post the output of ls -lt /etc/ssh?

EDIT: and when ssh 5.7 was installed from pacman.log...

What I would do, is to remove openssh on the server. Then, remove /etc/ssh/keys*, then install openssh back and regenerate all the host keys... Otherwise, this is some black magic :)
Comment by Petar Anastasov (parky6) - Wednesday, 23 February 2011, 06:40 GMT

Hi Leonid,

Yes, those files were generated when openssh is upgraded to 5.7p1:
from pacman.log:
[2011-01-31 07:00] upgraded openssh (5.6p1-2 -> 5.7p1-2)

ls -l /etc/ssh/{ssh_host_ecdsa_key,ssh_host_ecdsa_key.pub} :
-rw------- 1 root root 227 Jan 31 07:03 /etc/ssh/ssh_host_ecdsa_key
-rw-r--r-- 1 root root 176 Jan 31 07:03 /etc/ssh/ssh_host_ecdsa_key.pub

Comment by Fabio Varesano (fax8) - Thursday, 10 March 2011, 10:56 GMT
I'm also affected by this issue. The workaround suggested by Petar fixed it.

My logs:
client: http://pastebin.com/T1tYGbDL
server: http://pastebin.com/nUXDJfSj
Comment by Fabio Varesano (fax8) - Thursday, 10 March 2011, 11:22 GMT Comment by Gaetan Bisson (vesath) - Saturday, 09 April 2011, 10:46 GMT
This problem affects other distributions, so it should be solved upstream.

Shortening the cipher list (see above) is a workaround, but a cleaner fix requires more debugging.

Any relevant info should be posted in this thread: http://lists.mindrot.org/pipermail/openssh-unix-dev/2011-March/029456.html

Loading...