FS#59826 - [openssh] cannot connect, read from master failed: broken pipe

Attached to Project: Arch Linux
Opened by Alex Kabakaev (synapse) - Monday, 27 August 2018, 10:11 GMT
Last edited by Gaetan Bisson (vesath) - Wednesday, 29 August 2018, 07:24 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Gaetan Bisson (vesath)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 4
Private No

Details

Description:
OpenSSH client cannot connect after upgrade to the latest version 7.8p1-1.

A colleague of mine tried to upgrade openssh and got the very same error. Therefore i suppose the openssh-7.8p1-1 package is broken somehow.

Downgrade to `openssh-7.7p1-2` resolves the issue for both of us:
sudo pacman -U /var/cache/pacman/pkg/openssh-7.7p1-2-x86_64.pkg.tar.xz

The `ssh host.name -vv` output running verson openssh-7.8p1-1 is given below.

Additional info:
* package version: openssh-7.8p1-1
* config and/or log files etc.

Steps to reproduce:

$ ssh root@host.name -vv
OpenSSH_7.8p1, OpenSSL 1.1.0i 14 Aug 2018
debug1: Reading configuration data /home/vagrant/.ssh/config
debug1: /home/vagrant/.ssh/config line 116: Applying options for host.name
debug1: /home/vagrant/.ssh/config line 219: Skipping Host block because of negated match for host.name
debug1: /home/vagrant/.ssh/config line 221: Skipping Host block because of negated match for host.name
debug1: /home/vagrant/.ssh/config line 9635: Applying options for org.*
debug1: /home/vagrant/.ssh/config line 9651: Applying options for *
debug1: /home/vagrant/.ssh/config line 9656: Skipping Host block because of negated match for host.name
debug1: auto-mux: Trying existing master
debug1: Control socket "/home/vagrant/.ssh/sockets/root@host.name-22" does not exist
debug2: resolving "host.name.fqdn" port 22
debug2: ssh_connect_direct
debug1: Connecting to host.name.fqdn [10.2.3.4] port 22.
debug1: Connection established.
debug1: identity file /home/vagrant/.ssh/id_rsa_org type 0
debug1: identity file /home/vagrant/.ssh/id_rsa_org-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to host.name.fqdn:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:asdfasdfasd/asdfasdfasdfasdf
debug1: Host 'host.name.fqdn' is known and matches the ECDSA host key.
debug1: Found key in /home/vagrant/.ssh/known_hosts:59
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /home/vagrant/.ssh/id_rsa_org (0x5572cc295820), explicit, agent
debug2: key: /home/vagrant/.ssh/id_rsa_prjsvc (0x5572cc298a80), agent
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:as/asdfasdfasd/asdfasdfasdasdfasdfasdfasdfasdfsd /home/vagrant/.ssh/id_rsa_org
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg rsa-sha2-512 blen 277
debug2: input_userauth_pk_ok: fp SHA256:as/asdfasdfasd/asdfasdfasdasdfasdfasdfasdfasdfsd
debug1: Authentication succeeded (publickey).
Authenticated to host.name.fqdn ([10.2.3.4]:22).
debug1: setting up multiplex master socket
debug2: fd 5 setting O_NONBLOCK
debug1: channel 0: new [/home/vagrant/.ssh/sockets/root@host.name-22]
debug2: fd 3 setting TCP_NODELAY
debug1: control_persist_detach: backgrounding master process
debug2: control_persist_detach: background process is 1222
debug2: fd 5 setting O_NONBLOCK
debug1: forking to background
debug1: Entering interactive session.
debug1: pledge: id
debug2: set_control_persist_exit_time: schedule exit in 3600 seconds
debug1: multiplexing control connection
debug2: fd 6 setting O_NONBLOCK
debug1: channel 1: new [mux-control]
debug2: set_control_persist_exit_time: cancel scheduled exit
debug2: process_mux_master_hello: channel 1 slave version 4
debug2: mux_client_hello_exchange: master version 4
debug2: process_mux_alive_check: channel 1: alive check
debug2: process_mux_new_session: channel 1: request tty 1, X 0, agent 0, subsys 0, term "xterm-256color", cmd "", env 0
debug1: channel 2: new [client-session]
debug2: process_mux_new_session: channel_new: 2 linked to control channel 1
debug2: channel 2: send open
packet_write_wait: Connection to 10.2.3.4 port 22: Broken pipe
mux_client_request_session: read from master failed: Broken pipe
Failed to connect to new control master
This task depends upon

Closed by  Gaetan Bisson (vesath)
Wednesday, 29 August 2018, 07:24 GMT
Reason for closing:  Upstream
Comment by Alex Kabakaev (synapse) - Monday, 27 August 2018, 13:01 GMT
This issue is not reproducible on direct network connection between an OpenSSH client (archlinux) and an OpenSSH server (centos7).
The issue is only seen when a connection is routed through some (yet unknown) corporate firewalls.
Therefore I suppose that a problem lie somewhere in our internal network.
Comment by Bjørn T Johansen (bjorntj) - Monday, 27 August 2018, 13:12 GMT
I have the same problem, downgrading to openssh-7.7p1-2 fixes it.
Comment by L. Bradley LaBoon (lb.laboon) - Monday, 27 August 2018, 14:11 GMT
I can also confirm that this is an issue on my company's internal network, but not on my home network.
Comment by Daniel (dash.42) - Monday, 27 August 2018, 14:39 GMT
confirmed: problem is proxy - "solution" is downgrading
Comment by Nicholas Curran (gadfly) - Monday, 27 August 2018, 14:53 GMT
Just updated and attempting to connect from behind vmware NAT causes this for me. After switching vmware to bridged networking the guest could once again ssh.
Comment by Alex Kabakaev (synapse) - Tuesday, 28 August 2018, 11:06 GMT
@gadfly, I also run archlinux in `vmware workstation` behind NAT.
What could be wrong there?
Comment by mancha (mancha) - Wednesday, 29 August 2018, 05:35 GMT
OpenSSH 7.8 changed its defaults from TOS:lowdelay/throughput to DSCP:af21/cs1 [1]. This might be triggering a bug in VMware's network stack.

To test this, try reverting to pre-7.8 defaults by adding this to your sshd_config and restarting the daemon:

IPQoS lowdelay throughput


---
[1] https://www.iana.org/assignments/dscp-registry/dscp-registry.xhtml
Comment by Gaetan Bisson (vesath) - Wednesday, 29 August 2018, 07:24 GMT
Somebody has reported this issue upstream; see the thread starting from: http://lists.mindrot.org/pipermail/openssh-unix-dev/2018-August/037156.html

Note that this is a client issue, not a server issue. To fix it, add:

Host *
IPQoS lowdelay throughput

to ~/.ssh/config (not sshd_config).

Loading...