FS#16886 - [openssh] sshd daemon does not start after openntpd update

Attached to Project: Arch Linux
Opened by (N/A) (wantilles) - Wednesday, 28 October 2009, 17:14 GMT
Last edited by Dan Griffiths (Ghost1227) - Saturday, 27 February 2010, 01:49 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Aaron Griffin (phrakture)
Thomas Bächler (brain0)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 23
Private No

Details

Description:

sshd daemon does not start after openntpd update.

Additional info:
* package version(s)

openntpd 3.9p1-8

Steps to reproduce:

1. Update openntpd to the above mentioned version.

2. Stop the sshd daemon by issuing:
/etc/rc.d/sshd stop

3. Attempt to start again the sshd daemon:
/etc/rc.d/sshd stop

It will fail:

:: Starting Secure Shell Daemon [BUSY]
Missing privilege separation directory: /var/empty
[FAIL]

If you make the directory by hand:

mkdir /var/empty

The daemon will start.

Apparently it has to do with the following line at the post_install section of the .install file of the openntpd package:

/usr/sbin/useradd -g ntp -u 87 -d /var/empty -s /bin/false -c 'Network Time Protocol' ntp

This bug is serious.

Because a remote machine, accessible only via ssh, at the next reboot, it will become inaccessible, with no warning.

As it happened here in this thread:

http://bbs.archlinux.org/viewtopic.php?id=83425
This task depends upon

Closed by  Dan Griffiths (Ghost1227)
Saturday, 27 February 2010, 01:49 GMT
Reason for closing:  Fixed
Additional comments about closing:  /var/empty is now included in the filesystem package
Comment by c (toxygen) - Wednesday, 28 October 2009, 17:48 GMT
just found about this today, after I updated via ssh as well, and lost connection, now i cant reconnect, same update of openntpd
Comment by Jan de Groot (JGC) - Wednesday, 28 October 2009, 17:53 GMT
It's not a bug in updating anything, it's a bug in the way openssh and openntpd are packaged. Both packages contain or contained the /var/empty directory, which is, well, just empty. The current openntpd package no longer contains the /var/empty directory, so pacman deletes the empty directory on upgrade. As openssh can't live without this directory, it will fail.

I would recommend to move this directory to the filesystem package and remove them from the openssh package just like it was done with the openntp package. To make sure the directory exists, a check should be added to the post_install/post_upgrade of the packages that require this directory.
Comment by Johan R (cleanrock) - Wednesday, 28 October 2009, 19:19 GMT
Isnt this a bug in the package handling if openntpd removes a directory which is part of openssh ?
Comment by Gerardo Exequiel Pozzi (djgera) - Wednesday, 28 October 2009, 20:46 GMT
See more details here  FS#15668  - [openntpd] 3.9p1-7 uninstall removes /var/empty
Comment by Jan de Groot (JGC) - Wednesday, 28 October 2009, 22:35 GMT
Directories don't belong to packages, they're just part of the directory layout. When pacman removes a package, it removes any empty directory inside that package.
Comment by (N/A) (wantilles) - Wednesday, 28 October 2009, 23:12 GMT
Then the only viable solution I see, is was JCG proposed earlier.

This directory has to be included in the "filesystem" package.
Comment by (N/A) (wantilles) - Wednesday, 28 October 2009, 23:13 GMT
I meant "what" insted of "was", in my previous comment.
Comment by Johan R (cleanrock) - Thursday, 29 October 2009, 04:18 GMT
If pacman works this way i think this is a bug in the openssh package:
If openssh require /var/empty to exist but dont create any files in it i think the package should add a dummy file in /var/empty to make sure the directory is not removed.

But if possible i think it would be a better solution to make pacman do not remove empty directories if they belong to a package.
Comment by Gerardo Exequiel Pozzi (djgera) - Thursday, 29 October 2009, 04:40 GMT
Seepacman-dev@archlinux.org/msg02826.html"> http://www.mail-archive.com/pacman-dev@archlinux.org/msg02826.html

This was the only case with openssh. One posibility is create (if not exist) the "privilege separation directory" at rc.d/sshd.
Comment by Johan R (cleanrock) - Thursday, 29 October 2009, 05:05 GMT
After reading links from djgera it looks like this will not be fixed in a safe way and that this problem can popup in the future again.
Is it really that expensive in pacman to check if an empty dir can be removed ? Maybe its worth that extra time to avoid problems like this in the future.
Comment by Thomas Bächler (brain0) - Thursday, 29 October 2009, 08:34 GMT
What about a safer fix: Add 'mkdir /var/empty' in the sshd startup script. Btw, I hadn't noticed, but I also just hit that problem.
Comment by Tobias Kieslich (tobias) - Thursday, 29 October 2009, 17:16 GMT
I would second Jans approach of making /var/empty part of the filesytem package, because even though openttpd does not have it now there is always a chance that more than one package relies on that directory.

But for the time being a banaid in the startup script will help, too.
Comment by Thomas Bächler (brain0) - Thursday, 29 October 2009, 18:46 GMT
I am not sure what the proper fix is.

However - as the /var/empty directory is that critical for sshd to work, it is generally a very good idea to create it in the init script. Package is in testing, see http://repos.archlinux.org/wsvn/packages/?compare[]=%2Fopenssh%2Ftrunk%2F%4056859&compare[]=%2Fopenssh%2Ftrunk%2F%4054385&op=comp
Comment by smyrman (smyrman) - Sunday, 01 November 2009, 11:53 GMT
1+ for seriousness for this bug. For us using Archlinux as a server, this bug really fscks things up. Since we use Archlinux we expect that the following might break: PHP, Apache, and all other kind of fscking packages. It doesn't really matter, because you could just log in and fix it. But breaking sshd is fscked up. I really which you would ALWAYS test that EVERYTHING works before updating such a package as openssh.

Thanks for your attention.
Comment by Thomas Bächler (brain0) - Sunday, 01 November 2009, 12:00 GMT
You misunderstood this: Not an update of openssh broke it, but an update of another package (openntpd) which resulted in the removal of the directory. openssh was fine all the time.
Comment by smyrman (smyrman) - Sunday, 01 November 2009, 12:12 GMT
Yes. I read.. sorry for the rage.. But still: +1 for seriousness:-)
Comment by Johan R (cleanrock) - Saturday, 07 November 2009, 11:37 GMT
The fix is not good:
Another package can still remove /var/empty, e.g. :
- login to server via ssh
- maintain packages => /var/empty can be deleted
- logout => you can no longer login again

A better fix is needed.
I would prefer to have pacman do not remove empty directories owned by other packages, this should be a safe generic solution.
Comment by Jan de Groot (JGC) - Saturday, 07 November 2009, 11:38 GMT
Looking at my OpenBSD box, my /var/empty isn't empty, it contains a /dev/log devicenode. As this looks like it doesn't have to be empty, I would advise to add this directory to the filesystem package and add a .keep file to it. This is the same method we use for dbus to make sure the system.d and session.d dirs exist after deleting the last package that installs a file there.
Comment by traaf (traaf) - Tuesday, 17 November 2009, 16:24 GMT
i was also affected after removing openntpd
Comment by Thomas Bächler (brain0) - Tuesday, 17 November 2009, 16:37 GMT
Hrm, so removing the directory while openssh is running is still critical? Should we put a hidden empty file to /var/empty then?

touch $pkgdir/var/empty/.keep.openssh

should do the trick. Opinions?
Comment by Jan de Groot (JGC) - Tuesday, 17 November 2009, 17:36 GMT
Adding /var/empty/.keep to the "filesystem" package should fix this also.
Comment by Kitty Blake (Kitty) - Saturday, 19 December 2009, 04:57 GMT
Would this patch do the job? It simply moves the existing `[ -d /var/empty ] || mkdir -p /var/empty` line to the beginning of the startup process. This way the directory will be there before ssh-* programs are first called.
   sshd.patch (0.7 KiB)
Comment by Johan R (cleanrock) - Saturday, 19 December 2009, 05:25 GMT
Kitty, that wont help. If sshd is running and /var/empty is removed you will not be able to login after this. (see my comment above)
Comment by Paul Mattal (paul) - Saturday, 06 February 2010, 14:29 GMT
Sounds like we're left with Jan's suggestion of modifying the filesystem package.

Who has privileges to do that?
Comment by Roman Kyrylych (Romashka) - Friday, 26 February 2010, 14:03 GMT
filesystem-2010.02-4 includes /var/empty/.keep
Can this be closed now?

Loading...