Arch Linux

Please read this before reporting a bug:
https://wiki.archlinux.org/index.php/Reporting_Bug_Guidelines

Do NOT report bugs when a package is just outdated, or it is in Unsupported. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!
Tasklist

FS#55943 - [systemd] [ecryptfs] Systemd breaks eCryptfs

Attached to Project: Arch Linux
Opened by John (graysky) - Tuesday, 10 October 2017, 19:18 GMT
Last edited by Doug Newgard (Scimmia) - Wednesday, 11 October 2017, 17:15 GMT
Task Type Bug Report
Category Packages: Core
Status Assigned
Assigned To Dave Reisner (falconindy)
Christian Hesse (eworm)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 2
Private No

Details

After updating to the systemd version indicated below, using ecryptfs is no longer possible (see below). Downgrading to systemd-234.11-9 restores the expected behavior. Functionally, this is the same problem for me that is described in  FS#54670 .

% ecryptfs-mount-private
Enter your login passphrase:
Inserted auth tok with sig [dd4d04babf75b9e6] into the user session keyring
mount: No such file or directory

When I inspect dmesg after executing the above command:
[ 247.799997] Could not find key with description: [ad8682aa06721393]
[ 247.799999] process_request_key_err: No key
[ 247.799999] Could not find valid key in user session keyring for sig specified in mount option: [ad8682aa06721393]
[ 247.800000] One or more global auth toks could not properly register; rc = [-2]
[ 247.800001] Error parsing options; rc = [-2]

Additional info:
* package version(s) systemd-235.0-1

Steps to reproduce:
1) Update to the current systemd pacakges.
2) Run ecryptfs-mount-private
This task depends upon

Comment by loqs (loqs) - Tuesday, 10 October 2017, 21:01 GMT
https://lists.archlinux.org/pipermail/arch-dev-public/2017-September/028991.html
Is pambase updated and is pam_keyring in the session part of the stack for whichever pam config the login is using?
Comment by John (graysky) - Tuesday, 10 October 2017, 23:45 GMT
@loqs - Yes, pambase was part of the package update, but retaining the new version with 234.11-9 of systemd works as expected.
Comment by loqs (loqs) - Tuesday, 10 October 2017, 23:58 GMT
Following https://wiki.archlinux.org/index.php/ECryptfs#With_configuration_files
testuser logging in from a local console
$ mount.ecryptfs_private secret
succeeds
Comment by Mario Korte (emkay1) - Thursday, 12 October 2017, 07:39 GMT
I have the same problem. It seems that "ecryptfs-insert-wrapped-passphrase-into-keyring" does not succeed. It somehow seems that the session keyring is somewhat broken with 235.0-1

Output of "keyctl show":


"keyctl show" for the good version (234.11-9):
Session Keyring
87034951 --alswrv 0 65534 keyring: _uid_ses.0
378599786 --alswrv 0 65534 \_ keyring: _uid.0
343506036 --alswrv 0 0 \_ user: b3f444e1348ea69f
425129823 --alswrv 0 0 \_ user: 8cc61a725e780b94
871062438 --alswrv 0 0 \_ user: 5f4ed5d10ebe127c
437269071 --alswrv 0 0 \_ user: 93ea5779080204e7

"keyctl show" for the bad version (235.0-1):
Session Keyring
700632667 --alswrv 0 0 keyring: _ses
128926383 ----s-rv 0 0 \_ user: invocation_id
Comment by Christian Hesse (eworm) - Thursday, 12 October 2017, 08:12 GMT
Make sure you have:

* pambase 20171006-1 installed
* changes merged in /etc/pam.d/system-login (look for /etc/pam.d/system-login.pacnew)
* your pam configuration includes system-login or system-local-login

or:

* your pam configuration includes pam_keyinit.so

What login manager do you use?
Comment by Mario Korte (emkay1) - Thursday, 12 October 2017, 08:27 GMT
I am using my arch installation headless - so no login manger.

(check) * pambase 20171006-1 installed
(check / no changes to default) * changes merged in /etc/pam.d/system-login (look for /etc/pam.d/system-login.pacnew)
(check / login includes system-local-login which includes system-login) * your pam configuration includes system-login or system-local-login

¯\_(ツ)_/¯

P.S.: system-login has the following entry:

session optional pam_keyinit.so

Is this sufficient?
Comment by Christian Hesse (eworm) - Thursday, 12 October 2017, 09:37 GMT
This line should read:

session optional pam_keyinit.so force revoke
Comment by Mario Korte (emkay1) - Thursday, 12 October 2017, 09:46 GMT
yes, it does
Comment by loqs (loqs) - Thursday, 12 October 2017, 17:08 GMT
@emkay1 the user you logged in as was root you did not use sudo or su correct?
Comment by Mario Korte (emkay1) - Thursday, 12 October 2017, 17:14 GMT
Yes, exactly. Everything was done as root logged in.
Comment by loqs (loqs) - Thursday, 12 October 2017, 17:53 GMT
Can you add a key to the user keyring with keyctl
keyctl add user mykey stuff @u
Edit also the output of the following under systemd 235.0-1:
keyctl add user mykey stuff @us
keyctl show @us
and repeat the last command after failing ecryptfs fails
Comment by loqs (loqs) - Thursday, 12 October 2017, 20:22 GMT
With keyinit
$ keyctl show
Session Keyring
947596928 --alswrv 1001 1001 keyring: _ses
859163753 --alswrv 1001 65534 \_ keyring: _uid.1001
Without keyinit
$ keyctl show
Session Keyring
14882859 --alswrv 0 0 keyring: _ses
846156920 ----s-rv 0 0 \_ user: invocation_id

So it appears for emkay1 keyinit is in use but the session keyring is not set correctly.
Comment by Mario Korte (emkay1) - Friday, 13 October 2017, 06:26 GMT
Working Version (234.11-9):

# keyctl show @us
Keyring
761899519 --alswrv 0 65534 keyring: _uid_ses.0
1033061305 --alswrv 0 65534 \_ keyring: _uid.0
463765567 --alswrv 0 0 \_ user: mykey
713061870 --alswrv 0 0 \_ user: 8cc61a725e780b94
413909133 --alswrv 0 0 \_ user: 5f4ed5d10ebe127c
980612108 --alswrv 0 0 \_ user: 93ea5779080204e7
1055043847 --alswrv 0 0 \_ user: b3f444e1348ea69f


# keyctl show
Session Keyring
761899519 --alswrv 0 65534 keyring: _uid_ses.0
1033061305 --alswrv 0 65534 \_ keyring: _uid.0
463765567 --alswrv 0 0 \_ user: mykey
713061870 --alswrv 0 0 \_ user: 8cc61a725e780b94
413909133 --alswrv 0 0 \_ user: 5f4ed5d10ebe127c
980612108 --alswrv 0 0 \_ user: 93ea5779080204e7
1055043847 --alswrv 0 0 \_ user: b3f444e1348ea69f


Defective Version (235.0-1):

# keyctl show @us
Keyring
312461122 --alswrv 0 65534 keyring: _uid_ses.0
418949746 --alswrv 0 65534 \_ keyring: _uid.0
796264845 --alswrv 0 0 \_ user: b3f444e1348ea69f
848554167 --alswrv 0 0 \_ user: mykey
287620381 --alswrv 0 0 \_ user: 8cc61a725e780b94
551403634 --alswrv 0 0 \_ user: 5f4ed5d10ebe127c
235940776 --alswrv 0 0 \_ user: 93ea5779080204e7

# keyctl show
Session Keyring
700632667 --alswrv 0 0 keyring: _ses
128926383 ----s-rv 0 0 \_ user: invocation_id

So, it seems the user keyring is indeed populated, but somehow the session keyring is broken.
Comment by loqs (loqs) - Friday, 13 October 2017, 16:24 GMT
The output of keyctl shown under 235.0-1 matches what I produced when keyinit was disabled in the pam stack.
Edit:
adding the debug option to pam_keyinit in /etc/pam.d/system-login do you see output similar to
Oct 13 17:51:47 arch systemd[1704]: pam_keyinit(systemd-user:session): OPEN 1
Oct 13 17:51:47 arch systemd[1704]: pam_keyinit(systemd-user:session): UID:1001 [0] GID:1001 [0]
Oct 13 17:51:47 arch systemd[1704]: pam_keyinit(systemd-user:session): JOIN = 383839925
in the journal after logging in?
Comment by Mario Korte (emkay1) - Saturday, 14 October 2017, 09:18 GMT
I do see these lines when logging in directly, I don't see those lines when logging in via ssh (with rsa publickkey) - which is the usual way I access the server.

But sshd includes system-remote-login which again includes system-login.

Why is keyinit not executed when logging in via ssh?

And why is this not a problem when using the old systemd version?

edited:
I found the following line in /etc/ssh/sshd_config
UsePAM no

This could be the culprit. But again: why is this a problem with the new systemd version, but not before?

Comment by loqs (loqs) - Saturday, 14 October 2017, 10:56 GMT
See
https://github.com/systemd/systemd/blob/v233/NEWS#L261
https://github.com/systemd/systemd/pull/6286
https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/systemd&id=d60e69151a5d43462b37f244179ef3574a2bde69 arch has been reverting the upstream change for v233 and v234 now with v235 continuing to use the revert would break other features and with pam_keyinit there was no need to keep doing so.
@graysky is this the same issue your system is affected by without feedback this bug report could easily address emkay1's issue but not yours.
Comment by John (graysky) - Saturday, 14 October 2017, 13:58 GMT Comment by loqs (loqs) - Saturday, 14 October 2017, 17:50 GMT
@emkay1 when logging in directly does does ecryptfs work as expected and the issue is only present when the pam stack is not executed?
@graysky not enough information to know if it is related. Is pam_keyinit being executed on your affected system? (see https://bugs.archlinux.org/task/55943#comment162435 for test)
Comment by Mario Korte (emkay1) - Sunday, 15 October 2017, 09:16 GMT
@loqs When using the direct login ecryptfs works as it should.

So I guess I will have to set "UsePAM yes" in my sshd_config, although I do not want the ability to log in via password at all :/

PasswordAuthentication no
ChallengeResponseAuthentication no

should suffice, though.
Comment by Mario Korte (emkay1) - Sunday, 15 October 2017, 09:34 GMT
So, conclusion: I guess this can be closed then...
Comment by John (graysky) - Sunday, 15 October 2017, 09:45 GMT
@Mario - That conclusion is incorrect in my case. I have 'UsePAM yes' setup on a headless box. When I ssh into the box using ssh-key-based auth (the only way I will ever do it), I am unable to use ecryptfs as described above with systemd-235.0-1 on the box. I have to downgrade it to 234.11-9.

% sed '/ *#/d; /^ *$/d' /etc/ssh/sshd_config
AuthorizedKeysFile .ssh/authorized_keys
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
Subsystem sftp /usr/lib/ssh/sftp-server
Comment by loqs (loqs) - Sunday, 15 October 2017, 10:04 GMT
@graysky Is pam_keyinit being executed on your affected system? (see https://bugs.archlinux.org/task/55943#comment162435 for test)
Comment by John (graysky) - Sunday, 15 October 2017, 10:11 GMT
@loqs - I can post the output of keyctl under each version but I am unclear how to "disable keyinit in the pam stack." Are you asking to see the debug option in /etc/pam.d/system-login?

The package provided file:
% grep key /etc/pam.d/system-login
session optional pam_keyinit.so force revoke
Comment by loqs (loqs) - Sunday, 15 October 2017, 14:04 GMT
Yes add the debug option to pam_keyinit in /etc/pam.d/system-login so it becomes:
session optional pam_keyinit.so force revoke debug
then login and check if the journal has output similar to:
Oct 13 17:51:47 arch systemd[1704]: pam_keyinit(systemd-user:session): OPEN 1
Oct 13 17:51:47 arch systemd[1704]: pam_keyinit(systemd-user:session): UID:1001 [0] GID:1001 [0]
Oct 13 17:51:47 arch systemd[1704]: pam_keyinit(systemd-user:session): JOIN = 383839925

Yes also the outputs of `keyctl show` and `keyctl show @us` if they are not similar to emkay1's
If pam_keyinit is never called then the situation would appear to be the same as emkay1 if pam_keyinit is called but the keyring is still wrong something different has occurred.
Comment by loqs (loqs) - Sunday, 15 October 2017, 15:30 GMT
@emkay1 from the default /etc/ssh/sshd_config
# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

https://wiki.mozilla.org/Security/Guidelines/OpenSSH
for OpenSSH 6.7+ uses the following to restict logins to publickey only
AuthenticationMethods publickey
Comment by John (graysky) - Sunday, 15 October 2017, 18:01 GMT
@loqs - I will add that line to sshd_config and do that test under 134 and 135 shortly.
Comment by John (graysky) - Wednesday, 18 October 2017, 18:54 GMT
Sorry, got distracted. After updating to systemd-235.8-1 and using a stock /etc/ssh/sshd_config, everything is working as expected. Suspect the problem was with systemd-235.0-1.
Comment by Christian Hesse (eworm) - Wednesday, 18 October 2017, 19:11 GMT
No, the problem was with your sshd_config. ;)

So anybody happy? Can we close?
Comment by Joe Rawson (jrawson) - Thursday, 09 November 2017, 16:12 GMT
This killed my morning, so thank you all for this.

Though it's a small thing, I was confused why this didn't work for me even when my settings matched everything graysky had, until I realized I was using mosh, which apparently doesn't behave like ssh with regard to session keys.

My work around was to do the obvious and mount the ecryptfs volume over ssh then start a mosh session, so I don't know how pedantic this project wants to be about the meaning of "fixed," though it seems that this upstream change in systemd has edge cases not yet covered and should be addressed.
Comment by Christian Hesse (eworm) - Thursday, 09 November 2017, 17:07 GMT
Does mosh use pam in any way?
Comment by Joe Rawson (jrawson) - Thursday, 09 November 2017, 17:45 GMT
I have no idea.

From https://mosh.org mosh establishes an ssh connection first and exchanges its own encryption keys for traffic and spawns a mosh-server instance, then closes the connection. I would guess it does nothing with pam and attempts to inherit whatever keys were established by ssh, but I'm not familiar at all with this session key business nor why it's necessary. I'm just a user that found an edge case that still seems broken.

The mosh-server remains persistent, but since the parent process exited I'm not sure what happens to the session keys that seem to be the problem here.

I'm asking around on #mosh on freenode to try to get a better answer.

My use case is similar to graysky's, headless server running remotely mostly used to serve files and run tmux.
Comment by Joe Rawson (jrawson) - Thursday, 09 November 2017, 18:56 GMT
A couple of knowledgeable individuals (shout-outs to achin and cgull) involved with mosh are aware of recent systemd funny business that have affected their project, the chief one in this case being: https://github.com/mobile-shell/mosh/issues/529.

They also confirmed mosh works by double-forking the mosh-server from the ssh process that was opened, so does not directly do anything with pam but should (you would hope) inherit whatever session keys the ssh process had.
Comment by loqs (loqs) - Friday, 10 November 2017, 23:44 GMT
@jrawson the issue started with the upgrade to systemd 235.0-1?
Is the session valid see: https://wiki.archlinux.org/index.php/General_troubleshooting#Session_permissions
Is the volume unlocked using pam_ecryptfs or manually?
What is the output of keyctl show when you log in via mosh?
Comment by Joe Rawson (jrawson) - Saturday, 11 November 2017, 01:50 GMT
The issue started with the upgrade to the latest systemd, which is at 235.38-1. My previous version was 234.11-4 before my last upgrade.

I always mount the volume manually using ecryptfs-mount-private over a remote connection, logout and back in again.

This server does not have X installed, which is mentioned on the general troubleshooting page you linked to so I figured I would mention it.

Running the command: `loginctl show-session $XDG_SESSION_ID`
from a mosh session yields: `Failed to get session path: No session '9' known`

Using ssh:
Id=-redacted-
User=-redacted-
Name=jrawson
Timestamp=Fri 2017-11-10 17:30:25 PST
TimestampMonotonic=122573834144
VTNr=0
Remote=yes
RemoteHost=-redacted-
Service=sshd
Scope=-redacted-.scope
Leader=-redacted-
Audit=-redacted-
Type=tty
Class=user
Active=yes
State=active
IdleHint=no
IdleSinceHint=0
IdleSinceHintMonotonic=0
LockedHint=no

So that's right off the bat.

keyctl using ssh and mosh show identical results using @u and @us. If you give me a command I'll run it and see what it says.

Trying to set up an ecryptfs volume for a test user on this server is also prevented:

[root@redacted ~]# ecryptfs-migrate-home -u testuser
INFO: Checking disk space, this may take a few moments. Please be patient.
INFO: Checking for open files in /home/testuser
Enter your login passphrase [testuser]:

************************************************************************
YOU SHOULD RECORD YOUR MOUNT PASSPHRASE AND STORE IT IN A SAFE LOCATION.
ecryptfs-unwrap-passphrase ~/.ecryptfs/wrapped-passphrase
THIS WILL BE REQUIRED IF YOU NEED TO RECOVER YOUR DATA AT A LATER TIME.
************************************************************************


Done configuring.

keyctl_link: Key has been revoked
chown: cannot access '/dev/shm/.ecryptfs-testuser': No such file or directory
mount: /home/testuser: mount(2) system call failed: No such file or directory.
ERROR: Could not mount

And produces an unusable private folder.

Doing this over ssh is fine.

Once the ecryptfs homedir is established:

[testuser@redacted ~]$ ecryptfs-mount-private
Enter your login passphrase:
Inserted auth tok with sig [a745ffdaf6006db9] into the user session keyring
mount: No such file or directory
[testuser@redacted ~]$

Over ssh, it mounts fine.

Loading...