Arch Linux

https://lists.archlinux.org/pipermail/arch-dev-public/2017-September/028991.html
Is pambase updated and is pam_keyring in the session part of the stack for whichever pam config the login is using?

@loqs - Yes, pambase was part of the package update, but retaining the new version with 234.11-9 of systemd works as expected.

Following https://wiki.archlinux.org/index.php/ECryptfs#With_configuration_files
testuser logging in from a local console
$ mount.ecryptfs_private secret
succeeds

I have the same problem. It seems that "ecryptfs-insert-wrapped-passphrase-into-keyring" does not succeed. It somehow seems that the session keyring is somewhat broken with 235.0-1

Output of "keyctl show":


"keyctl show" for the good version (234.11-9):
Session Keyring
87034951 --alswrv 0 65534 keyring: _uid_ses.0
378599786 --alswrv 0 65534 \_ keyring: _uid.0
343506036 --alswrv 0 0 \_ user: b3f444e1348ea69f
425129823 --alswrv 0 0 \_ user: 8cc61a725e780b94
871062438 --alswrv 0 0 \_ user: 5f4ed5d10ebe127c
437269071 --alswrv 0 0 \_ user: 93ea5779080204e7

"keyctl show" for the bad version (235.0-1):
Session Keyring
700632667 --alswrv 0 0 keyring: _ses
128926383 ----s-rv 0 0 \_ user: invocation_id

Make sure you have:

* pambase 20171006-1 installed
* changes merged in /etc/pam.d/system-login (look for /etc/pam.d/system-login.pacnew)
* your pam configuration includes system-login or system-local-login

or:

* your pam configuration includes pam_keyinit.so

What login manager do you use?

I am using my arch installation headless - so no login manger.

(check) * pambase 20171006-1 installed
(check / no changes to default) * changes merged in /etc/pam.d/system-login (look for /etc/pam.d/system-login.pacnew)
(check / login includes system-local-login which includes system-login) * your pam configuration includes system-login or system-local-login

¯\_(ツ)_/¯

P.S.: system-login has the following entry:

session optional pam_keyinit.so

Is this sufficient?

This line should read:

session optional pam_keyinit.so force revoke

yes, it does

@emkay1 the user you logged in as was root you did not use sudo or su correct?

Yes, exactly. Everything was done as root logged in.

Can you add a key to the user keyring with keyctl
keyctl add user mykey stuff @u
Edit also the output of the following under systemd 235.0-1:
keyctl add user mykey stuff @us
keyctl show @us
and repeat the last command after failing ecryptfs fails

With keyinit
$ keyctl show
Session Keyring
947596928 --alswrv 1001 1001 keyring: _ses
859163753 --alswrv 1001 65534 \_ keyring: _uid.1001
Without keyinit
$ keyctl show
Session Keyring
14882859 --alswrv 0 0 keyring: _ses
846156920 ----s-rv 0 0 \_ user: invocation_id

So it appears for emkay1 keyinit is in use but the session keyring is not set correctly.

Working Version (234.11-9):

# keyctl show @us
Keyring
761899519 --alswrv 0 65534 keyring: _uid_ses.0
1033061305 --alswrv 0 65534 \_ keyring: _uid.0
463765567 --alswrv 0 0 \_ user: mykey
713061870 --alswrv 0 0 \_ user: 8cc61a725e780b94
413909133 --alswrv 0 0 \_ user: 5f4ed5d10ebe127c
980612108 --alswrv 0 0 \_ user: 93ea5779080204e7
1055043847 --alswrv 0 0 \_ user: b3f444e1348ea69f


# keyctl show
Session Keyring
761899519 --alswrv 0 65534 keyring: _uid_ses.0
1033061305 --alswrv 0 65534 \_ keyring: _uid.0
463765567 --alswrv 0 0 \_ user: mykey
713061870 --alswrv 0 0 \_ user: 8cc61a725e780b94
413909133 --alswrv 0 0 \_ user: 5f4ed5d10ebe127c
980612108 --alswrv 0 0 \_ user: 93ea5779080204e7
1055043847 --alswrv 0 0 \_ user: b3f444e1348ea69f


Defective Version (235.0-1):

# keyctl show @us
Keyring
312461122 --alswrv 0 65534 keyring: _uid_ses.0
418949746 --alswrv 0 65534 \_ keyring: _uid.0
796264845 --alswrv 0 0 \_ user: b3f444e1348ea69f
848554167 --alswrv 0 0 \_ user: mykey
287620381 --alswrv 0 0 \_ user: 8cc61a725e780b94
551403634 --alswrv 0 0 \_ user: 5f4ed5d10ebe127c
235940776 --alswrv 0 0 \_ user: 93ea5779080204e7

# keyctl show
Session Keyring
700632667 --alswrv 0 0 keyring: _ses
128926383 ----s-rv 0 0 \_ user: invocation_id

So, it seems the user keyring is indeed populated, but somehow the session keyring is broken.

The output of keyctl shown under 235.0-1 matches what I produced when keyinit was disabled in the pam stack.
Edit:
adding the debug option to pam_keyinit in /etc/pam.d/system-login do you see output similar to
Oct 13 17:51:47 arch systemd[1704]: pam_keyinit(systemd-user:session): OPEN 1
Oct 13 17:51:47 arch systemd[1704]: pam_keyinit(systemd-user:session): UID:1001 [0] GID:1001 [0]
Oct 13 17:51:47 arch systemd[1704]: pam_keyinit(systemd-user:session): JOIN = 383839925
in the journal after logging in?

I do see these lines when logging in directly, I don't see those lines when logging in via ssh (with rsa publickkey) - which is the usual way I access the server.

But sshd includes system-remote-login which again includes system-login.

Why is keyinit not executed when logging in via ssh?

And why is this not a problem when using the old systemd version?

edited:
I found the following line in /etc/ssh/sshd_config
UsePAM no

This could be the culprit. But again: why is this a problem with the new systemd version, but not before?

See
https://github.com/systemd/systemd/blob/v233/NEWS#L261
https://github.com/systemd/systemd/pull/6286
https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/systemd&id=d60e69151a5d43462b37f244179ef3574a2bde69 arch has been reverting the upstream change for v233 and v234 now with v235 continuing to use the revert would break other features and with pam_keyinit there was no need to keep doing so.
@graysky is this the same issue your system is affected by without feedback this bug report could easily address emkay1's issue but not yours.

Related? https://github.com/systemd/systemd/issues/7074

@emkay1 when logging in directly does does ecryptfs work as expected and the issue is only present when the pam stack is not executed?
@graysky not enough information to know if it is related. Is pam_keyinit being executed on your affected system? (see https://bugs.archlinux.org/task/55943#comment162435 for test)

@loqs When using the direct login ecryptfs works as it should.

So I guess I will have to set "UsePAM yes" in my sshd_config, although I do not want the ability to log in via password at all :/

PasswordAuthentication no
ChallengeResponseAuthentication no

should suffice, though.

So, conclusion: I guess this can be closed then...

@Mario - That conclusion is incorrect in my case. I have 'UsePAM yes' setup on a headless box. When I ssh into the box using ssh-key-based auth (the only way I will ever do it), I am unable to use ecryptfs as described above with systemd-235.0-1 on the box. I have to downgrade it to 234.11-9.

% sed '/ *#/d; /^ *$/d' /etc/ssh/sshd_config
AuthorizedKeysFile .ssh/authorized_keys
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
Subsystem sftp /usr/lib/ssh/sftp-server

@graysky Is pam_keyinit being executed on your affected system? (see https://bugs.archlinux.org/task/55943#comment162435 for test)

@loqs - I can post the output of keyctl under each version but I am unclear how to "disable keyinit in the pam stack." Are you asking to see the debug option in /etc/pam.d/system-login?

The package provided file:
% grep key /etc/pam.d/system-login
session optional pam_keyinit.so force revoke

Yes add the debug option to pam_keyinit in /etc/pam.d/system-login so it becomes:
session optional pam_keyinit.so force revoke debug
then login and check if the journal has output similar to:
Oct 13 17:51:47 arch systemd[1704]: pam_keyinit(systemd-user:session): OPEN 1
Oct 13 17:51:47 arch systemd[1704]: pam_keyinit(systemd-user:session): UID:1001 [0] GID:1001 [0]
Oct 13 17:51:47 arch systemd[1704]: pam_keyinit(systemd-user:session): JOIN = 383839925

Yes also the outputs of `keyctl show` and `keyctl show @us` if they are not similar to emkay1's
If pam_keyinit is never called then the situation would appear to be the same as emkay1 if pam_keyinit is called but the keyring is still wrong something different has occurred.

@emkay1 from the default /etc/ssh/sshd_config
# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

https://wiki.mozilla.org/Security/Guidelines/OpenSSH
for OpenSSH 6.7+ uses the following to restict logins to publickey only
AuthenticationMethods publickey

@loqs - I will add that line to sshd_config and do that test under 134 and 135 shortly.

Sorry, got distracted. After updating to systemd-235.8-1 and using a stock /etc/ssh/sshd_config, everything is working as expected. Suspect the problem was with systemd-235.0-1.

No, the problem was with your sshd_config. ;)

So anybody happy? Can we close?

This killed my morning, so thank you all for this.

Though it's a small thing, I was confused why this didn't work for me even when my settings matched everything graysky had, until I realized I was using mosh, which apparently doesn't behave like ssh with regard to session keys.

My work around was to do the obvious and mount the ecryptfs volume over ssh then start a mosh session, so I don't know how pedantic this project wants to be about the meaning of "fixed," though it seems that this upstream change in systemd has edge cases not yet covered and should be addressed.

Does mosh use pam in any way?

I have no idea.

From https://mosh.org mosh establishes an ssh connection first and exchanges its own encryption keys for traffic and spawns a mosh-server instance, then closes the connection. I would guess it does nothing with pam and attempts to inherit whatever keys were established by ssh, but I'm not familiar at all with this session key business nor why it's necessary. I'm just a user that found an edge case that still seems broken.

The mosh-server remains persistent, but since the parent process exited I'm not sure what happens to the session keys that seem to be the problem here.

I'm asking around on #mosh on freenode to try to get a better answer.

My use case is similar to graysky's, headless server running remotely mostly used to serve files and run tmux.

A couple of knowledgeable individuals (shout-outs to achin and cgull) involved with mosh are aware of recent systemd funny business that have affected their project, the chief one in this case being: https://github.com/mobile-shell/mosh/issues/529.

They also confirmed mosh works by double-forking the mosh-server from the ssh process that was opened, so does not directly do anything with pam but should (you would hope) inherit whatever session keys the ssh process had.

@jrawson the issue started with the upgrade to systemd 235.0-1?
Is the session valid see: https://wiki.archlinux.org/index.php/General_troubleshooting#Session_permissions
Is the volume unlocked using pam_ecryptfs or manually?
What is the output of keyctl show when you log in via mosh?

The issue started with the upgrade to the latest systemd, which is at 235.38-1. My previous version was 234.11-4 before my last upgrade.

I always mount the volume manually using ecryptfs-mount-private over a remote connection, logout and back in again.

This server does not have X installed, which is mentioned on the general troubleshooting page you linked to so I figured I would mention it.

Running the command: `loginctl show-session $XDG_SESSION_ID`
from a mosh session yields: `Failed to get session path: No session '9' known`

Using ssh:
Id=-redacted-
User=-redacted-
Name=jrawson
Timestamp=Fri 2017-11-10 17:30:25 PST
TimestampMonotonic=122573834144
VTNr=0
Remote=yes
RemoteHost=-redacted-
Service=sshd
Scope=-redacted-.scope
Leader=-redacted-
Audit=-redacted-
Type=tty
Class=user
Active=yes
State=active
IdleHint=no
IdleSinceHint=0
IdleSinceHintMonotonic=0
LockedHint=no

So that's right off the bat.

keyctl using ssh and mosh show identical results using @u and @us. If you give me a command I'll run it and see what it says.

Trying to set up an ecryptfs volume for a test user on this server is also prevented:

[root@redacted ~]# ecryptfs-migrate-home -u testuser
INFO: Checking disk space, this may take a few moments. Please be patient.
INFO: Checking for open files in /home/testuser
Enter your login passphrase [testuser]:

************************************************************************
YOU SHOULD RECORD YOUR MOUNT PASSPHRASE AND STORE IT IN A SAFE LOCATION.
ecryptfs-unwrap-passphrase ~/.ecryptfs/wrapped-passphrase
THIS WILL BE REQUIRED IF YOU NEED TO RECOVER YOUR DATA AT A LATER TIME.
************************************************************************


Done configuring.

keyctl_link: Key has been revoked
chown: cannot access '/dev/shm/.ecryptfs-testuser': No such file or directory
mount: /home/testuser: mount(2) system call failed: No such file or directory.
ERROR: Could not mount

And produces an unusable private folder.

Doing this over ssh is fine.

Once the ecryptfs homedir is established:

[testuser@redacted ~]$ ecryptfs-mount-private
Enter your login passphrase:
Inserted auth tok with sig [a745ffdaf6006db9] into the user session keyring
mount: No such file or directory
[testuser@redacted ~]$

Over ssh, it mounts fine.

This is an automated comment as this bug is open for more then 2 years. Please reply if you still experience this bug otherwise this issue will be closed after 1 month.