FS#78828 - [nix] Conflicting environment variables and tmpfiles

Attached to Project: Arch Linux
Opened by Éric (gileri) - Sunday, 18 June 2023, 21:00 GMT
Last edited by George Rawlinson (rawlinsong) - Friday, 01 September 2023, 02:11 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To Caleb Maclennan (alerque)
George Rawlinson (rawlinsong)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 4
Private No

Details

Description:

When installing the `nix` arch package, nix commands can be executed as root, but fail as unprivileged users.

$ nix-shell -p hello

warning: Nix search path entry '/nix/var/nix/profiles/per-user/twix/channels/nixpkgs' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/twix/channels' does not exist, ignoring
error:
… <borked>
at «none»:0: (source not available)

… while calling the 'import' builtin
at «string»:1:18:

1| {...}@args: with import <nixpkgs> args; (pkgs.runCommandCC or pkgs.runCommand) "shell" { buildInputs = [ (hello) ]; } ""
| ^

(stack trace truncated; use '--show-trace' to show the full trace)
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)
at «none»:0: (source not available)

The upstream multi-user installation script [1] seem to create only the nix-daemon socket [2] and no other tmpfile, or environment variables.
By keeping only the nix-daemon tmpfile.d script, and removing everything else in tmpfiles.conf, user.environment.conf, user.tmpfiles.conf from the arch package, both root and unprivileged nix commands works.

Is there a reason to have those tmpfiles and environment variables as they are ?

PS: I'm pretty new to Nix, so please take this with a pinch of salt.

Additional info:
* package version(s) : nix 2.16.1-1
* see also :  FS#73120 


Steps to reproduce:
1. # pacman -S nix && systemctl enable --now nix-daemon
2. Apply tmpfiles and environment variables, or reboot
3. # nix-channel --add https://nixos.org/channels/nixpkgs-unstable && nix-channel --update
4. # nix-shell -p hello # Works fine as root
5. $ nix-shell -p hello # Fails as an user, see output above

[1] https://github.com/NixOS/nix/blob/master/scripts/install-systemd-multi-user.sh
[2] https://github.com/NixOS/nix/blob/master/misc/systemd/nix-daemon.conf.in
This task depends upon

Closed by  George Rawlinson (rawlinsong)
Friday, 01 September 2023, 02:11 GMT
Reason for closing:  Fixed
Additional comments about closing:  2.17.0-3
Comment by Toolybird (Toolybird) - Monday, 19 June 2023, 05:49 GMT
Looks like you forgot to add yourself to the "nix-users" group [1], then log out (or reboot) for it to take effect.

[1] https://wiki.archlinux.org/title/Nix#Configuration
Comment by Éric (gileri) - Tuesday, 20 June 2023, 04:58 GMT
Can you share what you did different, with a clean machine, from the bug report ?

I tested again, on a clean VM, and the error is present. But only if user.environment.conf is applied. In a TTY, as the NIX_PATH environment variable is not applied, it works. In a shell where user.environment.conf is applied, it will exhibit the original issue.
Comment by Toolybird (Toolybird) - Tuesday, 20 June 2023, 04:59 GMT
See also  FS#78507 . I have no idea what could be causing it. All I can tell you is, it works for me and 1 of the PM's. I just simply followed the Wiki. There is no bug apparent. The bug tracker is not for support. Please take any further discussion to the appropriate support channels (Forum/IRC/Mailing Lists/Reddit/etc). Thanks
Comment by Vladimir Panteleev (CyberShadow) - Wednesday, 28 June 2023, 00:36 GMT
This bug is still linked from the wiki page; furthermore, I can reproduce the issue.

Some things about the package seem suspicious:

1. The warning is only output when NIX_PATH is set in the environment. However, the way that NIX_PATH is set is a little strange. It is configured in the file /usr/lib/environment.d/nix-daemon.conf, which apparently affects only systemd user services. So, a systemd user unit will have $NIX_PATH, but ssh-ing into the box will not have it on the session. This discrepancy may be why only some users observe this problem: if the interactive session is started from systemd, then it will inherit the NIX_PATH variable from it, but possibly not otherwise.

2. The Arch Linux package has a `/usr/share/user-tmpfiles.d/nix-daemon.conf` with the line: `d /nix/var/nix/profiles/per-user/%u 0755`. However, I don't understand how it could possibly work. `user-tmpfiles.d` are executed by a systemd user unit which runs unprivileged, so that could only possibly work if `per-user` was world-writable. `systemd-tmpfiles --create --user` fails to create the path, and `sudo systemd-tmpfiles --create --user` creates the path for the `root` user, not the user running `sudo`.

It's possible I'm just missing something in both of the above, but maybe this information will help find the root cause of the problems.
Comment by Toolybird (Toolybird) - Wednesday, 28 June 2023, 00:46 GMT
> The warning is only output when NIX_PATH is set in the environment

You might be on to something. In my testing (ssh session or local console login) NIX_PATH is *not* set (which is expected because these are not spawned by systemd user sessions). If I manually export it and set it to something bogus, I then see the error.

> with the line: `d /nix/var/nix/profiles/per-user/%u 0755`

Yeah, it looks a bit suss. I don't understand it either :(

So, it seems the folks affected by this bug have NIX_PATH set and they are spawning via systemd user session? How does that even happen?
Comment by Vladimir Panteleev (CyberShadow) - Wednesday, 28 June 2023, 04:18 GMT
> and they are spawning via systemd user session? How does that even happen?

Probably with something like https://wiki.archlinux.org/title/Systemd/User#Xorg_as_a_systemd_user_service or https://wiki.archlinux.org/title/emacs#As_a_systemd_unit?
Comment by Boris Bera (dotboris) - Sunday, 09 July 2023, 15:14 GMT
I believe that this is an issue with `NIX_PATH` being set. I get a similar issue. Instead of manifesting through nix failing to load <nixpkgs>. In my case, this issue manifests itself as `nix-channel --update` effectively doing nothing because of the way `NIX_PATH` is being set. The word "effectively" is important here because under the hood the channel does get updated but because of `NIX_PATH` users can't see it and instead see an old version of the channel. I believe that I get this issue because I've had nix installed for a while now and the `NIX_PATH` I have happens to have an old version of <nixpkgs> installed.

Here's an example of what I experience:

When `NIX_PATH` doing a channel update does nothing.

```
$ echo $NIX_PATH
nixpkgs=/nix/var/nix/profiles/per-user/dotboris/channels/nixpkgs:/nix/var/nix/profiles/per-user/dotboris/channels
$ nix-instantiate --eval --expr '(import <nixpkgs> {}).lib.version'
"23.05pre440241.b3818a46e68"
$ nix-channel --update
unpacking channels...
$ nix-instantiate --eval --expr '(import <nixpkgs> {}).lib.version'
"23.05pre440241.b3818a46e68"
```

This points to the following commit https://github.com/NixOS/nixpkgs/commit/b3818a46e68 which is from Jan 6th 2023.

But when you unset `NIX_PATH` you see that the channel update did in fact work.

```
$ env -u NIX_PATH nix-instantiate --eval --expr '(import <nixpkgs> {}).lib.version'
"23.11pre502508.d485da9d003"
```

This points to the following commit https://github.com/NixOS/nixpkgs/commit/d485da9d003 which is from July 8th 2023 (13 hours ago as of writing this).

For reference:

```
$ pacman -Qi nix | grep Version
Version : 2.16.1-2
$ nix-channel --list
nixpkgs https://nixos.org/channels/nixpkgs-unstable
$ env -u NIX_PATH nix shell nixpkgs#nix-info -c nix-info -m
- system: `"x86_64-linux"`
- host os: `Linux 6.4.1-arch2-1, Arch Linux, noversion, rolling`
- multi-user?: `yes`
- sandbox: `yes`
- version: `nix-env (Nix) 2.16.1`
- channels(root): `""`
- channels(dotboris): `"nixpkgs"`
- nixpkgs: `/home/dotboris/.nix-defexpr/channels/nixpkgs`
```

I've done some digging and I believe that I found out why this is happening. I think that there was a change around v2.3 or v2.4 of nix where non-NixOS installs should no longer set `NIX_PATH` and instead rely on the default search path. See https://discourse.nixos.org/t/where-is-nix-path-supposed-to-be-set/16434/8

I took the time to validate that this is correct and why removing `NIX_PATH` fixes the issue.

Looking at the doc for `NIX_PATH` (https://nixos.org/manual/nix/stable/command-ref/env-common.html#env-NIX_PATH) we can see the default search path:
- $HOME/.nix-defexpr/channels
- nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixpkgs
- /nix/var/nix/profiles/per-user/root/channels

Looking at the doc for `nix-channel` (https://nixos.org/manual/nix/stable/command-ref/nix-channel.html#channels) we can see that it manages symlinks to channels in:
- $XDG_STATE_HOME/nix/profiles/channels for regular users
- $NIX_STATE_DIR/profiles/per-user/root/channels for root

At first glance, these don't seem to match but that's because of symlinks. Here's what theme symlinks look on my system.

```
$ ls -l ~/.nix-defexpr/channels
lrwxrwxrwx 1 dotboris dotboris 49 Jul 9 09:09 /home/dotboris/.nix-defexpr/channels -> /home/dotboris/.local/state/nix/profiles/channels
$ ls -l ~/.local/state/nix/profiles/channels
lrwxrwxrwx 1 dotboris dotboris 15 Jul 9 09:09 /home/dotboris/.local/state/nix/profiles/channels -> channels-9-link
$ ls -l ~/.local/state/nix/profiles/channels-9-link
lrwxrwxrwx 1 dotboris dotboris 60 Jul 9 09:09 /home/dotboris/.local/state/nix/profiles/channels-9-link -> /nix/store/6zyma0pkim4xmwqjnl9c38kykaanmbja-user-environment
$ ls -l /nix/store/6zyma0pkim4xmwqjnl9c38kykaanmbja-user-environment
total 4
lrwxrwxrwx 1 root root 60 Dec 31 1969 manifest.nix -> /nix/store/w518gzzw6i3x8s1i7pmqrjfwcrk9lch3-env-manifest.nix
lrwxrwxrwx 1 root root 59 Dec 31 1969 nixpkgs -> /nix/store/fpyy2737m2364s492a1bib3kzmvxdiiw-nixpkgs/nixpkgs
```

So `~/.nix-defexpr/channels` points to `~/.local/state/nix/profiles/channels` which points to a revision of the channel which points to the installed channels in the nix store.

Looking at the channel revisions, we can see there's a bunch of recent channel updates:

```
$ ls -l ~/.local/state/nix/profiles/channels*
lrwxrwxrwx 1 dotboris dotboris 15 Jul 9 09:09 /home/dotboris/.local/state/nix/profiles/channels -> channels-9-link
lrwxrwxrwx 1 dotboris dotboris 60 Jun 28 09:30 /home/dotboris/.local/state/nix/profiles/channels-1-link -> /nix/store/d71a6slh6aq8djravxlwfcl92kq0zx4g-user-environment
lrwxrwxrwx 1 dotboris dotboris 60 Jun 28 09:32 /home/dotboris/.local/state/nix/profiles/channels-2-link -> /nix/store/zazyii52v57az1rgmv51s8v4pb8mw7g3-user-environment
lrwxrwxrwx 1 dotboris dotboris 60 Jul 8 19:49 /home/dotboris/.local/state/nix/profiles/channels-3-link -> /nix/store/piskw72b8rm2kya3lw1wwjhvx9f9ayak-user-environment
lrwxrwxrwx 1 dotboris dotboris 60 Jul 8 20:22 /home/dotboris/.local/state/nix/profiles/channels-4-link -> /nix/store/57904rflsfh46y7fxi62rzc7sidc9vmq-user-environment
lrwxrwxrwx 1 dotboris dotboris 60 Jul 8 20:24 /home/dotboris/.local/state/nix/profiles/channels-5-link -> /nix/store/piskw72b8rm2kya3lw1wwjhvx9f9ayak-user-environment
lrwxrwxrwx 1 dotboris dotboris 60 Jul 8 20:42 /home/dotboris/.local/state/nix/profiles/channels-6-link -> /nix/store/si70lnh0lyi9qlw2pwiwj1vbi51n33sz-user-environment
lrwxrwxrwx 1 dotboris dotboris 60 Jul 8 21:07 /home/dotboris/.local/state/nix/profiles/channels-7-link -> /nix/store/57904rflsfh46y7fxi62rzc7sidc9vmq-user-environment
lrwxrwxrwx 1 dotboris dotboris 60 Jul 8 21:26 /home/dotboris/.local/state/nix/profiles/channels-8-link -> /nix/store/si70lnh0lyi9qlw2pwiwj1vbi51n33sz-user-environment
lrwxrwxrwx 1 dotboris dotboris 60 Jul 9 09:09 /home/dotboris/.local/state/nix/profiles/channels-9-link -> /nix/store/6zyma0pkim4xmwqjnl9c38kykaanmbja-user-environment
```

Looking at the channels defined in `NIX_PATH` we can see that they haven't changed in a while:

```
$ ls -la /nix/var/nix/profiles/per-user/dotboris/channel*
lrwxrwxrwx 1 dotboris dotboris 15 Jan 8 13:31 /nix/var/nix/profiles/per-user/dotboris/channels -> channels-1-link
lrwxrwxrwx 1 dotboris dotboris 60 Jan 8 13:31 /nix/var/nix/profiles/per-user/dotboris/channels-1-link -> /nix/store/9hiiasd8palifxwp4jsbjmih9bpn3041-user-environment
```

I believe that removing `NIX_PATH` from `/usr/lib/environment.d/nix-daemon.conf` should solve this issue.
Comment by Christian Poveda (pvdrz93) - Tuesday, 18 July 2023, 21:21 GMT
I can confirm that running `env -u NIX_PATH nix-shell` works as intended
Comment by Bryan Lai (bryango) - Thursday, 31 August 2023, 01:31 GMT
I missed this bug and opened a new one: https://bugs.archlinux.org/task/79516. Basically,

- I can confirm that `NIX_PATH` is not just unnecessary; it's simply wrong (outdated), as already pointed out by: https://bugs.archlinux.org/task/78828#comment219391

- `PATH` in `user.environment.conf` is also unnecessary as it's automatically set by `/etc/profile.d/nix-daemon.sh`

I think it's best to leave everything to the upstream `/etc/profile.d/nix-daemon.sh` which already does the work nicely. I have hence shadowed `user.environment.conf` entirely with `~/.config/environment.d/nix-daemon.conf -> /dev/null` and things have been working smoothly for me.

Loading...