FS#67173 - [gitlab] systemd sandboxing breaks secret file creation, prevents puma from starting

Attached to Project: Community Packages
Opened by Matt Stickney (mtstickney) - Thursday, 02 July 2020, 06:09 GMT
Last edited by Buggy McBugFace (bugbot) - Saturday, 25 November 2023, 20:01 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Anatol Pomozov (anatolik)
Caleb Maclennan (alerque)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 2
Private No

Details

Description:
The gitlab-puma service uses ProtectSystem=full, which means that /usr and /etc are mounted read-only for the puma server. If gitlab-puma needs to write secret files (like gitlab_workhorse_secret), which are linked from /usr/share/webapps/gitlab/ to /etc/webapps/gitlab, this will fail with a "read-only FS" error, and gitlab-puma will fail to start. I've verified that starting gitlab-puma manually (as the gitlab user, using the environment file, etc.) is able to write the files without issue.

If the file exists, and has valid content in it (i.e. not an empty file), gitlab-puma won't try to create it, and you won't run into the problem. This is probably how it's gone under the radar until now.

Initially I thought using ProtectSystem=true would be better option (since /etc would then be writable), but my testing shows that writing to a symlink in a read-only fs will fail, even if the link points to a writable fs, so I'm not sure that any version of ProtectSystem can be used.

Additional info:
Package version: 13.1.1-1

Log entries of the failure (journalctl -u gitlab-puma):
Jul 02 01:51:10 palladium gitlab-puma[1336081]: {"timestamp":"2020-07-02T05:51:10.458Z","pid":1336081,"message":"! Unable to load application: Errno::EROFS: Read-only file system @ rb_sysopen - /usr/share/webapps/gitlab/.gitlab_workhorse_secret"}
Jul 02 01:51:10 palladium gitlab-puma[1336081]: bundler: failed to load command: puma (/usr/share/webapps/gitlab/vendor/bundle/ruby/2.7.0/bin/puma)
Jul 02 01:51:10 palladium gitlab-puma[1336081]: Errno::EROFS: Read-only file system @ rb_sysopen - /usr/share/webapps/gitlab/.gitlab_workhorse_secret

Steps to reproduce:
1. Attempt to start the gitlab-puma service
2. Have service fail to start after several attempts, see failure messages in journal
This task depends upon

Closed by  Buggy McBugFace (bugbot)
Saturday, 25 November 2023, 20:01 GMT
Reason for closing:  Moved
Additional comments about closing:  https://gitlab.archlinux.org/archlinux/p ackaging/packages/gitlab/issues/2
Comment by Anatol Pomozov (anatolik) - Wednesday, 08 July 2020, 05:59 GMT
Thank you for reporting this bug. I vaguely remember I had similar issue when I setup my gitlab instance last time. It is time to figure out how to fix the issue.

Opening /etc for writing to a service opens up an avenue for possible security breaches. I would much prefer to keep /etc read-only.

Actually it reminds me one packaging thing that itches me for a while. /etc is suppose to be kind of an input "administrator->service". Services should not really keep any service data/transnational data in /etc. There is /var/ directory for it.

So my question is do files like "gitlab_workhorse_secret secret secrets.yml" really belong to "/etc/webapps/gitlab/". Are these files something that is managed by an administrator? If not I would move the files to /var/lib/gitlab instead.
Comment by Matt Stickney (mtstickney) - Wednesday, 08 July 2020, 07:12 GMT
/etc/webapps/gitlab-shell/secret and /etc/webapps/gitlab/gitlab_workhorse_secret are definitely not administrative inputs (they're just cookies used to protect access to the unix sockets), and since they get written automatically by gitlab & co, they should probably live in /var/lib/gitlab{,-shell}. /etc/webapps/gitlab/secrets.yml is generated as a part of a build[1], and can be used by administrators to roll secrets, so that should be ok to live in /etc.

I'm just noticing that systemd has ReadWritePaths, which would presumably let us make /usr/share/webapps/gitlab writable for gitlab-puma. I _think_ that's all we need, since gitlab-shell isn't doing filesystem-protection stuff, /usr/share/webapps/gitlab-shell should be writable for it anyway.

[1] This is actually problematic itself, because it means everybody who installs a particular build of this package is using the same encryption keys. I couldn't find where in their code they generate this, so I don't know if it can be done at install-time or not.
Comment by Jakub Klinkovský (lahwaacz) - Thursday, 09 July 2020, 10:21 GMT
I did not know that gitlab_workhorse_secret gets created automatically. Anyway, gitlab supports manual creation of *all* secrets at least as an optional step (see https://docs.gitlab.com/charts/installation/secrets.html - not written for an "installation from source" but that doesn't matter) so it's not like administrators never touch gitlab_workhorse_secret etc.

Users of the Arch package have to manually create /etc/webapps/gitlab/secrets.yml, /etc/webapps/gitlab/secret and /etc/webapps/gitlab-shell/secret anyway (see https://wiki.archlinux.org/index.php/GitLab#Secret_strings), so I don't think adding more files to this list is a considerable hassle. But if you decide that some file should be writable by gitlab, I'd prefer moving it to /var/lib/gitlab instead of making some subdirs in /etc or /usr/share writable.
Comment by Matt Stickney (mtstickney) - Thursday, 09 July 2020, 20:17 GMT
The trick is that the user _doesn't_ have to create secrets.yml -- the package will install one and the server will start without so much as a peep (it's not the one you want, but it's there and if you start the server you're stuck with it). I've certainly never created one, and yet there it is -- all I did was upgrade from a version that predated secrets.yml. We should leave the you-really-shouldn't-use-this file out of the package, and have a postinstall prompt to create it if it doesn't exist. The other secrets files should probably be left out of the packages as well.

gitlab_workhorse_secret and secret really are just auth tokens (see https://gitlab.com/gitlab-org/gitlab-foss/-/blob/6ff3eb60e3ddbfe7450b760efe57112ddacbdb69/doc/development/internal_api.md#authentication); the only reason an admin would need to set them is if they're running several instances of the rails server on separate machines that all need to talk to the same gitlab-shell/gitlab-gitaly instances (see https://gitlab.com/gitlab-org/gitlab-foss/-/blob/082b24b03bbb9dca7edf1341aa7b0a51c9aeb18b/doc/administration/high_availability/gitlab.md#extra-configuration-for-additional-gitlab-application-servers). If they're in /var/lib, an admin can still set those by creating the file, but that's a pretty specialty case. Also note they're not intended to be included in backups, unlike secrets.yml, so they're not quite like regular config data.

The good news is that I was mistaken with my earlier tests: you _can_ write to a symlink in a read-only fs, so we don't need ReadWritePaths, and changing the symlinks to /var/lib would be sufficient for secret and gitlab_workhorse_secret.

The bad news is that apparently gitlab will attempt to generate/write to secrets.yml -- see https://gitlab.com/gitlab-org/gitlab-foss/-/blob/d081e00aa79079792b040af3323883f1f43830c5/config/initializers/01_secret_token.rb#L38 and discussion at https://gitlab.com/gitlab-org/gitlab/-/issues/222690 -- if I read it right, this happens whenever a secret the app wants is missing. That means the server might fail to start after an upgrade, but I don't think there's a big problem with requiring the user to create new secrets if necessary. It'd be great if upstream had a way to do that in a postinstall step, but...

Comment by Jakub Klinkovský (lahwaacz) - Thursday, 09 July 2020, 20:28 GMT
The Arch gitlab package currently *does not* provide any secret file (and I don't think it ever provided secrets.yml) - /usr/share/webapps/gitlab/config/secrets.yml which is in the package is just a symlink to /etc/webapps/gitlab/secrets.yml which does not exist until the user creates it.

The only "secret" file contained in the Arch packages is /etc/webapps/gitlab-shell/secret, but it is empty by default (the user has to set the secret string, see the previous link to the wiki).
https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/gitlab-shell#n78
Comment by Jakub Klinkovský (lahwaacz) - Thursday, 09 July 2020, 20:32 GMT
Also note that the discussion if ReadWritePaths is needed or not is meaningless - you can set the full path to the gitlab_workhorse_secret file directly in /etc/webapps/gitlab/gitlab.yml and other config files. I'd say this is even much better than creating artificial symlinks only to make the default paths work.
Comment by Matt Stickney (mtstickney) - Thursday, 09 July 2020, 20:32 GMT
[Edit: double-post, didn't read carefully when refreshing]
Comment by Matt Stickney (mtstickney) - Thursday, 09 July 2020, 21:51 GMT
About secrets files: perfect, I must have got mixed up when the server didn't create it on start (because /etc/ was write-only), and because I didn't check pacman -Qo like I should have. I'm very certain I didn't create that file, but I'll chalk that up to having run gitlab outside the service at some point. gitlab-shell lists /etc/webapps/gitlab-shell/secret in the backup array, so it would create an empty secret.pacnew on update (but see below).

I disagree about ReadWritePaths (or rather, about using symlinks): altering the config file is only useful for new installs, and doesn't do anything for upgraded systems (i.e. the fix requires user intervention). Fixing the symlinks works for both. I agree that having to use symlinks for those files is kind of ridiculous, but that's due to upstream badness.

gitlab-shell appears to be doing both things? It's creating a symlink from /usr/share/webapp/gitlab-shell/.gitlab_shell_secret to /etc/webapps/gitlab-shell/secret (https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/gitlab-shell#n82), but then it alters the config to use 'secret_file: /var/lib/gitlab/gitlab-shell/.gitlab_shell_secret' (https://git.archlinux.org/svntogit/community.git/tree/trunk/configs.patch?h=packages/gitlab-shell#n49), so that wiki step isn't actually doing anything for gitlab-shell. Whatever we do, we should definitely pick _one_ approach.
Comment by Jakub Klinkovský (lahwaacz) - Friday, 10 July 2020, 06:35 GMT
About altering config files: there are already many important modifications in the packaged config files and even more may be added in the future, which will lead to a .pacnew file being created. Users are expected to always merge .pacnew files after each upgrade, so this is a perfectly valid way to roll out package changes.
https://wiki.archlinux.org/index.php/System_maintenance#Deal_promptly_with_new_configuration_files

About gitlab-shell: all paths actually lead to the same file in /etc/webapps/gitlab-shell/secret, because /var/lib/gitlab/gitlab-shell is actually a symlink to /usr/share/webapps/gitlab-shell ...
Comment by Caleb Maclennan (alerque) - Saturday, 08 May 2021, 08:24 GMT
Just ran into this on a new server. It's not an issue for upgrades but setting up fresh is a royal pain.
Comment by Buggy McBugFace (bugbot) - Tuesday, 08 August 2023, 19:11 GMT
This is an automated comment as this bug is open for more then 2 years. Please reply if you still experience this bug otherwise this issue will be closed after 1 month.

Loading...