I confirm I see this issue as well.

But adding ReadWritePaths=/usr/share/webapps/gitlab/tmp to the /usr/lib/systemd/system/gitlab-backup.service file does not really help.

Could you please share the /usr/lib/systemd/system/gitlab-backup.service that works for you without the drop-in file?

In my environment (I made a brand-new Gitlab environment), /usr/lib/systemd/system/gitlab-backup.service is
```
[Unit]
Description=GitLab Backup process
Requires=
After=network.target

[Service]
Type=oneshot
User=gitlab
Group=gitlab
SyslogIdentifier=gitlab-backup
WorkingDirectory=/usr/share/webapps/gitlab
EnvironmentFile=/usr/share/webapps/gitlab/environment
CapabilityBoundingSet=
PrivateTmp=true
PrivateDevices=true
ProtectSystem=full
ProtectHome=true
NoNewPrivileges=true
ExecStart=/usr/bin/bundle-2.7 exec rake gitlab:backup:create
ReadWritePaths=/usr/share/webapps/gitlab/tmp
```

I just added "ReadWritePaths=/usr/share/webapps/gitlab/tmp" on the last of te service file.

-------

- mitty@gitlab-tmp:~$ sudo vim /usr/lib/systemd/system/gitlab-backup.service
```
--- /usr/lib/systemd/system/gitlab-backup.service.orig 2022-11-05 09:27:58.000000000 +0900
+++ /usr/lib/systemd/system/gitlab-backup.service 2022-11-22 20:25:07.136666805 +0900
@@ -17,3 +17,4 @@
ProtectHome=true
NoNewPrivileges=true
ExecStart=/usr/bin/bundle-2.7 exec rake gitlab:backup:create
+ReadWritePaths=/usr/share/webapps/gitlab/tmp
```
- mitty@gitlab-tmp:~$ sudo systemctl daemon-reload
- mitty@gitlab-tmp:~$ sudo systemctl start gitlab-backup
```
Nov 22 20:25:51 gitlab-tmp systemd[1]: Starting GitLab Backup process...
Nov 22 20:26:02 gitlab-tmp gitlab-backup[829]: /usr/share/webapps/gitlab/lib/gitlab/instrumentation/redis.rb:9: warning: already initialized constant Gitlab::Instrumentation::Redis::ActionCable
Nov 22 20:26:02 gitlab-tmp gitlab-backup[829]: /usr/share/webapps/gitlab/lib/gitlab/instrumentation/redis.rb:9: warning: previous definition of ActionCable was here
Nov 22 20:26:05 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:05 +0900 -- Dumping main_database ...
Nov 22 20:26:08 gitlab-tmp gitlab-backup[829]: Dumping PostgreSQL database gitlabhq_production ... [DONE]
Nov 22 20:26:08 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:08 +0900 -- Dumping main_database ... done
Nov 22 20:26:08 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:08 +0900 -- Dumping ci_database ... [DISABLED]
Nov 22 20:26:08 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:08 +0900 -- Dumping repositories ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: {"command":"create","gl_project_path":"gitlab-instance-7188cfa5/Monitoring","level":"info","msg":"started create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","storage_name":"default","time":"2022-11-22T11:26:09.251Z"}
Nov 22 20:26:09 gitlab-tmp gitlab-gitaly[351]: time="2022-11-22T11:26:09.265Z" level=info msg="finished unary call with code OK" command.count=1 command.cpu_time_ms=2 command.inblock=32 command.majflt=0 command.maxrss=308976 command.minflt=144 command.oublock=0 command.real_time_ms=3 command.system_time_ms=2 command.user_time_ms=0 correlation_id=01GJFHD6BCHVE7VQCCS29TM69K grpc.code=OK grpc.meta.deadline_type=none grpc.meta.method_type=unary grpc.method=HasLocalBranches grpc.request.fullMethod=/gitaly.RepositoryService/HasLocalBranches grpc.request.glProjectPath=gitlab-instance-7188cfa5/Monitoring grpc.request.glRepository= grpc.request.payload_bytes=133 grpc.request.repoPath=@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git grpc.request.repoStorage=default grpc.response.payload_bytes=0 grpc.service=gitaly.RepositoryService grpc.start_time="2022-11-22T20:26:09.260" grpc.time_ms=4.511 peer.address=@ pid=351 span.kind=server system=grpc
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: {"command":"create","error":"manager: repository empty: repository skipped","gl_project_path":"gitlab-instance-7188cfa5/Monitoring","level":"warning","msg":"skipped create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","storage_name":"default","time":"2022-11-22T11:26:09.266Z"}
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: {"command":"create","gl_project_path":"gitlab-instance-7188cfa5/Monitoring.wiki","level":"info","msg":"started create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git","storage_name":"default","time":"2022-11-22T11:26:09.388Z"}
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: {"command":"create","gl_project_path":"gitlab-instance-7188cfa5/Monitoring","level":"info","msg":"started create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.design.git","storage_name":"default","time":"2022-11-22T11:26:09.390Z"}
Nov 22 20:26:09 gitlab-tmp gitlab-gitaly[351]: time="2022-11-22T11:26:09.390Z" level=info msg="finished unary call with code OK" command.count=1 command.cpu_time_ms=1 command.inblock=0 command.majflt=0 command.maxrss=308976 command.minflt=146 command.oublock=0 command.real_time_ms=1 command.system_time_ms=0 command.user_time_ms=1 correlation_id=01GJFHD6FCMJK21S5A8SXVH623 grpc.code=OK grpc.meta.deadline_type=none grpc.meta.method_type=unary grpc.method=HasLocalBranches grpc.request.fullMethod=/gitaly.RepositoryService/HasLocalBranches grpc.request.glProjectPath=gitlab-instance-7188cfa5/Monitoring.wiki grpc.request.glRepository= grpc.request.payload_bytes=143 grpc.request.repoPath=@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git grpc.request.repoStorage=default grpc.response.payload_bytes=0 grpc.service=gitaly.RepositoryService grpc.start_time="2022-11-22T20:26:09.388" grpc.time_ms=2.239 peer.address=@ pid=351 span.kind=server system=grpc
Nov 22 20:26:09 gitlab-tmp gitlab-gitaly[351]: time="2022-11-22T11:26:09.391Z" level=info msg="finished unary call with code NotFound" correlation_id=01GJFHD6FFRBG59HNGEZDAN790 error="GetRepoPath: not a git repository: \"/var/lib/gitlab/repositories/@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.design.git\"" grpc.code=NotFound grpc.meta.deadline_type=none grpc.meta.method_type=unary grpc.method=HasLocalBranches grpc.request.fullMethod=/gitaly.RepositoryService/HasLocalBranches grpc.request.glProjectPath=gitlab-instance-7188cfa5/Monitoring grpc.request.glRepository= grpc.request.payload_bytes=140 grpc.request.repoPath=@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.design.git grpc.request.repoStorage=default grpc.response.payload_bytes=0 grpc.service=gitaly.RepositoryService grpc.start_time="2022-11-22T20:26:09.391" grpc.time_ms=0.165 peer.address=@ pid=351 span.kind=server system=grpc
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: {"command":"create","error":"manager: repository empty: repository skipped","gl_project_path":"gitlab-instance-7188cfa5/Monitoring.wiki","level":"warning","msg":"skipped create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git","storage_name":"default","time":"2022-11-22T11:26:09.391Z"}
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: {"command":"create","error":"manager: repository empty: repository skipped","gl_project_path":"gitlab-instance-7188cfa5/Monitoring","level":"warning","msg":"skipped create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.design.git","storage_name":"default","time":"2022-11-22T11:26:09.391Z"}
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping repositories ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping uploads ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping uploads ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping builds ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping builds ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping artifacts ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping artifacts ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping pages ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping pages ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping lfs objects ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping lfs objects ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping terraform states ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping terraform states ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping container registry images ... [DISABLED]
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping packages ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping packages ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Creating backup archive: 1669116365_2022_11_22_15.5.2_gitlab_backup.tar ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Creating backup archive: 1669116365_2022_11_22_15.5.2_gitlab_backup.tar ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Uploading backup archive to remote storage ... [SKIPPED]
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Deleting old backups ... [SKIPPED]
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Deleting tar staging files ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/backup_information.yml
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/db
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/uploads.tar.gz
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/builds.tar.gz
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/artifacts.tar.gz
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/pages.tar.gz
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/lfs.tar.gz
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/terraform_state.tar.gz
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/packages.tar.gz
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Deleting tar staging files ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Deleting backups/tmp ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Deleting backups/tmp ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: and are not included in this backup. You will need these files to restore a backup.
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: Please back them up manually.
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Backup 1669116365_2022_11_22_15.5.2 is done.
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Deleting backup and restore lock file
Nov 22 20:26:09 gitlab-tmp systemd[1]: gitlab-backup.service: Deactivated successfully.
Nov 22 20:26:09 gitlab-tmp systemd[1]: Finished GitLab Backup process.
```

You may need to "mkdir /var/lib/gitlab/shared/pages /var/lib/gitlab/shared/terraform_state /var/lib/gitlab/shared/packages" and "chown gitlab" these directories.
Missing these directories causes "Errno::ENOENT: No such file or directory" error on gitlab-backup.service

(This might be another bug?)

Got it.

In my case it was also a problem with /usr/share/webapps/gitlab/tmp owned by root. I had to change ownership to gitlab:gitlab and set mod to 755 to make the backup work again.

Merging your change to the repo. It will be available with the next gitlab build.

Please check gitlab-15.6.0 in the community-testing repo and let me know if you see any additional issues.

I confirmed the issue is resolved with gitlab-15.6.0-1, thank you!

---

- mitty@gitlab-tmp:~$ sudo vim /usr/lib/systemd/system/gitlab-backup.service
(revert modification)
- mitty@gitlab-tmp:~$ sudo systemctl daemon-reload
- mitty@gitlab-tmp:~$ sudo systemctl start gitlab-backup
(gitlab-backup.service returns to "Errno::EROFS: Read-only file system" error state)
- mitty@gitlab-tmp:~$ wget https://mirror.cmt.de/archlinux/community-testing/os/x86_64/gitlab-15.6.0-1-x86_64.pkg.tar.zst
- mitty@gitlab-tmp:~$ sudo pacman -U gitlab-15.6.0-1-x86_64.pkg.tar.zst
(exec rake db:migrate and restart gitlab services)
- mitty@gitlab-tmp:~$ sudo systemctl start gitlab-backup
=> Finished GitLab Backup process.

The issue still persists because /usr/share/webapps/gitlab/tmp is still owned by root:root in the package (as pointed out in https://bugs.archlinux.org/task/76630#comment213071 ).

Also why not just make /usr/share/webapps/gitlab/tmp a symlink to /var/lib/gitlab ? That's a writable dir for gitlab and /usr/share/webapps/gitlab/tmp/builds would automatically exist as expected by gitlab-backup.

Jakub, your proposal sounds ds good. Could you please share a patch for it?

Also if you see any additional issues feel free to make a patch, this will speed up the fixing process.

Actually, I noticed that in the past, around version 13, /usr/share/webapps/gitlab/tmp used to be a symlink to /var/tmp: https://github.com/archlinux/svntogit-community/commit/3688bc43637e9c179f965bde90ecc3772b308e74#diff-37538beb61ff63edebbf735dfcf39e5d732f49183d6beb097169d971875ca422L152

The /usr/share/webapps/gitlab/tmp/backups path is not needed at all, the correct path to the backups dir is configured in gitlab.yml. So I tested 'ln -fs /var/tmp "${pkgdir}${_appdir}/tmp"' and it fixes gitlab-backup.service too. The patch is attached.

Jakub, do we really need to drop "ReadWritePaths=<APPDIR>/tmp"?

I think "ReadWritePaths=<APPDIR>/tmp" does not make a difference when the target is a symlink?

Thank you, Jakub! Briefly tested the patch, and things look good so far.

I merged your commit to SVN except for the version bump. A rebuild for gitlab will be soon anyway and it will include the changes.

Looks good, thanks for taking care of it, Anatol.