FS#76630 - [gitlab] gitlab-backup.service failed due to "ProtectSystem=full"

Attached to Project: Community Packages
Opened by Ken-ichi Mito (mittyorz) - Sunday, 20 November 2022, 09:56 GMT
Last edited by Toolybird (Toolybird) - Friday, 28 April 2023, 22:21 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Anatol Pomozov (anatolik)
Architecture x86_64
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:

When I run gitlab-backup through the systemd unit 'gitlab-backup.service', I've got

```
Nov 20 18:44:50 gitlab-tmp systemd[1]: Starting GitLab Backup process...
Nov 20 18:45:00 gitlab-tmp gitlab-backup[2172]: /usr/share/webapps/gitlab/lib/gitlab/instrumentation/redis.rb:9: warning: already initialized constant Gitlab::Instrumentation::Redis::ActionCable
Nov 20 18:45:00 gitlab-tmp gitlab-backup[2172]: /usr/share/webapps/gitlab/lib/gitlab/instrumentation/redis.rb:9: warning: previous definition of ActionCable was here
Nov 20 18:45:04 gitlab-tmp gitlab-backup[2172]: rake aborted!
Nov 20 18:45:04 gitlab-tmp gitlab-backup[2172]: Errno::EROFS: Read-only file system @ rb_sysopen - /usr/share/webapps/gitlab/tmp/backup_restore.pid
Nov 20 18:45:04 gitlab-tmp gitlab-backup[2172]: /usr/share/webapps/gitlab/lib/tasks/gitlab/backup.rake:184:in `initialize'
Nov 20 18:45:04 gitlab-tmp gitlab-backup[2172]: /usr/share/webapps/gitlab/lib/tasks/gitlab/backup.rake:184:in `open'
Nov 20 18:45:04 gitlab-tmp gitlab-backup[2172]: /usr/share/webapps/gitlab/lib/tasks/gitlab/backup.rake:184:in `lock'
Nov 20 18:45:04 gitlab-tmp gitlab-backup[2172]: /usr/share/webapps/gitlab/lib/tasks/gitlab/backup.rake:13:in `block (3 levels) in <top (required)>'
Nov 20 18:45:04 gitlab-tmp gitlab-backup[2172]: /usr/share/webapps/gitlab/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
Nov 20 18:45:04 gitlab-tmp gitlab-backup[2172]: /usr/share/webapps/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
Nov 20 18:45:04 gitlab-tmp gitlab-backup[2172]: Tasks: TOP => gitlab:backup:create
Nov 20 18:45:04 gitlab-tmp gitlab-backup[2172]: (See full trace by running task with --trace)
Nov 20 18:45:04 gitlab-tmp systemd[1]: gitlab-backup.service: Main process exited, code=exited, status=1/FAILURE
Nov 20 18:45:04 gitlab-tmp systemd[1]: gitlab-backup.service: Failed with result 'exit-code'.
Nov 20 18:45:04 gitlab-tmp systemd[1]: Failed to start GitLab Backup process.
```

Additional info:
* package version(s): 15.5.2-1
* config and/or log files etc.
* link to upstream bug report, if any

Steps to reproduce:

run 'gitlab-backup.service'

Solution (temporary):

make a drop-in file like
* /etc/systemd/system/gitlab-backup.service.d/readwrite-tmp.conf
```
[Service]
ReadWritePaths=/usr/share/webapps/gitlab/tmp
```

---

A lockfile mehachism was added on https://gitlab.com/gitlab-org/gitlab/-/commit/193741e4ff3e386eefabe7c8f3c88499457705b5,
and this commit was merged in Gitlab 15.5 https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97218.

Writing to /usr is prohibited by "ProtectSystem=full" in gitlab-backup.service
This task depends upon

Closed by  Toolybird (Toolybird)
Friday, 28 April 2023, 22:21 GMT
Reason for closing:  Fixed
Additional comments about closing:  gitlab 15.7.0-1
Comment by Anatol Pomozov (anatolik) - Monday, 21 November 2022, 04:32 GMT
I confirm I see this issue as well.

But adding ReadWritePaths=/usr/share/webapps/gitlab/tmp to the /usr/lib/systemd/system/gitlab-backup.service file does not really help.

Could you please share the /usr/lib/systemd/system/gitlab-backup.service that works for you without the drop-in file?
Comment by Ken-ichi Mito (mittyorz) - Tuesday, 22 November 2022, 11:29 GMT
In my environment (I made a brand-new Gitlab environment), /usr/lib/systemd/system/gitlab-backup.service is
```
[Unit]
Description=GitLab Backup process
Requires=
After=network.target

[Service]
Type=oneshot
User=gitlab
Group=gitlab
SyslogIdentifier=gitlab-backup
WorkingDirectory=/usr/share/webapps/gitlab
EnvironmentFile=/usr/share/webapps/gitlab/environment
CapabilityBoundingSet=
PrivateTmp=true
PrivateDevices=true
ProtectSystem=full
ProtectHome=true
NoNewPrivileges=true
ExecStart=/usr/bin/bundle-2.7 exec rake gitlab:backup:create
ReadWritePaths=/usr/share/webapps/gitlab/tmp
```

I just added "ReadWritePaths=/usr/share/webapps/gitlab/tmp" on the last of te service file.

-------

- mitty@gitlab-tmp:~$ sudo vim /usr/lib/systemd/system/gitlab-backup.service
```
--- /usr/lib/systemd/system/gitlab-backup.service.orig 2022-11-05 09:27:58.000000000 +0900
+++ /usr/lib/systemd/system/gitlab-backup.service 2022-11-22 20:25:07.136666805 +0900
@@ -17,3 +17,4 @@
ProtectHome=true
NoNewPrivileges=true
ExecStart=/usr/bin/bundle-2.7 exec rake gitlab:backup:create
+ReadWritePaths=/usr/share/webapps/gitlab/tmp
```
- mitty@gitlab-tmp:~$ sudo systemctl daemon-reload
- mitty@gitlab-tmp:~$ sudo systemctl start gitlab-backup
```
Nov 22 20:25:51 gitlab-tmp systemd[1]: Starting GitLab Backup process...
Nov 22 20:26:02 gitlab-tmp gitlab-backup[829]: /usr/share/webapps/gitlab/lib/gitlab/instrumentation/redis.rb:9: warning: already initialized constant Gitlab::Instrumentation::Redis::ActionCable
Nov 22 20:26:02 gitlab-tmp gitlab-backup[829]: /usr/share/webapps/gitlab/lib/gitlab/instrumentation/redis.rb:9: warning: previous definition of ActionCable was here
Nov 22 20:26:05 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:05 +0900 -- Dumping main_database ...
Nov 22 20:26:08 gitlab-tmp gitlab-backup[829]: Dumping PostgreSQL database gitlabhq_production ... [DONE]
Nov 22 20:26:08 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:08 +0900 -- Dumping main_database ... done
Nov 22 20:26:08 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:08 +0900 -- Dumping ci_database ... [DISABLED]
Nov 22 20:26:08 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:08 +0900 -- Dumping repositories ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: {"command":"create","gl_project_path":"gitlab-instance-7188cfa5/Monitoring","level":"info","msg":"started create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","storage_name":"default","time":"2022-11-22T11:26:09.251Z"}
Nov 22 20:26:09 gitlab-tmp gitlab-gitaly[351]: time="2022-11-22T11:26:09.265Z" level=info msg="finished unary call with code OK" command.count=1 command.cpu_time_ms=2 command.inblock=32 command.majflt=0 command.maxrss=308976 command.minflt=144 command.oublock=0 command.real_time_ms=3 command.system_time_ms=2 command.user_time_ms=0 correlation_id=01GJFHD6BCHVE7VQCCS29TM69K grpc.code=OK grpc.meta.deadline_type=none grpc.meta.method_type=unary grpc.method=HasLocalBranches grpc.request.fullMethod=/gitaly.RepositoryService/HasLocalBranches grpc.request.glProjectPath=gitlab-instance-7188cfa5/Monitoring grpc.request.glRepository= grpc.request.payload_bytes=133 grpc.request.repoPath=@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git grpc.request.repoStorage=default grpc.response.payload_bytes=0 grpc.service=gitaly.RepositoryService grpc.start_time="2022-11-22T20:26:09.260" grpc.time_ms=4.511 peer.address=@ pid=351 span.kind=server system=grpc
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: {"command":"create","error":"manager: repository empty: repository skipped","gl_project_path":"gitlab-instance-7188cfa5/Monitoring","level":"warning","msg":"skipped create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","storage_name":"default","time":"2022-11-22T11:26:09.266Z"}
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: {"command":"create","gl_project_path":"gitlab-instance-7188cfa5/Monitoring.wiki","level":"info","msg":"started create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git","storage_name":"default","time":"2022-11-22T11:26:09.388Z"}
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: {"command":"create","gl_project_path":"gitlab-instance-7188cfa5/Monitoring","level":"info","msg":"started create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.design.git","storage_name":"default","time":"2022-11-22T11:26:09.390Z"}
Nov 22 20:26:09 gitlab-tmp gitlab-gitaly[351]: time="2022-11-22T11:26:09.390Z" level=info msg="finished unary call with code OK" command.count=1 command.cpu_time_ms=1 command.inblock=0 command.majflt=0 command.maxrss=308976 command.minflt=146 command.oublock=0 command.real_time_ms=1 command.system_time_ms=0 command.user_time_ms=1 correlation_id=01GJFHD6FCMJK21S5A8SXVH623 grpc.code=OK grpc.meta.deadline_type=none grpc.meta.method_type=unary grpc.method=HasLocalBranches grpc.request.fullMethod=/gitaly.RepositoryService/HasLocalBranches grpc.request.glProjectPath=gitlab-instance-7188cfa5/Monitoring.wiki grpc.request.glRepository= grpc.request.payload_bytes=143 grpc.request.repoPath=@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git grpc.request.repoStorage=default grpc.response.payload_bytes=0 grpc.service=gitaly.RepositoryService grpc.start_time="2022-11-22T20:26:09.388" grpc.time_ms=2.239 peer.address=@ pid=351 span.kind=server system=grpc
Nov 22 20:26:09 gitlab-tmp gitlab-gitaly[351]: time="2022-11-22T11:26:09.391Z" level=info msg="finished unary call with code NotFound" correlation_id=01GJFHD6FFRBG59HNGEZDAN790 error="GetRepoPath: not a git repository: \"/var/lib/gitlab/repositories/@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.design.git\"" grpc.code=NotFound grpc.meta.deadline_type=none grpc.meta.method_type=unary grpc.method=HasLocalBranches grpc.request.fullMethod=/gitaly.RepositoryService/HasLocalBranches grpc.request.glProjectPath=gitlab-instance-7188cfa5/Monitoring grpc.request.glRepository= grpc.request.payload_bytes=140 grpc.request.repoPath=@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.design.git grpc.request.repoStorage=default grpc.response.payload_bytes=0 grpc.service=gitaly.RepositoryService grpc.start_time="2022-11-22T20:26:09.391" grpc.time_ms=0.165 peer.address=@ pid=351 span.kind=server system=grpc
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: {"command":"create","error":"manager: repository empty: repository skipped","gl_project_path":"gitlab-instance-7188cfa5/Monitoring.wiki","level":"warning","msg":"skipped create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git","storage_name":"default","time":"2022-11-22T11:26:09.391Z"}
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: {"command":"create","error":"manager: repository empty: repository skipped","gl_project_path":"gitlab-instance-7188cfa5/Monitoring","level":"warning","msg":"skipped create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.design.git","storage_name":"default","time":"2022-11-22T11:26:09.391Z"}
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping repositories ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping uploads ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping uploads ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping builds ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping builds ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping artifacts ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping artifacts ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping pages ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping pages ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping lfs objects ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping lfs objects ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping terraform states ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping terraform states ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping container registry images ... [DISABLED]
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping packages ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Dumping packages ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Creating backup archive: 1669116365_2022_11_22_15.5.2_gitlab_backup.tar ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Creating backup archive: 1669116365_2022_11_22_15.5.2_gitlab_backup.tar ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Uploading backup archive to remote storage ... [SKIPPED]
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Deleting old backups ... [SKIPPED]
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Deleting tar staging files ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/backup_information.yml
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/db
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/uploads.tar.gz
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/builds.tar.gz
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/artifacts.tar.gz
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/pages.tar.gz
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/lfs.tar.gz
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/terraform_state.tar.gz
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Cleaning up /var/lib/gitlab/backups/packages.tar.gz
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Deleting tar staging files ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Deleting backups/tmp ...
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Deleting backups/tmp ... done
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: and are not included in this backup. You will need these files to restore a backup.
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: Please back them up manually.
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Backup 1669116365_2022_11_22_15.5.2 is done.
Nov 22 20:26:09 gitlab-tmp gitlab-backup[829]: 2022-11-22 20:26:09 +0900 -- Deleting backup and restore lock file
Nov 22 20:26:09 gitlab-tmp systemd[1]: gitlab-backup.service: Deactivated successfully.
Nov 22 20:26:09 gitlab-tmp systemd[1]: Finished GitLab Backup process.
```
Comment by Ken-ichi Mito (mittyorz) - Tuesday, 22 November 2022, 11:33 GMT
You may need to "mkdir /var/lib/gitlab/shared/pages /var/lib/gitlab/shared/terraform_state /var/lib/gitlab/shared/packages" and "chown gitlab" these directories.
Missing these directories causes "Errno::ENOENT: No such file or directory" error on gitlab-backup.service

(This might be another bug?)
Comment by Anatol Pomozov (anatolik) - Tuesday, 22 November 2022, 16:35 GMT
Got it.

In my case it was also a problem with /usr/share/webapps/gitlab/tmp owned by root. I had to change ownership to gitlab:gitlab and set mod to 755 to make the backup work again.

Merging your change to the repo. It will be available with the next gitlab build.
Comment by Anatol Pomozov (anatolik) - Tuesday, 22 November 2022, 19:36 GMT
Please check gitlab-15.6.0 in the community-testing repo and let me know if you see any additional issues.
Comment by Ken-ichi Mito (mittyorz) - Wednesday, 23 November 2022, 14:03 GMT
I confirmed the issue is resolved with gitlab-15.6.0-1, thank you!

---

- mitty@gitlab-tmp:~$ sudo vim /usr/lib/systemd/system/gitlab-backup.service
(revert modification)
- mitty@gitlab-tmp:~$ sudo systemctl daemon-reload
- mitty@gitlab-tmp:~$ sudo systemctl start gitlab-backup
(gitlab-backup.service returns to "Errno::EROFS: Read-only file system" error state)
- mitty@gitlab-tmp:~$ wget https://mirror.cmt.de/archlinux/community-testing/os/x86_64/gitlab-15.6.0-1-x86_64.pkg.tar.zst
- mitty@gitlab-tmp:~$ sudo pacman -U gitlab-15.6.0-1-x86_64.pkg.tar.zst
(exec rake db:migrate and restart gitlab services)
- mitty@gitlab-tmp:~$ sudo systemctl start gitlab-backup
=> Finished GitLab Backup process.
Comment by Jakub Klinkovský (lahwaacz) - Saturday, 10 December 2022, 21:52 GMT
  • Field changed: Percent Complete (100% → 0%)
The issue still persists because /usr/share/webapps/gitlab/tmp is still owned by root:root in the package (as pointed out in https://bugs.archlinux.org/task/76630#comment213071 ).

Also why not just make /usr/share/webapps/gitlab/tmp a symlink to /var/lib/gitlab ? That's a writable dir for gitlab and /usr/share/webapps/gitlab/tmp/builds would automatically exist as expected by gitlab-backup.
Comment by Anatol Pomozov (anatolik) - Saturday, 10 December 2022, 22:01 GMT
Jakub, your proposal sounds ds good. Could you please share a patch for it?

Also if you see any additional issues feel free to make a patch, this will speed up the fixing process.
Comment by Jakub Klinkovský (lahwaacz) - Sunday, 11 December 2022, 11:53 GMT
Actually, I noticed that in the past, around version 13, /usr/share/webapps/gitlab/tmp used to be a symlink to /var/tmp: https://github.com/archlinux/svntogit-community/commit/3688bc43637e9c179f965bde90ecc3772b308e74#diff-37538beb61ff63edebbf735dfcf39e5d732f49183d6beb097169d971875ca422L152

The /usr/share/webapps/gitlab/tmp/backups path is not needed at all, the correct path to the backups dir is configured in gitlab.yml. So I tested 'ln -fs /var/tmp "${pkgdir}${_appdir}/tmp"' and it fixes gitlab-backup.service too. The patch is attached.
Comment by Anatol Pomozov (anatolik) - Sunday, 11 December 2022, 19:51 GMT
Jakub, do we really need to drop "ReadWritePaths=<APPDIR>/tmp"?
Comment by Jakub Klinkovský (lahwaacz) - Sunday, 11 December 2022, 20:04 GMT
I think "ReadWritePaths=<APPDIR>/tmp" does not make a difference when the target is a symlink?
Comment by Anatol Pomozov (anatolik) - Sunday, 11 December 2022, 21:01 GMT
Thank you, Jakub! Briefly tested the patch, and things look good so far.

I merged your commit to SVN except for the version bump. A rebuild for gitlab will be soon anyway and it will include the changes.
Comment by Jakub Klinkovský (lahwaacz) - Sunday, 11 December 2022, 21:22 GMT
Looks good, thanks for taking care of it, Anatol.

Loading...