FS#78036 - [gitlab] pulling over https produces error 500

Attached to Project: Community Packages
Opened by Oliver Mangold (omangold) - Wednesday, 29 March 2023, 14:42 GMT
Last edited by Anatol Pomozov (anatolik) - Thursday, 30 March 2023, 15:52 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Anatol Pomozov (anatolik)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:

Since upgrading to 15.10.0 I cannot pull git repos via https.

git clone says:

---
error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500
fatal: expected flush after ref listing
---

Server-side workhorse says:

---
Mar 29 15:39:41 gitlab-workhorse[1826234]: gitlab.**REDACTED* 127.0.0.1 - - [2023/03/29:15:39:41 +0200] "POST /**REDACTED*/**REDACTED*.git/git-upload-pack HTTP/1.0" 500 0 "" "git/2.40.0" 140
Mar 29 15:39:41 gitlab-workhorse[1826234]: time="2023-03-29T15:39:41+02:00" level=error correlation_id=01GWPSGZ9THJJDJ33RP1QN4A7D error="handleUploadPack: smarthttp.UploadPack: rpc error: code = Unimplemented desc = unknown method PostUploadPack for service gitaly.SmartHTTPService" method=POST uri=/**REDACTED*/**REDACTED*.git/git-upload-pack
Mar 29 15:39:40 gitlab-workhorse[1826234]: gitlab.**REDACTED* 127.0.0.1 - - [2023/03/29:15:39:40 +0200] "GET /**REDACTED*/**REDACTED*.git/info/refs?service=git-upload-pack HTTP/1.0" 200 160 "" "git/2.40.0" 196
---

Additional info:

* Package versions: gitlab-*: 15.10.0-1

Steps to reproduce:

do a git clone via https
This task depends upon

Closed by  Anatol Pomozov (anatolik)
Thursday, 30 March 2023, 15:52 GMT
Reason for closing:  Fixed
Additional comments about closing:  gitlab-15.10.0-2
Comment by loqs (loqs) - Wednesday, 29 March 2023, 15:10 GMT
What if gitlab-workhorse is switched to use the bundled version? Please try rebuilding gitlab with the attached patch. Note the generated package replaces both gitlab and gitlab-workhorse.
   PKGBUILD.diff (4.8 KiB)
Comment by Oliver Mangold (omangold) - Wednesday, 29 March 2023, 15:29 GMT
Hmm, there seems to something a bit wrong when trying that PKGBUILD:

---
Missing Rails.application.secrets.secret_key_base for production environment. The secret will be generated and stored in config/secrets.yml.
Missing Rails.application.secrets.otp_key_base for production environment. The secret will be generated and stored in config/secrets.yml.
Missing Rails.application.secrets.db_key_base for production environment. The secret will be generated and stored in config/secrets.yml.
Missing Rails.application.secrets.openid_connect_signing_key for production environment. The secret will be generated and stored in config/secrets.yml.
WARNING: This version of GitLab depends on gitlab-shell 14.18.0, but you're running 14.15.0. Please update gitlab-shell.
*REDACTED*/gitlab/trunk/src/gitlab-foss/lib/gitlab/instrumentation/redis.rb:9: warning: already initialized constant Gitlab::Instrumentation::Redis::ActionCable
*REDACTED*/gitlab/trunk/src/gitlab-foss/lib/gitlab/instrumentation/redis.rb:9: warning: previous definition of ActionCable was here
rake aborted!
Errno::EEXIST: File exists @ syserr_fail2_in - /usr/share/webapps/gitlab-shell/.gitlab_shell_secret
*REDACTED*/gitlab/trunk/src/gitlab-foss/lib/gitlab/shell.rb:78:in `generate_and_link_secret_token'
*REDACTED*/gitlab/trunk/src/gitlab-foss/lib/gitlab/shell.rb:35:in `ensure_secret_token!'
*REDACTED*/gitlab/trunk/src/gitlab-foss/config/initializers/gitlab_shell_secret_token.rb:3:in `<top (required)>'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/railties-6.1.7.2/lib/rails/engine.rb:681:in `load'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/railties-6.1.7.2/lib/rails/engine.rb:681:in `block in load_config_initializer'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/notifications.rb:205:in `instrument'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/railties-6.1.7.2/lib/rails/engine.rb:680:in `load_config_initializer'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/railties-6.1.7.2/lib/rails/engine.rb:634:in `block (2 levels) in <class:Engine>'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/railties-6.1.7.2/lib/rails/engine.rb:633:in `each'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/railties-6.1.7.2/lib/rails/engine.rb:633:in `block in <class:Engine>'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/railties-6.1.7.2/lib/rails/initializable.rb:32:in `instance_exec'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/railties-6.1.7.2/lib/rails/initializable.rb:32:in `run'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/railties-6.1.7.2/lib/rails/initializable.rb:61:in `block in run_initializers'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/railties-6.1.7.2/lib/rails/initializable.rb:50:in `each'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/railties-6.1.7.2/lib/rails/initializable.rb:50:in `tsort_each_child'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/railties-6.1.7.2/lib/rails/initializable.rb:60:in `run_initializers'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/railties-6.1.7.2/lib/rails/application.rb:391:in `initialize!'
*REDACTED*/gitlab/trunk/src/gitlab-foss/config/environment.rb:7:in `<top (required)>'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/dependencies.rb:332:in `block in require'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/dependencies.rb:299:in `load_dependency'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/dependencies.rb:332:in `require'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/railties-6.1.7.2/lib/rails/application.rb:367:in `require_environment!'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/railties-6.1.7.2/lib/rails/application.rb:533:in `block in run_tasks_blocks'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:24:in `execute'
*REDACTED*/gitlab/trunk/src/gitlab-foss/lib/tasks/gettext.rake:10:in `block (2 levels) in <top (required)>'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:24:in `execute'
*REDACTED*/gitlab/trunk/src/gitlab-foss/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
Tasks: TOP => gettext:po_to_json => environment
(See full trace by running task with --trace)
```

The original problem, it couldn't be that it has something to do with using ruby 2.7? The gitlab manual says, only >=3.0 is supported.
Comment by Oliver Mangold (omangold) - Wednesday, 29 March 2023, 16:47 GMT
Accidental Repost.
Comment by loqs (loqs) - Wednesday, 29 March 2023, 16:47 GMT
The patched package builds for me in a clean chroot. Attached is the source tarball with patch pre-applied. The built package I produced is linked below [1].

Ruby 3 is complicated by the stdgems being de-vendored [2], gitlab only supports version 3 of the psych gem which the vendored psych 3.3.2 supplied by ruby 3.0.5 satisfies but the packaged ruby-psych 4.0.6 does not which breaks the build. When ruby is updated to 3.2 gitlab can use a new ruby3.0 package.

[1] https://drive.google.com/file/d/1vKkq6EXDIWKLoiNom8srpMrqCzi7LDNw/view?usp=share_link gitlab-15.10.0-1.1-x86_64.pkg.tar.zst
[2] https://lists.archlinux.org/archives/list/arch-dev-public%40lists.archlinux.org/thread/24P6GYY43AVP7DB4LVDQ2M6B4TOAXKF2/
(application/x-gzip)    gitlab-15.10.0-1.1.src.tar.gz (10.7 KiB)
Comment by Oliver Mangold (omangold) - Wednesday, 29 March 2023, 16:48 GMT
Accidental Repost.
Comment by Oliver Mangold (omangold) - Wednesday, 29 March 2023, 16:58 GMT
Ah, with this package it indeed works again. Thanks a lot.
Comment by Anatol Pomozov (anatolik) - Thursday, 30 March 2023, 03:18 GMT
Why does gitlab have 2 workhorse projects?

https://gitlab.com/gitlab-org/gitlab-workhorse
https://gitlab.com/gitlab-org/gitlab/-/tree/master/workhorse

EDIT, it looks like https://gitlab.com/gitlab-org/gitlab-workhorse is the legacy location and should not be used anymore. In this case gitlab-workhorse much be merged into gitlab package.
Comment by Anatol Pomozov (anatolik) - Thursday, 30 March 2023, 04:14 GMT
I am pushing gitlab-15.10.0-2 to community-testing. It replaces standalone gitlab-workhorse package with the embedded workhorse.

Oliver, please pull the testing version and let me know if you still have any issues.
Comment by Oliver Mangold (omangold) - Thursday, 30 March 2023, 06:18 GMT
Yes. That testing version also seems to work.
Comment by Anatol Pomozov (anatolik) - Thursday, 30 March 2023, 15:51 GMT
Thank you loqs!

Loading...