FS#55899 - [gitlab] gitlab-gitaly fails to start
Attached to Project:
Community Packages
Opened by Tiago Peixoto (count0) - Saturday, 07 October 2017, 15:47 GMT
Last edited by Sven-Hendrik Haase (Svenstaro) - Saturday, 14 October 2017, 11:49 GMT
Opened by Tiago Peixoto (count0) - Saturday, 07 October 2017, 15:47 GMT
Last edited by Sven-Hendrik Haase (Svenstaro) - Saturday, 14 October 2017, 11:49 GMT
|
Details
Description:
The service gitlab-gitaly fails to start, rendering gitlab partially dysfunctional. The output from `systemctl status` reads: ● gitlab-gitaly.service - Gitaly is a Git RPC service for handling all the git calls made by GitLab. Loaded: loaded (/usr/lib/systemd/system/gitlab-gitaly.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Sat 2017-10-07 17:38:36 CEST; 5min ago Main PID: 3626 (code=exited, status=1/FAILURE) Oct 07 17:38:35 skewed.de systemd[1]: Started Gitaly is a Git RPC service for handling all the git calls made by GitLab.. Oct 07 17:38:35 skewed.de gitlab-gitaly[3626]: time="2017-10-07T17:38:35+02:00" level=info msg="Starting Gitaly" version="Gitaly, version 0.43.0, built 20171006.023852" Oct 07 17:38:35 skewed.de gitlab-gitaly[3626]: time="2017-10-07T17:38:35+02:00" level=warning msg="git path not configured. Using default path resolution" resolvedPath=/usr/sbin/git Oct 07 17:38:36 skewed.de gitlab-gitaly[3626]: time="2017-10-07T17:38:36+02:00" level=fatal msg="load config" config_path=/etc/gitlab-gitaly/config.toml error="exit status 1; stderr: \"/opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/spec_set.rb:87:in `block in materialize': Could not find rake-12.1.0 in any of the sources (Bundler::GemNotFound)\\n\\tfrom /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/spec_set.rb:81:in `map!'\\n\\tfrom /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/spec_set.rb:81:in `materialize'\\n\\tfrom /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/definition.rb:159:in `specs'\\n\\tfrom /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/definition.rb:218:in `specs_for'\\n\\tfrom /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/definition.rb:207:in `requested_specs'\\n\\tfrom /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/runtime.rb:109:in `block in definition_method'\\n\\tfrom /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/runtime.rb:21:in `setup'\\n\\tfrom /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler.rb:101:in `setup'\\n\\tfrom /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/setup.rb:19:in `<top (required)>'\\n\\tfrom /usr/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'\\n\\tfrom /usr/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'\\n\"" Oct 07 17:38:36 skewed.de systemd[1]: gitlab-gitaly.service: Main process exited, code=exited, status=1/FAILURE Oct 07 17:38:36 skewed.de systemd[1]: gitlab-gitaly.service: Unit entered failed state. Oct 07 17:38:36 skewed.de systemd[1]: gitlab-gitaly.service: Failed with result 'exit-code'. Furthermore, the gitlab environment seems inconsistent: $ su - gitlab -s /bin/sh -c "cd '/usr/share/webapps/gitlab'; bundle-2.3 exec rake gitlab:env:info RAILS_ENV=production" System information System: Current User: gitlab Using RVM: no Ruby Version: 2.4.2p198 Gem Version: /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/spec_set.rb:87:in `block in materialize': Could not find rake-12.0.0 in any of the sources (Bundler::GemNotFound) from /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/spec_set.rb:81:in `map!' from /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/spec_set.rb:81:in `materialize' from /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/definition.rb:159:in `specs' from /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/definition.rb:218:in `specs_for' from /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/definition.rb:207:in `requested_specs' from /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/runtime.rb:109:in `block in definition_method' from /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/runtime.rb:21:in `setup' from /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler.rb:101:in `setup' from /opt/ruby2.3/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/setup.rb:19:in `<top (required)>' from /usr/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require' Bundler Version:2.3 Rake Version: 12.0.0 Redis Version: 4.0.2 Git Version: 2.14.2 Sidekiq Version:5.0.4 Go Version: go1.9 linux/amd64 GitLab information Version: 10.0.3 Revision: 8895150 Directory: /usr/share/webapps/gitlab DB Adapter: postgresql URL: https://git.skewed.de HTTP Clone URL: https://git.skewed.de/some-group/some-project.git SSH Clone URL: gitlab@git.skewed.de:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: google_oauth2, twitter, github GitLab Shell Version: 5.9.0 Repository storage paths: - default: /var/lib/gitlab/repositories Hooks: /usr/share/webapps/gitlab-shell/hooks Git: /usr/bin/git Additional info: * package version: gitlab-10.0.3-1 |
This task depends upon
Closed by Sven-Hendrik Haase (Svenstaro)
Saturday, 14 October 2017, 11:49 GMT
Reason for closing: Fixed
Saturday, 14 October 2017, 11:49 GMT
Reason for closing: Fixed
Has it only started happening recently? Was it ever functional for you?
FS#55865.What is kind of strange: When checking the rake version, it says 12.0.0, but querying the gem version it shows the following error:
# su - gitlab -s /bin/sh -c "cd '/usr/share/webapps/gitlab'; bundle-2.3 exec rake --version"
rake, version 12.0.0
# su - gitlab -s /bin/sh -c "cd '/usr/share/webapps/gitlab'; bundle-2.3 exec gem --version"
Could not find rake-12.0.0 in any of the sources
Run `bundle install` to install missing gems.
FS#55865are not at fault. I just removed the changes and run into the same issue. The problem has manifested itself with the update to 0.43.0. I've been looking into this quite a bit but so far have come up empty-handed.I had ruby 2.4 installed next to ruby-2.3. I uninstalled ruby2.4, symlinked gem, rake, ruby and bundler "-2.3" to the ones without the suffix in /usr/bin. Afterwards, rake gitlab:env:info works fine and gitaly doesn't crash anymore.
The reason for the error in 'rake gitlab:env:info' is, that the command executed is "gem --version", see https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/tasks/gitlab/info.rake#L10. "gem" is not linked to the -2.3 version, therefore using the 2.4 one, which hasn't the dependencies installed.
I think the same goes for gitaly, at least from what I can see from the logs. The first two traces are from /usr/lib/ruby/2.4.0, the rest from /opt/ruby2.3/.
So, for a workaround see above. We now have to figure out why suddenly the environment uses the wrong ruby version initially. I hope this helps your investigation @Svenstaro.
Apparently, gitaly may need rake-12.1.0 EDIT: gitaly already has rake-12.1.0 according to
# bundle exec rake --version
rake, version 12.1.0
I only updated to gitaly to 0.46.0 and added this line to PKGBUILD:
sed -i "s/\"ruby\"/\"ruby-2.3\"/" internal/linguist/linguist.go
And I also updated gitlab-workhorse, without updating it, it throws this error, it was refactored in gitaly:
handleGetInfoRefs: GetInfoRefsHandler: copy Gitaly response: rpc error: code = 12 desc = unknown service gitaly.SmartHTTP