FS#58352 - gitlab-gitaly.service failure at startup

Attached to Project: Community Packages
Opened by Lauri Annala (noxoriaz) - Wednesday, 25 April 2018, 08:55 GMT
Last edited by Eli Schwartz (eschwartz) - Thursday, 26 April 2018, 17:35 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To No-one
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

Description:
Gitlab cannot be started because of its dependency startup failure in Gitlab-gitaly.

Additional info:

* package version(s)

10.7.0-3

* config and/or log files etc.

# systemctl status gitlab.target
systemd[1]: Dependency failed for GitLab - Self Hosted Git Management.
systemd[1]: gitlab.target: Job gitlab.target/start failed with result 'dependency'.

Gitlab-gitaly service fails:
# systemctl status gitlab-gitaly.service
gitlab-gitaly[1876]: time="2018-04-25T11:31:49Z" level=info msg="Failed to dial /tmp/gitaly-ruby049584799/socket.1: context canceled; please retry." system=system
systemd[1]: Stopping Gitaly is a Git RPC service for handling all the git calls made by GitLab....
gitlab-gitaly[1876]: time="2018-04-25T11:31:59Z" level=warning msg=exited error="signal: killed" supervisor.args="[bundle-2.3 exec bin/ruby-cd /usr/share/webapps/gitlab /usr/share/webapps/gitlab-gitaly/ruby/bin/gitaly-ruby 1876 /tmp/gitaly-ruby049584799/socket.0]" supervisor.name=gitaly-ruby.0
gitlab-gitaly[1876]: time="2018-04-25T11:31:59Z" level=warning msg=exited error="signal: killed" supervisor.args="[bundle-2.3 exec bin/ruby-cd /usr/share/webapps/gitlab /usr/share/webapps/gitlab-gitaly/ruby/bin/gitaly-ruby 1876 /tmp/gitaly-ruby049584799/socket.1]" supervisor.name=gitaly-ruby.1
gitlab-gitaly[1876]: time="2018-04-25T11:31:59Z" level=fatal msg="shutting down" error="received signal \"terminated\""

Gitlab-unicorn service fails (attempts to restart on failure too many times)
systemd[1]: gitlab-unicorn.service: Service hold-off time over, scheduling restart.
systemd[1]: gitlab-unicorn.service: Scheduled restart job, restart counter is at 5.
systemd[1]: Stopped GitLab Unicorn Server.
systemd[1]: gitlab-unicorn.service: Start request repeated too quickly.
systemd[1]: gitlab-unicorn.service: Failed with result 'exit-code'.
systemd[1]: Failed to start GitLab Unicorn Server.

Gitlab-gitaly has now failed on calling the socket. And if I now attempt to run this service individually:
# systemctl start gitlab-gitaly.service
gitlab-gitaly[3025]: time="2018-04-25T11:42:57Z" level=warning msg=spawned supervisor.args="[bundle-2.3 exec bin/ruby-cd /usr/share/webapps/gitlab /usr/share/webapps/gitlab-gitaly/ruby/bin/gitaly-ruby 3025 /tmp/gi>
gitlab-gitaly[3025]: time="2018-04-25T11:42:57Z" level=info msg="starting RSS monitor" supervisor.name=gitaly-ruby.0 supervisor.rss_threshold=314572800
gitlab-gitaly[3025]: time="2018-04-25T11:42:57Z" level=info msg="dialing to target with scheme: \"\"" system=system
gitlab-gitaly[3025]: time="2018-04-25T11:42:57Z" level=info msg="ccResolverWrapper: sending new addresses to cc: [{/tmp/gitaly-ruby696648417/socket.0 0 <nil>}]" system=system
gitlab-gitaly[3025]: time="2018-04-25T11:42:57Z" level=info msg="ClientConn switching balancer to \"pick_first\"" system=system
gitlab-gitaly[3025]: time="2018-04-25T11:42:57Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4202621f0, CONNECTING" system=system
gitlab-gitaly[3025]: time="2018-04-25T11:42:57Z" level=info msg="blockingPicker: the picked transport is not ready, loop back to repick" system=system
gitlab-gitaly[3025]: time="2018-04-25T11:42:57Z" level=info msg="grpc: addrConn.createTransport failed to connect to {/tmp/gitaly-ruby696648417/socket.0 0 <nil>}. Err :connection error: desc = \"transport: Error >
gitlab-gitaly[3025]: time="2018-04-25T11:42:57Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4202621f0, TRANSIENT_FAILURE" system=system
gitlab-gitaly[3025]: time="2018-04-25T11:42:57Z" level=info msg="Failed to dial /tmp/gitaly-ruby696648417/socket.0: context canceled; please retry." system=system

Steps to reproduce:
I have no custom configuration for gitlab-gitaly, please let me know if more details are needed.
Start gitlab and examine the results in failing dependency:
#systemctl start gitlab.target
#systemctl status gitlab-gitaly.service
This task depends upon

Closed by  Eli Schwartz (eschwartz)
Thursday, 26 April 2018, 17:35 GMT
Reason for closing:  Not a bug
Additional comments about closing:  OP request: Cannot reproduce after updating to latest version: 10.7.1-1.
Comment by Lauri Annala (noxoriaz) - Thursday, 26 April 2018, 04:07 GMT
After updating to version 10.7.1-1 I can no longer reproduce this issue, gitlab works for me again:

gitlab-gitaly[11233]: time="2018-04-26T06:58:13Z" level=info msg="dialing to target with scheme: \"\"" system=system
gitlab-gitaly[11233]: time="2018-04-26T06:58:13Z" level=info msg="ccResolverWrapper: sending new addresses to cc: [{/tmp/gitaly-ruby833155858/socket.0 0 <nil>}]" system=system
gitlab-gitaly[11233]: time="2018-04-26T06:58:13Z" level=info msg="ClientConn switching balancer to \"pick_first\"" system=system
gitlab-gitaly[11233]: time="2018-04-26T06:58:13Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4202160e0, CONNECTING" system=system
gitlab-gitaly[11233]: time="2018-04-26T06:58:13Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4202160e0, READY" system=system
gitlab-gitaly[11233]: time="2018-04-26T06:58:13Z" level=info msg="dialing to target with scheme: \"\"" system=system
gitlab-gitaly[11233]: time="2018-04-26T06:58:13Z" level=info msg="ccResolverWrapper: sending new addresses to cc: [{/tmp/gitaly-ruby833155858/socket.1 0 <nil>}]" system=system
gitlab-gitaly[11233]: time="2018-04-26T06:58:13Z" level=info msg="ClientConn switching balancer to \"pick_first\"" system=system
gitlab-gitaly[11233]: time="2018-04-26T06:58:13Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4202164a0, CONNECTING" system=system
gitlab-gitaly[11233]: time="2018-04-26T06:58:13Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4202164a0, READY" system=system

Loading...