FS#49497 - [gitlab-ci-multi-runner] Building with docker results in fatal "no such image" error

Attached to Project: Community Packages
Opened by Melvin Vermeeren (melvinvermeeren) - Saturday, 28 May 2016, 15:24 GMT
Last edited by Sven-Hendrik Haase (Svenstaro) - Tuesday, 05 July 2016, 16:44 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Sven-Hendrik Haase (Svenstaro)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 3
Private No

Details

Description:
gitlab-ci-multi-runner is inoperable using the docker executioner.

Additional info:
* docker 1.11.1-1
* gitlab-ci-multi-runner 1.2.0-1
* gitlab 8.8.2-3 (fresh install)


Steps to reproduce:
* Install ci runner
* Add gitlab-runner user to docker group
* sudo -u gitlab-runner docker info works properly
* Attach to gitlab instance with docker executioner type
* All builds fail

GitLab build log:
> gitlab-ci-multi-runner 1.2.0 (3a4fcd4)
> Using Docker executor with image debian:testing ...
> ERROR: Failed to create container volume for /cache no such image
> ERROR: Failed to create container volume for /builds/melvin no such image
>
> ERROR: Build failed: no such image

It seems like there is a mismatch between supplied docker images and the one the runner requests.

In the docker.service log:
> "Handler for GET /v1.18/images/gitlab-runner-build:3a4fcd4/json returned error: No such image: gitlab-runner-build:3a4fcd4"
> "Handler for GET /v1.18/images/gitlab-runner-cache:3a4fcd4/json returned error: No such image: gitlab-runner-cache:3a4fcd4"

However, running "docker images" gives:
> REPOSITORY TAG IMAGE ID CREATED SIZE
> gitlab-runner-build 9e2fd1a f4595b443011 13 days ago 43.75 MB
> gitlab-runner-service 9e2fd1a 3ad17ec10f22 12 weeks ago 4.79 MB
> gitlab-runner-cache 9e2fd1a 1cde79874428 12 weeks ago 1.114 MB

It could be me being inexperienced with docker, but it seems like the worker requests version/tag 3a4fcd4 even though only 9e2fd1a is installed, resulting in this message.
This task depends upon

Closed by  Sven-Hendrik Haase (Svenstaro)
Tuesday, 05 July 2016, 16:44 GMT
Reason for closing:  Fixed
Comment by Melvin Vermeeren (melvinvermeeren) - Saturday, 28 May 2016, 15:29 GMT
I can confirm that manually adding the requested tag to the installed tag causes the builds to start successfully.

docker tag gitlab-runner-build:9e2fd1a gitlab-runner-build:3a4fcd4
docker tag gitlab-runner-service:9e2fd1a gitlab-runner-service:3a4fcd4
docker tag gitlab-runner-cache:9e2fd1a gitlab-runner-cache:3a4fcd4
Comment by Sven-Hendrik Haase (Svenstaro) - Saturday, 28 May 2016, 16:27 GMT
How can this be fixed from a packaging perspective?
Comment by Melvin Vermeeren (melvinvermeeren) - Saturday, 28 May 2016, 17:52 GMT
The cheap fix would be running the 3 docker tag commands, but I believe those only get created after gitlab-runner has been started in docker mode at least once which makes it impossible to easily add them to the install script.

I found some sources that indicated that in the past this was a problem too when runner versions and prebuilt docker image versions mismatched, but with the current PKGBUILD that is clearly no longer the case.

This might be something upstream, I'm not entirely sure. But the runner itself identifies as commit 3a4fcd4, yet somehow the installed docker images target commit 9e2fd1a.

https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/commit/3a4fcd4
https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/commit/9e2fd1a

Could it be that they retagged a release later in time?

It seems like upstream made a commit changing version to 1.3.0, but it's not officially tagged yet. Hopefully it will magically fix itself when the next version is released.

For now however, you might want to add a warning message about this specific issue (specific to docker builds) and link to this FS or something similar.

Thanks for all the work on gitlab so far by the way. :)
Comment by Danilo (danilo) - Monday, 06 June 2016, 09:07 GMT Comment by sathors (Sathors) - Friday, 10 June 2016, 02:41 GMT
I have opened the issue https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1243.
It seems that the problem was that the docker images did not correspond to the package version, but were always using the last one.

Now that it is resolved, we still have a similar problem.
From the gitlab makefile, it seems that the image tag comes from the git revision number. The thing is that is does not correspond to the tags set in the file at https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/v${pkgver}/docker/prebuilt.tar.gz.

As stated in the gitlab issue, don't you think it would be possible to compile the docker images instead of downloading them?

I tried to tinker with it, but without luck.

A dirty trick would be to run the commands Melvin gave us above, which resolved it for me.

Loading...