One of the CI/CD-Pipelines I’m developing is supposed to push the tag of an image into a production system. See my post on deploying Docker Swarm versions with Puppet to find out more about this pipeline. The core of this task is basically just a simple SSH command, but before I push the tag to production I want to make sure this version of the image actually exists in Gitlab’s container registry.

A few assumptions in this post:

• CI_SERVER_HOST: URL of your Gitlab instance, e.g. gitlab.com
• CI_REGISTRY: URL of your Gitlab Container Registry instance, e.g. registry.gitlab.com
• ÌMAGE_NAME: relative path of your Docker image, e.g. jacksgt/hello-world
• ÌMAGE_TAG: the image version, e.g. v1.2.3

Note: all environment variables beginning with CI_ used in this post are automatically created by Gitlab-CI.

Usually, you can check whether an image-tag combibation is present in the registry with the following simple command:

curl --fail -lSL https://$CI_REGISTRY/v2/repositories/$IMAGE_NAME/tags/$IMAGE_TAG

This command will exit with 0 if the image-tag combination exists in the registry, otherwise it will fail with a non-zero exit code (as per https://stackoverflow.com/questions/50937857/check-if-docker-image-exists-in-cloud-repo).

However, the story gets a bit more complicated when you have a private repository, thus the images (and their metadata) is only accessible after authentication.

There are various scripts and snippets out there that show how to do it for DockerHub:

• https://stackoverflow.com/a/34351467
• https://success.docker.com/article/how-do-i-authenticate-with-the-v2-api

However, these did not work for Gitlab’s included registry since you need to obtain the authorization token from another URL. Gitlab gives us some slight hints on where to obtain this token in the WWW-Authenticate Header:

$ curl -i https://$CI_REGISTRY/v2/
HTTP/1.1 401 Unauthorized
Content-Type: application/json
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer realm="https://$GITLAB/jwt/auth",service="container_registry"
X-Content-Type-Options: nosniff
Date: Fri, 29 Nov 2019 18:28:13 GMT
Content-Length: 87

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

With this information - and after digging through Docker’s source code and looking at the Token Authentication Specification - I was finally able to find this excellent post by Pim Widdershoven which documented what these things mean and how to obtain the token. Read his post for the details.

Basically we need to query the following endpoint with our Gitlab credentials:

curl --user "$CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD" "https://$CI_SERVER_HOST/jwt/auth?offline_token=true&service=container_registry&scope=repository:$IMAGE_NAME:pull

Then we can use this token to authenticate ourselves to the registry and check the image:

curl -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Bearer $TOKEN" "https://$REGISTRY/v2/$IMAGE_NAME/manifests/$IMAGE_TAG"

Finally!

Combining all the above steps into a Gitlab-CI pipeline might look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# .gitlab-ci.yml
variables:
  IMAGE_NAME: 'jacksgt/hello-world'

deploy:
  image: debian:buster
  before_script:
    - apt update && apt install -y curl jq
  script:
    - 'export IMAGE_NAME=$(echo $CI_REGISTRY_IMAGE) | cut -d "/" -f2-)'
    - 'export GITLAB_TOKEN=$(curl -s --user "${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD}" "https://gitlab.cc-asp.fraunhofer.de/jwt/auth?offline_token=true&service=container_registry&scope=repository:${IMAGE_NAME}:pull" | jq -r ".token")'
    - 'if ! curl --fail -s -o /dev/null -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Bearer $GITLAB_TOKEN" "https://${CI_REGISTRY}/v2/${IMAGE_NAME}/manifests/${IMAGE_TAG}"; then echo ERROR ${IMAGE_NAME}:${IMAGE_TAG} does not exist; exit 1; fi'
    - 'echo Deploying ${IMAGE_NAME}:${IMAGE_TAG}'
  only:
    variables:
      - $IMAGE_TAG

Also mind all the single quotes (') around these shell statements since YAML is sensitive to colons (:) in strings. Otherwise you might get “Invalid YAML” errors from Gitlab: jobs:deploy:script config should be an array of strings or a string. For more information about Gitlab-CI pipelines, check their configuration reference.

I have also written a script that (given all of the above environment variables) retrieves the token and checks whether the image is present in the registry: gitlab-ci-check-image.sh and put it into a Docker Image, so it’s easy to re-use across multiple repositories (and makes the gitlab-ci.yml look a lot cleaner).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

#!/bin/bash
set -e;

if [ -z "$VERSION" ]; then
    echo "ERROR: No VERSION environment variable set";
    exit 1;
fi

IMAGE_NAME=$(echo $CI_REGISTRY_IMAGE | cut -d '/' -f2-)

export GITLAB_TOKEN=$(curl -s --user "${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD}" "https://${CI_SERVER_HOST}/jwt/auth?offline_token=true&service=container_registry&scope=repository:${IMAGE_NAME}:pull" | jq -r ".token")
if ! curl --fail -s -I -o /dev/null -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Bearer $GITLAB_TOKEN" "https://${CI_REGISTRY}/v2/${IMAGE_NAME}/manifests/${VERSION}"; then
    echo "ERROR: Image version ${IMAGE_NAME}:${VERSION} not found in registry $CI_REGISTRY";
    exit 1;
fi

exit 0;

Happy deploying!