At work we have our own Grafana Kubernetes operator for deploying Grafana instances on OpenShift clusters.
Since we are running the operator for our users, they cannot freely choose the version of Grafana themselves, but rather need to go through us.
GrafanaLabs frequently releases new version of their software (awesome!) and we want to make those versions available as soon as possible to our users.
Long story short, we want to update our operator every time a new Grafana version is released.
On GitHub one could use the recently acquired Dependabot for this purpose, but I’m not aware of a similar tool for GitLab.
So I decided to write some automation around this for our use-case.
#
Step 1: Fetch latest version and compare
The first step is obtaining the version string of latest available Grafana release.
This can be done through the Github API:
1
2
3
4
5
6
| all_releases="$(curl -s \
-H "Accept: application/vnd.github.v3+json" \
https://api.github.com/repos/grafana/grafana/releases)"
latest_release="$(echo "$all_releases" | jq '.[0].tag_name' | tr -d '"')"
# trim leading "v"
latest_release="${latest_release#v}"
|
Now the latest_release variable contains a string like 8.1.2.
Next we obtain the version currently pinned in our repository.
From where exactly this information comes depends on the repository structure, but generally speaking we want to extract a pattern like x.y.z from one of the files.
Unfortunately, while GNU grep has a regex engine for matching lines, it does not support printing only the regex capture group.
Instead, we can use sed for this purpose, as outlined in this StackOverflow answer.
1
2
3
| # Look for the line ` tag: "x.y.z"` in values.yaml and print `x.y.z` only
version_file=helm-charts/grafana/values.yaml
current_release="$(sed -nr 's/ tag: "([0-9]+\.[0-9]+\.[0-9]+)"/\1/p' "$version_file")"
|
We have the current version and the latest version, so next we need to compare the two.
Credits to Max Rittmüller for the following idea of comparing two semantic versions in Bash.
I like it because it is very readable and easy to extend (for example if the upstream software has version suffixes like -dev).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
| is_up_to_date() {
# Replace dots with spaces
current="$(echo $1 | tr . ' ')"
latest="$(echo $2 | tr . ' ')"
# So, we just need to extract each number like this:
current_patch=$(echo $current | awk '{print $3}')
current_minor=$(echo $current | awk '{print $2}')
current_major=$(echo $current | awk '{print $1}')
latest_patch=$(echo $latest | awk '{print $3}')
latest_minor=$(echo $latest | awk '{print $2}')
latest_major=$(echo $latest | awk '{print $1}')
# And now, we can simply compare the variables, like:
[ $latest_major -gt $current_major ] && return 1
[ $latest_minor -gt $current_minor ] && return 1
[ $latest_patch -gt $current_patch ] && return 1
return 0
}
if is_up_to_date "$current_release" "$latest_release"; then
echo "Up-to-date"
else
echo "$latest_release"
fi
|
Great, now we have a script that we can use to detect whether we need to update our Grafana operator.
#
Step 2: Update and commit
In the second step we will update version string stored in our repository.
If necessary, this step can also be used to update dependencies, bump other version numbers, update documentation etc.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
| # Set valid username so git doesn't complain
git config user.email "update.bot@example.com"
git config user.name "Update Bot"
BRANCH_NAME="update-${CI_PROJECT_NAME}-${CI_PIPELINE_ID}"
git checkout -b "${BRANCH_NAME}"
# Update Grafana version
sed -i "s/^ tag: \".*\"$/ tag: \"${latest_release}\"/" helm-charts/grafana/values.yaml
git add helm-charts/grafana/values.yaml
# [...] more update tasks
git commit -F - <<EOF
Bump Grafana version to ${latest_release}
https://grafana.com/docs/grafana/latest/release-notes/release-notes-$(echo "$latest_release" | tr . -)/
EOF
|
The script and comments are fairly self explanatory.
Once nice aspect in this case is the fact that we can rely on GrafanaLabs always publishing release notes under the same URL pattern, thus we can helpfully include this information in the commit and merge request – which we need to create next.
#
Step 3: Push code and create MR
In this final step, we push the newly created commit to the GitLab repository and create a merge request for the branch.
To be able to do this, we need to create a personal access token (PAT) in the GitLab account and add it to the CI/CD variables section of the repository.
The PAT needs the api permission for creating the merge request and write_repository permission for pushing code to the repository.
In my case, I put the PAT into the CI variable UPDATE_BOT_AUTH_TOKEN.
Then, the following snippet can be used to publish the changes.
All variables beginning with CI_ and GITLAB_ are automatically injected by GitLab, see Predefined environment variables).
1
2
3
4
5
6
7
8
9
10
11
12
13
| # Overwrite origin remote so we can push there (the default CI token can only read)
git remote set-url origin "https://update-bot:${UPDATE_BOT_AUTH_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git"
export GITLAB_TOKEN="$UPDATE_BOT_AUTH_TOKEN"
export GITLAB_HOST="$CI_SERVER_HOST"
glab mr create \
--assignee "$GITLAB_USER_LOGIN" \
--fill \
--label update,bot \
--push \
--remove-source-branch \
--target-branch master \
--yes
|
I was first considering to manually fiddle with the GitLab API, but luckily found the awesome glab CLI client for GitLab to be a much better (and maintained!) alternative.
It also supports managing GitLab issues, pipelines, labels, release and repositories.
#
Step 4: Combine everything in a scheduled pipeline
At this point we can combine all the code snippets into a coherent script and run it in a GitLab CI/CD pipeline.
.gitlab-ci.yml:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
| stages:
- update
Check for updates and create MR:
stage: update
image: example.com/image-with-git-curl-jq-glab:latest
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule"'
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" || $CI_PIPELINE_SOURCE == "push"'
when: manual
allow_failure: true # makes this job non-blocking
script:
# Set valid username so git doesn't complain
- git config user.email "update.bot@example.com"
- git config user.name "Update Bot"
# Overwrite origin remote so we can push there (the default CI token can only read)
- git remote set-url origin "https://update-bot:${UPDATE_BOT_AUTH_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git"
- ./check-version-and-create-mr.sh
|
check-version-and-create-mr.sh:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
| #!/bin/bash
# This script will check if a more recent Grafana release exists upstream,
# update the Grafana version specified in this repository
# and create a merge request for the change.
# Bash 'Strict Mode'
# http://redsymbol.net/articles/unofficial-bash-strict-mode
# https://github.com/xwmx/bash-boilerplate#bash-strict-mode
set -o nounset
set -o errexit
set -o pipefail
IFS=$'\n\t'
all_releases="$(curl -s \
-H "Accept: application/vnd.github.v3+json" \
https://api.github.com/repos/grafana/grafana/releases)"
latest_release="$(echo "$all_releases" | jq '.[0].tag_name' | tr -d '"')"
# trim leading "v"
latest_release="${latest_release#v}"
# use sed instead of grep because it just prints the regex capture group (instead of the whole line)
current_release="$(sed -nr 's/ tag: "([0-9]+\.[0-9]+\.[0-9]+)"/\1/p' helm-charts/grafana/values.yaml)"
is_up_to_date() {
# Replace dots with spaces
current="$(echo $1 | tr . ' ')"
latest="$(echo $2 | tr . ' ')"
# So, we just need to extract each number like this:
current_patch=$(echo $current | awk '{print $3}')
current_minor=$(echo $current | awk '{print $2}')
current_major=$(echo $current | awk '{print $1}')
latest_patch=$(echo $latest | awk '{print $3}')
latest_minor=$(echo $latest | awk '{print $2}')
latest_major=$(echo $latest | awk '{print $1}')
# And now, we can simply compare the variables, like:
[ $latest_major -gt $current_major ] && return 1
[ $latest_minor -gt $current_minor ] && return 1
[ $latest_patch -gt $current_patch ] && return 1
return 0
}
if is_up_to_date "$current_release" "$latest_release"; then
echo "Up-to-date"
exit 0
else
echo "Found newer version: '$latest_release'"
fi
BRANCH_NAME="update-${CI_PROJECT_NAME}-${CI_PIPELINE_ID}"
git checkout -b "${BRANCH_NAME}"
# Update Grafana version
sed -i "s/^ tag: \".*\"$/ tag: \"${latest_release}\"/" helm-charts/grafana/values.yaml
git add helm-charts/grafana/values.yaml
# [...] more update tasks
git commit -F - <<EOF
Bump Grafana version to ${latest_release}
https://grafana.com/docs/grafana/latest/release-notes/release-notes-$(echo "$latest_release" | tr . -)/
EOF
export GITLAB_TOKEN="$UPDATE_BOT_AUTH_TOKEN"
export GITLAB_HOST="$CI_SERVER_HOST"
glab mr create \
--assignee "$GITLAB_USER_LOGIN" \
--fill \
--label update,bot \
--push \
--remove-source-branch \
--target-branch master \
--yes
exit 0
|
Note: I initially had trouble with glab due to the following error message.
This turned out to be caused by an old version of Git (~1.8) and was resolved by using a base image with a more recent version of Git (~2.28).
fatal: ambiguous argument 'origin/master...update-12345': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Finally, after we put all the pieces in place, we set up a scheduled pipeline for it.
For our use-case, checking for updates once a day is totally sufficient, but this will vary depending on the upstream you are tracking.
#
Outlook
This post went through the details of building a custom update bot with GitLab CI, similar to Dependabot.
Since this automation is a common requirement in many repositories, it makes sense to build a more general-purpose solution for this, which can be reused across different repositories.
In a future post I will explore which are the right abstractions (version checking, updating etc.) for building such a solution.
Until then, happy updating!
#
References