Building a custom OKD Machine OS image
Kubernetes and OpenShift are highly complex solutions that allow you to work in the cloud. But sometimes you also need to come back down to earth again, for example when you have instability or data corruption issues with your kernel (see Fedora CoreOS Tracker #957).
Then, you need to wade through layers of abstraction to understand how you can perform a simple task like changing the kernel of your operation system. OKD (the open-source project of OpenShift) comes with custom version of Fedora CoreOS (FCOS) as the base operating systems for its nodes. One of the special attributes of FCOS is its immutable filesystem which is implemented with rpm-ostree (a CoreOS project). The entire operating system is composed by “layers”, much like the layers of a container image. While this sounds great and desirable, it soundly makes (seemingly) simple tasks like building an image with your kernel a multi-day tasks of figuring out which tools you need to use, what the build environment needs to look like and which commands need to be run. After reading lots of fluffy documentation, makefiles, build scripts and CI configurations, we eventually managed to pin down the steps needed to build an image for OKD / OpenShift nodes (and also how to customize it according to our needs).
The following procduce describes the required steps along with some explanation of what’s happening (at least the parts that I could understand).
It is based on the convoluted build process of openshift/okd-machine-os, which itself is built on the fedora-coreos-config repo.
It assumes that you have working environment for podman and buildah.
# Clone the okd-machine-os repo
The first step is to obtain a local copy of the okd-machine-os repository and fetch all Git submodules.
git clone --recurse-submodules https://github.com/openshift/okd-machine-os
cd okd-machine-os
Note: If you didn’t run the first command with –recurse-submodules, fetch the submodules by executing the following command inside the repo:
git submodule update --init
# Use the appropriate OKD release
To avoid introducing more difference between the currently running version and the new version (e.g. miscellaneous tooling and library updates), checkout the right branch for your OKD release (in this example: 4.7):
git checkout release-4.7
# Apply changes
Now it’s time to make the required changes to the repository.
For example, to select (override) a specific version of a package, add it to manifest-lock.overrides.yaml.
Example:
| |
# Build environment
Next, we need to build the container image for the build environment (The build environment is based on CoreOS Assembler (coas)) with the Dockerfile.cosa.
buildah bud -t okd-machine-os.cosa -f Dockerfile.cosa .
podman run --rm --privileged -it --entrypoint /bin/sh -v /dev/kvm:/dev/kvm okd-machine-os.cosa -i
Note that the first command copies the repository contents into the container image (instead of mounting it dynamically). This means if you update the anything in repository, you will need to run this command (and the following ones) again.
# Build the local package repository and bundle the packages:
These instructions are based on the entrypoint.sh
# Login in to your favorite registry
podman login registry.example.com
export REGISTRY_AUTH_FILE=/run/containers/0/auth.conf
export COSA_SKIP_OVERLAY=1
cosa init /src --force
# Copy overrides
mkdir -p ./overrides/rootfs
cp -rvf /overrides/* ./overrides
cp -rvf /src/overlay.d ./overrides/rootfs/
# Create repo for OKD RPMs
pushd /srv/okd-repo
createrepo_c .
popd
# build ostree commit
cosa fetch
cosa build ostree
# Note: if this step fails with
# tar: ./tmp/build/coreos-assembler-config.tar.gz: file changed as we read it
# simply append "|| true" to the tar command in /usr/lib/coreos-assembler/cmdlib.sh
# Create repo for OS Extensions
mkdir -p /overlay/extensions
pushd /overlay/extensions
createrepo_c .
popd
# Build container and push it to the registry (tag will be automatically generated)
cosa upload-oscontainer --name "registry.example.com/jack/okd-machine-os" --add-directory /overlay
Add the end of this, note the reference of the newly generated image, e.g. registry.example.com/jack/okd-os:47.34.202110121035-0.
You can now leave the build environment (okd-machine-os.cosa container).
The contents of the container image we have just built should similar to the structure shown on the right side of the following figure.
Using dive to analyze the intermediate image produced by COSA.
# Build the final machine-os-contents image
While the image built in the previous step contains (almost) everything we need, the directories are not in right place for the Machine Config Operator yet. This is the responsibility of the Dockerfile.template:
| |
INITIAL_TEMPLATE needs to be replaced with the reference of the newly generated image which was just pushed to the registry.
Then, we can the create the final (no really!) image:
buildah bud -t okd-machine-os.content -f Dockerfile.template .
The contents should now look something like this:
Using dive to analyze the final ‘machine-os-conten’ image.
Good luck!