Tools for analyzing and working with container images

Thu, 06 Jun 2024 00:00:00 +0000

In this post I’d like to share some tools from my toolbox for working with container images (Docker/OCI). I specifically excluded tools for building container images (Buildkit, buildah, kaniko & friends).

# What is a container image?

Before we get to the tools, let’s first take a minute to understand what a container image actually is. This will allow us to better understand what these tools do and how they can help us.

In a nutshell, a container image is a collection of layers alongside some configuration metadata which are tied together by an image manifest. The layers contain files and directories, which are stacked on top of each other to form the final filesystem for the container. The principal is similar to a Git repository: each repository is made up of one or more commits (= layers), a j.git/config (= configuration) and an index (= manifest). The container image format was initially developed by Docker, but has been spun out into the dedicated Open Container Initiative project to help with standardization and interoperability. These days the OCI image specification formally defines what an image consists of and is supported by many tools in the cloud native ecosystem.

Sidenote: the OCI also defines the Distribution Specification (which API clients and servers need to implement for uploading and downloading container images) as well as the Runtime Specification (how a container should be run).

Let’s look at a practical example (all subsequent podman command can be substituted with docker):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


$ mkdir /tmp/container-tools
$ cd /tmp/container-tools
$ podman image pull docker.io/library/httpd:latest
Trying to pull docker.io/library/httpd:latest...
Getting image source signatures
Copying blob ea83e81966d6 ... done
Copying blob b0a0cf830b12 ... done
Copying blob 4f4fb700ef54 ... done
Copying blob 943a2b3cf551 ... done
Copying blob 851c52adaa9b ... done
Copying blob 39d9f60535a6 ... done
Copying config 67c2fc9e3d ... done
Writing manifest to image destination
67c2fc9e3d849b21a35b4c96f5ad8ec4dc9b73ca44c56537039e8c6f3054db0a

The previous command downloaded the most recent version of the Apache httpd container image from DockerHub. Based on the progressive download output we can already see that the container image is not “one file”, but is made up of multiple parts (referred to as blobs in the output above).

The image is now stored locally on the machine, but the exact location will vary depending on the container manager (Docker, Podman, …) and its configuration. Let’s extract the contents of the container image into an empty directory:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


$ mkdir /tmp/httpd && cd /tmp/httpd
$ podman image save docker.io/library/httpd:latest | tar x
$ tree
.
├── 30bc0a59d494d75238f1f9d26e3851eb33debbfca69ed5dac18588bdcb58d9d4
│   ├── json
│   ├── layer.tar -> ../53920c8b9c11c3f431a7f096a6a496f587eec1ccbc3b9e9649470db799c96758.tar
│   └── VERSION
├── 46176a0cbe9f3e95a08e3914ef9f4ac33fb1d412c9bacf2d1217616f76077fad.tar
├── 52ec5a4316fadc09a4a51f82b8d7b66ead0d71bea4f75e81e25b4094c4219061.tar
├── 53920c8b9c11c3f431a7f096a6a496f587eec1ccbc3b9e9649470db799c96758.tar
├── 5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef.tar
├── 67c2fc9e3d849b21a35b4c96f5ad8ec4dc9b73ca44c56537039e8c6f3054db0a.json
├── 7b341f868c2e467ddb56f7dfcbe82578403faa3da4ae0cb264048f3fa7a9bacd
│   ├── json
│   ├── layer.tar -> ../5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef.tar
│   └── VERSION
├── a1f21c4adbc031095cd0bf494d36e6d2f7c8c0a4fb0e8f22ff559f3e4b436e66
│   ├── json
│   ├── layer.tar -> ../dba1169a4ef830544ed14026d745670ee754a929e251440482009a5e89d4f20c.tar
│   └── VERSION
├── a2ba897b57547df4c74318915ba74bac4c5603d3ed56ea70eb7397914479d033
│   ├── json
│   ├── layer.tar -> ../ed8b961e754fdc176a3307cc94471a8c7db19edfcbce503deeb40d0043af2f44.tar
│   └── VERSION
├── d4112ecc8ff8a228860a3851d8cdf391987fc47b0c1ffa9e3ddbc1c744da4b16
│   ├── json
│   ├── layer.tar -> ../46176a0cbe9f3e95a08e3914ef9f4ac33fb1d412c9bacf2d1217616f76077fad.tar
│   └── VERSION
├── dba1169a4ef830544ed14026d745670ee754a929e251440482009a5e89d4f20c.tar
├── e6fa7777a000792ac871b6ab8c18b5dd6778dbe61537b3864e4aa50445c45d2c
│   ├── json
│   ├── layer.tar -> ../52ec5a4316fadc09a4a51f82b8d7b66ead0d71bea4f75e81e25b4094c4219061.tar
│   └── VERSION
├── ed8b961e754fdc176a3307cc94471a8c7db19edfcbce503deeb40d0043af2f44.tar
├── manifest.json
└── repositories
7 directories, 27 files

The image manifest (manifest.json) is the file that ties everything together, let’s take a look at it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


$ jq . manifest.json
[
{
"Config": "67c2fc9e3d849b21a35b4c96f5ad8ec4dc9b73ca44c56537039e8c6f3054db0a.json",
"RepoTags": [
"docker.io/library/httpd:latest"
],
"Layers": [
"52ec5a4316fadc09a4a51f82b8d7b66ead0d71bea4f75e81e25b4094c4219061.tar",
"46176a0cbe9f3e95a08e3914ef9f4ac33fb1d412c9bacf2d1217616f76077fad.tar",
"5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef.tar",
"53920c8b9c11c3f431a7f096a6a496f587eec1ccbc3b9e9649470db799c96758.tar",
"ed8b961e754fdc176a3307cc94471a8c7db19edfcbce503deeb40d0043af2f44.tar",
"dba1169a4ef830544ed14026d745670ee754a929e251440482009a5e89d4f20c.tar"
]
}
]

It tells us where we can find the configuration (Config) of the container image and where we can find the layers. Notice that layers is an array: the layers need to be assembled in the correct order to get the correct result. It’s also interesting to observe that all referenced files use content-addressable storage, i.e. the name of the files corresponds to their SHA256 checksum. The first advantage is that we can easily check if any errors occurred during the download (if that was the cause, the filename and checksum would not match):

1
2
3
4
5
6
7
8


$ sha256sum *.json *.tar
67c2fc9e3d849b21a35b4c96f5ad8ec4dc9b73ca44c56537039e8c6f3054db0a 67c2fc9e3d849b21a35b4c96f5ad8ec4dc9b73ca44c56537039e8c6f3054db0a.json
46176a0cbe9f3e95a08e3914ef9f4ac33fb1d412c9bacf2d1217616f76077fad 46176a0cbe9f3e95a08e3914ef9f4ac33fb1d412c9bacf2d1217616f76077fad.tar
52ec5a4316fadc09a4a51f82b8d7b66ead0d71bea4f75e81e25b4094c4219061 52ec5a4316fadc09a4a51f82b8d7b66ead0d71bea4f75e81e25b4094c4219061.tar
53920c8b9c11c3f431a7f096a6a496f587eec1ccbc3b9e9649470db799c96758 53920c8b9c11c3f431a7f096a6a496f587eec1ccbc3b9e9649470db799c96758.tar
5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef 5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef.tar
dba1169a4ef830544ed14026d745670ee754a929e251440482009a5e89d4f20c dba1169a4ef830544ed14026d745670ee754a929e251440482009a5e89d4f20c.tar
ed8b961e754fdc176a3307cc94471a8c7db19edfcbce503deeb40d0043af2f44 ed8b961e754fdc176a3307cc94471a8c7db19edfcbce503deeb40d0043af2f44.tar

The second advantage is that it allows sharing layers between container images. For example if two container images are built from the same base image, that particular base layer only needs to be stored once (both in the container registry and on the local machine) as well transmitted once (the check can be carried out before downloading the layers).

The configuration file (in this case 67c2fc9e3d849b21a35b4c96f5ad8ec4dc9b73ca44c56537039e8c6f3054db0a.json) contains information about how and when the image was built (history), which platform it is compatible with (architecture, os) and how the container should be run (elements under config).

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119


$ jq . 67c2fc9e3d849b21a35b4c96f5ad8ec4dc9b73ca44c56537039e8c6f3054db0a.json
{
"architecture": "amd64",
"config": {
"ExposedPorts": {
"80/tcp": {}
},
"Env": [
"PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"HTTPD_PREFIX=/usr/local/apache2",
"HTTPD_VERSION=2.4.59",
"HTTPD_SHA256=ec51501ec480284ff52f637258135d333230a7d229c3afa6f6c2f9040e321323",
"HTTPD_PATCHES="
],
"Cmd": [
"httpd-foreground"
],
"WorkingDir": "/usr/local/apache2",
"StopSignal": "SIGWINCH",
"ArgsEscaped": true
},
"created": "2024-04-05T21:49:16Z",
"history": [
{
"created": "2024-04-05T21:49:16Z",
"created_by": "/bin/sh -c #(nop) ADD file:4b1be1de1a1e5aa608c688cad2824587262081866180d7368feb79d33ca05953 in / "
},
{
"created": "2024-04-05T21:49:16Z",
"created_by": "/bin/sh -c #(nop) CMD [\"bash\"]",
"empty_layer": true
},
{
"created": "2024-04-05T21:49:16Z",
"created_by": "ENV HTTPD_PREFIX=/usr/local/apache2",
"comment": "buildkit.dockerfile.v0",
"empty_layer": true
},
{
"created": "2024-04-05T21:49:16Z",
"created_by": "ENV PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"comment": "buildkit.dockerfile.v0",
"empty_layer": true
},
{
"created": "2024-04-05T21:49:16Z",
"created_by": "RUN /bin/sh -c mkdir -p \"$HTTPD_PREFIX\" \t&& chown www-data:www-data \"$HTTPD_PREFIX\" # buildkit",
"comment": "buildkit.dockerfile.v0"
},
{
"created": "2024-04-05T21:49:16Z",
"created_by": "WORKDIR /usr/local/apache2",
"comment": "buildkit.dockerfile.v0"
},
{
"created": "2024-04-05T21:49:16Z",
"created_by": "RUN /bin/sh -c set -eux; \tapt-get update; \tapt-get install -y --no-install-recommends \t\tca-certificates \t\tlibaprutil1-ldap \t\tlibldap-common \t; \trm -rf /var/lib/apt/lists/* # buildkit",
"comment": "buildkit.dockerfile.v0"
},
{
"created": "2024-04-05T21:49:16Z",
"created_by": "ENV HTTPD_VERSION=2.4.59",
"comment": "buildkit.dockerfile.v0",
"empty_layer": true
},
{
"created": "2024-04-05T21:49:16Z",
"created_by": "ENV HTTPD_SHA256=ec51501ec480284ff52f637258135d333230a7d229c3afa6f6c2f9040e321323",
"comment": "buildkit.dockerfile.v0",
"empty_layer": true
},
{
"created": "2024-04-05T21:49:16Z",
"created_by": "ENV HTTPD_PATCHES=",
"comment": "buildkit.dockerfile.v0",
"empty_layer": true
},
{
"created": "2024-04-05T21:49:16Z",
"created_by": "RUN /bin/sh -c set -eux; \t\tsavedAptMark=\"$(apt-mark showmanual)\"; \tapt-get update; [...] ",
"comment": "buildkit.dockerfile.v0"
},
{
"created": "2024-04-05T21:49:16Z",
"created_by": "STOPSIGNAL SIGWINCH",
"comment": "buildkit.dockerfile.v0",
"empty_layer": true
},
{
"created": "2024-04-05T21:49:16Z",
"created_by": "COPY httpd-foreground /usr/local/bin/ # buildkit",
"comment": "buildkit.dockerfile.v0"
},
{
"created": "2024-04-05T21:49:16Z",
"created_by": "EXPOSE map[80/tcp:{}]",
"comment": "buildkit.dockerfile.v0",
"empty_layer": true
},
{
"created": "2024-04-05T21:49:16Z",
"created_by": "CMD [\"httpd-foreground\"]",
"comment": "buildkit.dockerfile.v0",
"empty_layer": true
}
],
"os": "linux",
"rootfs": {
"type": "layers",
"diff_ids": [
"sha256:52ec5a4316fadc09a4a51f82b8d7b66ead0d71bea4f75e81e25b4094c4219061",
"sha256:46176a0cbe9f3e95a08e3914ef9f4ac33fb1d412c9bacf2d1217616f76077fad",
"sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
"sha256:53920c8b9c11c3f431a7f096a6a496f587eec1ccbc3b9e9649470db799c96758",
"sha256:ed8b961e754fdc176a3307cc94471a8c7db19edfcbce503deeb40d0043af2f44",
"sha256:dba1169a4ef830544ed14026d745670ee754a929e251440482009a5e89d4f20c"
]
}
}

Finally, let’s take a look at the layers, which are individual archives (.tar). Sidenote: usually these layers are compressed with gzip or zstd to save space (and time!). This is not the case here because we locally exported the image, i.e. they have already been decompressed in the process.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


# the base layer (`FROM debian:bookworm-slim`)
$ tar tvf 52ec5a4316fadc09a4a51f82b8d7b66ead0d71bea4f75e81e25b4094c4219061.tar | wc -l
4222
# second layer only creates the apache directory
$ tar tvf 46176a0cbe9f3e95a08e3914ef9f4ac33fb1d412c9bacf2d1217616f76077fad.tar
drwxr-xr-x 0/0 0 2024-04-23 17:00 usr/
drwxr-xr-x 0/0 0 2024-04-24 06:54 usr/local/
drwxr-xr-x 33/33 0 2024-04-24 06:54 usr/local/apache2/
# third layer is empty (changing WORKDIR)
$ tar tvf 5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef.tar | wc -l
0
# fourth layer adds runtime dependencies, certificates etc.
$ tar tvf 53920c8b9c11c3f431a7f096a6a496f587eec1ccbc3b9e9649470db799c96758.tar | wc -l
608
# fifth layer contains the application (apache httpd)
$ tar tvf ed8b961e754fdc176a3307cc94471a8c7db19edfcbce503deeb40d0043af2f44.tar | wc -l
735
# final layer adds a script for running apache httpd in the foreground
$ tar tvf dba1169a4ef830544ed14026d745670ee754a929e251440482009a5e89d4f20c.tar
drwxr-xr-x 0/0 0 2024-04-23 17:00 usr/
drwxr-xr-x 0/0 0 2024-04-24 06:54 usr/local/
drwxr-xr-x 0/0 0 2024-04-24 06:57 usr/local/bin/
-rwxr-xr-x 0/0 138 2024-04-24 06:54 usr/local/bin/httpd-foreground

For reference you can compare these layers to the upstream Dockerfile. Notice that not all instructions result in the creation of a new layer. Instructions like ENV and STOPSIGNAL are recorded in the configuration manifest instead, which we looked at before.

Now we have a basic understanding of the contents of a container image, let’s take a look at the toolbox.

# dive

The first tool is called dive: an interactive TUI (terminal user interface) tool that effectively automates the process we did above: looking at individual layers of an image and understanding which files were added/removed/modified. This is super helpful for learning how to write efficient Dockerfiles (i.e. minimal size) and developing confidence on what exactly is part of the image being built. Diffing and seeing what files/directories go into each layer is handy to reduce the size of the container image, which in return makes it faster to build, upload and download since less data needs to transferred.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


$ dive podman://docker.io/library/httpd:latest
┃ ● Current Layer Contents ┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
│ Layers ├──────────────────────────────────────────────────────────────────────────── Permission UID:GID Size Filetree
Cmp Size Command -rwxrwxrwx 0:0 0 B ├── bin → usr/bin
75 MB FROM 52ec5a4316fadc0 drwxr-xr-x 0:0 0 B ├── boot
0 B RUN /bin/sh -c mkdir -p "$HTTPD_PREFIX" && chown www-data:www-data "$H drwxr-xr-x 0:0 0 B ├── dev
 0 B WORKDIR /usr/local/apache2 drwxr-xr-x 0:0 172 kB ├─⊕ etc
 10 MB RUN /bin/sh -c set -eux; apt-get update; apt-get install -y --no-i drwxr-xr-x 0:0 0 B ├── home
 62 MB RUN /bin/sh -c set -eux; savedAptMark="$(apt-mark showmanual)"; -rwxrwxrwx 0:0 0 B ├── lib → usr/lib
 138 B COPY httpd-foreground /usr/local/bin/ # buildkit -rwxrwxrwx 0:0 0 B ├── lib64 → usr/lib64
 drwxr-xr-x 0:0 0 B ├── media
 drwxr-xr-x 0:0 0 B ├── mnt
 drwxr-xr-x 0:0 0 B ├── opt
 drwxr-xr-x 0:0 0 B ├── proc
 drwx------ 0:0 732 B ├─⊕ root
 drwxr-xr-x 0:0 0 B ├─⊕ run
 -rwxrwxrwx 0:0 0 B ├── sbin → usr/sbin
│ Layer Details ├───────────────────────────────────────────────────────────────────── drwxr-xr-x 0:0 0 B ├── srv
 drwxr-xr-x 0:0 0 B ├── sys
Tags: (unavailable) drwxrwxrwx 0:0 0 B ├── tmp
Id: 46176a0cbe9f3e95a08e3914ef9f4ac33fb1d412c9bacf2d1217616f76077fad.tar drwxr-xr-x 0:0 70 MB ├── usr
Digest: sha256:46176a0cbe9f3e95a08e3914ef9f4ac33fb1d412c9bacf2d1217616f76077fad drwxr-xr-x 0:0 21 MB │ ├─⊕ bin
Command: drwxr-xr-x 0:0 0 B │ ├── games
RUN /bin/sh -c mkdir -p "$HTTPD_PREFIX" && chown www-data:www-data "$HTTPD_PREFIX" drwxr-xr-x 0:0 0 B │ ├── include
 # buildkit drwxr-xr-x 0:0 40 MB │ ├─⊕ lib
 drwxr-xr-x 0:0 0 B │ ├─⊕ lib64
 drwxr-xr-x 0:0 17 kB │ ├─⊕ libexec
 drwxr-xr-x 0:0 0 B │ ├── local
 drwxr-xr-x 33:33 0 B │ │ ├── apache2
 drwxr-xr-x 0:0 0 B │ │ ├── bin
 drwxr-xr-x 0:0 0 B │ │ ├── etc
 drwxr-xr-x 0:0 0 B │ │ ├── games
│ Image Details ├───────────────────────────────────────────────────────────────────── drwxr-xr-x 0:0 0 B │ │ ├── include
 drwxr-xr-x 0:0 0 B │ │ ├── lib
Image name: docker.io/library/httpd:latest -rwxrwxrwx 0:0 0 B │ │ ├── man → share/man
Total Image size: 147 MB drwxr-xr-x 0:0 0 B │ │ ├── sbin
Potential wasted space: 2.4 MB drwxr-xr-x 0:0 0 B │ │ ├─⊕ share
Image efficiency score: 99 % drwxr-xr-x 0:0 0 B │ │ └── src
 drwxr-xr-x 0:0 6.3 MB │ ├─⊕ sbin
Count Total Space Path drwxr-xr-x 0:0 3.2 MB │ ├─⊕ share
 2 1.5 MB /var/cache/debconf/templates.dat drwxr-xr-x 0:0 0 B │ └── src
 3 262 kB /var/lib/dpkg/status-old drwxr-xr-x 0:0 4.2 MB └─⊕ var
 3 262 kB /var/lib/dpkg/status
 2 109 kB /var/log/dpkg.log
 2 82 kB /var/lib/dpkg/info/perl-base.list
 2 43 kB /var/log/apt/term.log

dive also calculates an efficiency score by determining how many bytes of the final image (sum of all layers) are “wasted”. An ideal score is 100%. If a file is added in the first layer and then removed again in a following layer, this space is considered wasted and the efficiency score will be lower. This can be used in CI tests to prevent needlessly big container images.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


$ CI=true dive podman://docker.io/library/httpd:latest
Image Source: podman://docker.io/library/httpd:latest
Fetching image... (this can take a while for large images)
Analyzing image...
efficiency: 99.0433 %
wastedBytes: 2399436 bytes (2.4 MB)
userWastedPercent: 3.3051 %
Inefficient Files:
Count Wasted Space File Path
2 1.5 MB /var/cache/debconf/templates.dat
3 262 kB /var/lib/dpkg/status-old
3 262 kB /var/lib/dpkg/status
2 109 kB /var/log/dpkg.log
2 82 kB /var/lib/dpkg/info/perl-base.list
2 43 kB /var/log/apt/term.log
2 20 kB /var/cache/debconf/config.dat
3 20 kB /var/log/apt/eipp.log.xz
3 16 kB /etc/ld.so.cache
3 15 kB /var/lib/apt/extended_states
2 12 kB /var/cache/ldconfig/aux-cache
2 9.6 kB /var/log/apt/history.log
2 8.2 kB /var/cache/debconf/config.dat-old
3 0 B /var/lib/dpkg/lock
2 0 B /var/lib/dpkg/triggers/Unincorp
3 0 B /var/lib/dpkg/triggers/Lock
Results:
PASS: highestUserWastedPercent
SKIP: highestWastedBytes: rule disabled
PASS: lowestEfficiency
Result:PASS [Total:3] [Passed:2] [Failed:0] [Warn:0] [Skipped:1]

# diffoci

The next tool serves a similar, yet slightly different purpose: diffoci compares container images and shows the differences between them (unlike dive which shows differences between layers). It is the successor to the deprecated container-diff tool.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


$ diffoci diff --semantic docker.io/library/httpd:alpine3.17 docker.io/library/httpd:alpine3.18
TYPE NAME INPUT-0 INPUT-1
File etc/alpine-release 44d3bbd769620a3aa2fd5403d0755f6a4afba3c14e55c156f1c88905866ff04e aac1e8712e84121ed6a270d8003e4b52d6cb474b7d1709cd80a4bc5b6c82aa6d
File lib/libssl.so.3 7c6e060a33d5e0969aced1645e669666b58f223f27d783b8e49471088e3f756a c751d77b54c8ec62103a7052bbf6efdd4f05fa0df1168153ca88e7a81d27d9a6
File bin/busybox 36d96947f81bee3a5e1d436a333a52209f051bb3556028352d4273a748e2d136 8c6864b97c2b9ff892c859c59363482cb19a546a33a204b83a24f29048b2118f
File etc/os-release e72f89989f15487206f906fe8fb12a6d4363273c0ec886e66e66222a301e42f6 a001686cc7651f44cbbeae86e783d42dc05dce05940c55f6e394b6f9bea4c631
File etc/ssl/certs/ca-certificates.crt 92dd040a840947f00a14f66e9e17a799051ad7f88ddcde851b3711ee0a78a7ac 824cefcee69de918c76b7b92776f304c3a4b7f6281539118bc1d41a9dd8476d9
Layer ctx:/manifests-0/layers-0/layer name "lib/libapk.so.3.12.0" only appears in input 0
File etc/issue c2dc1872792bfb7e7af7467b82cf136ad468acab74d99a97d0e479dbf4665dab 774e6ac5abc275013a07cba05be43f0953cf759f852d177aa25f27ff62ab2eb4
File etc/ssl/openssl.cnf.dist 15c6ec001241f54bed98c135f0dab42f4a5e489f22544613b0671198f8ad8318 f6045e326b439e8ee31d4efd020ddf660d616c67d03e0e8e7a927eb14cbb5d1f
Layer ctx:/manifests-0/layers-0/layer name "etc/profile.d/locale.sh" only appears in input 0
File lib/libz.so.1.2.13 a7df4375fc9f37d5c284ef74e1d782d0100ce3a907ad2cbc6287f769cb90aac4 e0df969fd057eeaeb3064663b0f0af0f776fa42c697b56726a3eec6b13ca6986
File lib/libcrypto.so.3 4b061badaa55a61bcb65a05445f84d3003e3d554c10a56a44d7e6f0c0e349e6e 169037db2377a8227ddba422ca6e03dfb61c6660a2594ecf3b306bc3323ff80c
File etc/apk/repositories e23f0c9897bd5f21f1451995f7e05470026012288d7ec811ed6e5567cd521c01 293ab007ea57d023c3418a8063966a40407ceb90517e60ad710772c70694f9c5
...

# container-structure-test

If you’d like to make sure that the container images you’re building always contain certain configuration files, have dependencies installed etc., then container-structure-test is the right tool. Especially when you have to deal with a large number of (base) images and complex multi-stage builds, it is invaluable to run quick unit tests on them (after building) instead of having to go through a full integration test cycle (which usually involves uploading the container image to a registry, downloading it on the target host and then spinning up the application alongside all its dependencies).

container-structure-test takes a container image and a simple YAML configuration file as input:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


schemaVersion: "2.0.0"
commandTests:
- name: 'helm is installed'
command: 'helm'
args: ['version']
exitCode: 0
fileContentTests:
- name: 'Velero is configured'
path: '/root/.config/velero/config.json'
expectedContents:
- '{"namespace":"kube-system"}'
fileExistenceTests:
- name: 'kubectl bash completions are installed'
path: '/etc/bash_completion.d/kubectl_completion'
shouldExist: true

and will then validate that all conditions are met:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


$ container-structure-test test --image "registry.example.com/kube-tools" --config container-structure-test.yaml
======================================================
====== Test file: container-structure-test.yaml ======
======================================================
=== RUN: Command Test: helm is installed
--- PASS
duration: 500.813594ms
stdout: version.BuildInfo{Version:"v3.11.2", GitCommit:"912ebc1cd10d38d340f048efaf0abda047c3468e", GitTreeState:"clean", GoVersion:"go1.18.10"}
=== RUN: File Content Test: Velero is configured
--- PASS
duration: 0s
=== RUN: File Existence Test: kubectl bash completions are installed
--- PASS
duration: 0s
=======================================================
======================= RESULTS =======================
=======================================================
Passes: 3
Failures: 0
Duration: 0.519310751s
Total tests: 3
PASS

Note that container-structure-test works best with an actual Docker daemon, though with some workarounds it is possible to use Podman.

I previously gave a lightning talk about this testing tool, you can watch the recording here:

# container-explorer

Given the disk image of a virtual (or physical) machine, container-explorer is a neat tool that allows you to explore the containers and container images on the host. It supports containers run by containerd as well as Docker. Since OpenShift uses the CRI-O container runtime, I haven’t used this tool myself but I imagine it can be quite helpful for post-mortem troubleshooting and forensic analysis.

# skopeo

skopeo is a tool for working with container images in remote image registries. It can show information about an image repository (list tags, digests and configuration) as well as copy/delete/sync images from one registry to another (without having to store the images locally). This can be extremely handy when you are in an environment where you cannot run containers (e.g. inside a container!) or you don’t have root access.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77


$ skopeo list-tags docker://docker.io/library/httpd
{
"Repository": "docker.io/library/httpd",
"Tags": [
"2",
"2-alpine",
"2-alpine3.13",
"2-alpine3.14",
"2-alpine3.15",
"2-alpine3.16",
"2-alpine3.17",
...
$ skopeo inspect docker://docker.io/library/httpd:latest
{
"Name": "docker.io/library/httpd",
"Digest": "sha256:36c8c79f900108f0f09fd4148ad35ade57cba0dc19d13f3d15be24ce94e6a639",
"RepoTags": [ ... ],
"Created": "2024-04-05T21:49:16Z",
"DockerVersion": "",
"Labels": null,
"Architecture": "amd64",
"Os": "linux",
"Layers": [
"sha256:b0a0cf830b12453b7e15359a804215a7bcccd3788e2bcecff2a03af64bbd4df7",
"sha256:851c52adaa9bf680b2eaa45164baa37665418fb02666fb36c268002edd853994",
"sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1",
"sha256:39d9f60535a61833df2e57cc08221031d3ae50f3955a6d6b0153214619d357fb",
"sha256:943a2b3cf551375517593790b58407ffc7028b4b34bca6c0756b76708ea4b3ce",
"sha256:ea83e81966d64f34a57ffcc118b75ad19bc9321f9809cf9b0df25548a2fa695e"
],
"LayersData": [
{
"MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip",
"Digest": "sha256:b0a0cf830b12453b7e15359a804215a7bcccd3788e2bcecff2a03af64bbd4df7",
"Size": 29150479,
"Annotations": null
},
{
"MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip",
"Digest": "sha256:851c52adaa9bf680b2eaa45164baa37665418fb02666fb36c268002edd853994",
"Size": 146,
"Annotations": null
},
{
"MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip",
"Digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1",
"Size": 32,
"Annotations": null
},
{
"MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip",
"Digest": "sha256:39d9f60535a61833df2e57cc08221031d3ae50f3955a6d6b0153214619d357fb",
"Size": 4006988,
"Annotations": null
},
{
"MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip",
"Digest": "sha256:943a2b3cf551375517593790b58407ffc7028b4b34bca6c0756b76708ea4b3ce",
"Size": 26027444,
"Annotations": null
},
{
"MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip",
"Digest": "sha256:ea83e81966d64f34a57ffcc118b75ad19bc9321f9809cf9b0df25548a2fa695e",
"Size": 293,
"Annotations": null
}
],
"Env": [
"PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"HTTPD_PREFIX=/usr/local/apache2",
"HTTPD_VERSION=2.4.59",
"HTTPD_SHA256=ec51501ec480284ff52f637258135d333230a7d229c3afa6f6c2f9040e321323",
"HTTPD_PATCHES="
]
}

# crane

The use case for crane is very similar to skopeo, but it goes a step further: crane can also modify existing container images. Adding or changing layers, flattening (“squashing”) images, modifying metadata (environment variables, entrypoint, labels etc.). It does this all directly in the registry, meaning no local container runtime is needed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54


# list files in an image
$ crane export docker.io/library/httpd:latest - | tar -tvf - | head
drwxr-xr-x 0/0 0 2024-04-23 17:00 usr
drwxr-xr-x 0/0 0 2024-04-24 06:54 usr/local
drwxr-xr-x 0/0 0 2024-04-24 06:57 usr/local/bin
-rwxr-xr-x 0/0 138 2024-04-24 06:54 usr/local/bin/httpd-foreground
drwxr-xr-x 0/0 0 2024-04-24 06:57 etc
drwxr-xr-x 0/0 0 2024-04-24 06:54 etc/gss
drwxr-xr-x 0/0 0 2023-08-14 22:06 etc/gss/mech.d
-rw-r--r-- 0/0 6682 2024-04-24 06:57 etc/ld.so.cache
drwxr-xr-x 0/0 0 2024-04-24 06:57 usr/lib
drwxr-xr-x 0/0 0 2024-04-24 06:57 usr/lib/x86_64-linux-gnu
# squash an image into a single layer and push it to a different registry
$ crane flatten docker.io/library/httpd:latest --tag registry.example.com/jack/httpd:flat --platform linux/amd64
...
# check that the new image only has one layer
$ crane config registry.example.com/jack/httpd:flat | jq .
{
"architecture": "amd64",
"created": "2024-04-05T21:49:16Z",
"history": [
{
"created": "0001-01-01T00:00:00Z",
"created_by": "crane flatten sha256:518e9447a236de47cc2fb4f2dbf06466b6cd5cf50f9951742f9a20af76e8118d"
}
],
"os": "linux",
"rootfs": {
"type": "layers",
"diff_ids": [
"sha256:3da7dd597070d0a53bef421a06dc2b7e3e6e937449c6f78e144ed4095c07a52a"
]
},
"config": {
"Cmd": [
"httpd-foreground"
],
"Env": [
"PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"HTTPD_PREFIX=/usr/local/apache2",
"HTTPD_VERSION=2.4.59",
"HTTPD_SHA256=ec51501ec480284ff52f637258135d333230a7d229c3afa6f6c2f9040e321323",
"HTTPD_PATCHES="
],
"WorkingDir": "/usr/local/apache2",
"ExposedPorts": {
"80/tcp": {}
},
"ArgsEscaped": true,
"StopSignal": "SIGWINCH"
}
}

Check out their recipes page for some more examples and take a look at the go-containerregistry library (which exposes the primitives crane is built on through a Go API) as well.

# dredge

The final item in the toolbox is dredge. Much like skopeo and crane it allows you to inspect repositories and images from remote registries. One particularly neat feature is the diffing capability: it can show differences in files between two images.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


$ dredge image compare layers httpd:2.2 httpd:2.4 --os linux --arch amd64 --output=inline --compressed-size
- sha256:f49cf87b52c10aa83b4f4405800527a74400fb19ea1821d209293bc4d53966aa
- Size (compressed): 50.2 MB
+ sha256:b0a0cf830b12453b7e15359a804215a7bcccd3788e2bcecff2a03af64bbd4df7
+ Size (compressed): 27.8 MB
- sha256:24b1e09cbcb7a380a9c98956c6131125b3e0f0a7965d24db460f3548bf300eae
- Size (compressed): 155 bytes
+ sha256:851c52adaa9bf680b2eaa45164baa37665418fb02666fb36c268002edd853994
+ Size (compressed): 146 bytes
- sha256:8a4e0d64e915177206a8c2729a60e6b5001ae5bf47d598ad79a626e5cf983309
- Size (compressed): 11.4 MB
+ sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
+ Size (compressed): 32 bytes
- sha256:bcbe0eb4ca5196217c6368138d5e1feabdb83272dc52859a6e5f2e81aec0c939
- Size (compressed): 1.9 MB
+ sha256:39d9f60535a61833df2e57cc08221031d3ae50f3955a6d6b0153214619d357fb
+ Size (compressed): 3.8 MB
- sha256:16e370c15d382f71f43248107f9524085de33a42809f33034e4571e503f358bf
- Size (compressed): 304 bytes
+ sha256:943a2b3cf551375517593790b58407ffc7028b4b34bca6c0756b76708ea4b3ce
+ Size (compressed): 24.8 MB
+ sha256:ea83e81966d64f34a57ffcc118b75ad19bc9321f9809cf9b0df25548a2fa695e
+ Size (compressed): 293 bytes
$ dredge image compare files httpd:2.2 httpd:2.4 --os linux --arch amd64 --base-layer-index 4 --target-layer-index 4
...
$ dredge image save-layers httpd:2.2 /tmp/httpd-2.2 --layer-index 4 --arch amd64 --os linux --no-squash
Getting layers for httpd:2.2
Layer sha256:16e370c15d382f71f43248107f9524085de33a42809f33034e4571e503f358bf
Downloading layer...
Extracting layer...
$ tree /tmp/httpd-2.2
/tmp/httpd-2.2
└── layer4-16e370c15d382f71f43248107f9524085de33a42809f33034e4571e503f358bf
└── usr
└── local
└── bin
└── httpd-foreground

It’s also notable that dredge is written in C# with the .NET framework (unlike all other tools which are written in Go).

That’s all for today! I hope you found some new tools for your toolbox. Some of them have overlapping functionality, so you can pick and choose the appropriate tool for the task at hand. Also make sure to check out the awesome-docker and awesome-linux-containers lists.

container-image on Jack Henschel's Blog