Many of our test runners use binaries that rely on glibc, preventing the use of Alpine Linux.
We can achieve fairly small images, while retaining glibc, with chainguard. The Lua and SML test runners are now using it.
I have been using bin/verify-exercises-in-docker to verify each track’s examples and exemplars still pass.
As an example of the space savings possible, the awk image reduces from 151MB to 59.4MB
Should I update a few more tracks?
What is chainguard?
What does it do?
How is it reducing image sizes?
Are there licensing issues to watch out for?
What are the potential ramifications for test runner performance?
Thank you for checking into this and for working to get images as small as we can safely make them. 
What BNAndras said, wrt the libs tho. We wanna make sure things are OK and performant.
… and don’t bother with Python. It’s already on Alpine. 
: And R and Julia should stay as-is for now (they’re hairy to manage).
Wolfi is a stripped-down distro designed for the cloud-native era. It doesn’t have a kernel of its own, instead relying on the environment (such as the container runtime) to provide one.
It uses apks, so each test runner Dockerfile reads like one for Alpine Linux.
The license is Apache 2.0
Claude made the recommendation when I was looking for ways to minimise the Factor test runner image. (I am new to Factor, working my way through a generated syllabus.)
The next tracks I was going to measure
So for our purposes, it’s like Alpine but it has glibc. That could be useful for Arturo which is on a Debian slim image due to musl issues. Racket runs on a Debian base image, but I haven’t tried running it with Alpine. I think my other tracks are largely Alpine already.
Yes, it’s like Alpine in the sense that both are open source distros designed for minimal Docker images, and are available as pre-built Docker images.
Just asking to confirm my docker knowledge: If there are multiple images based on debian-slim, moving some of them over to alpine won’t reduce the storage requirements on the servers, right? Since the image layers are shared anyway. If we want to get rid of the additional size required by debian-slim over alpine, we’d have to move all tooling images to alpine, right?
TL;DR: use a separate build and test layer if possible!
The base debian-slim layers are all shared. The test runner hosts would need to pull the base layers for Debian, Ubuntu, Alpine and Wolfi. However, that’s only 4 layers. (It would be nice to dedupe Debian and Ubuntu.) Each of the 80 tracks’ test runners also needs to be stored.
I built the Lua test runner with Ubuntu and Wolfi.
» docker image history lua-ubuntu:latest
IMAGE CREATED CREATED BY SIZE COMMENT
926caa2a3564 About a minute ago /bin/sh -c #(nop) ENTRYPOINT ["/opt/test-ru… 0B
ab54ba577815 About a minute ago /bin/sh -c #(nop) WORKDIR /opt/test-runner 0B
d3852bf0e8ea About a minute ago /bin/sh -c #(nop) COPY dir:65f5fad2a2f1eefc4… 53kB
218d69b0c6bc About a minute ago /bin/sh -c luarocks install busted 7.32MB
6047dd36547b 2 minutes ago /bin/sh -c curl -R -O -L https://luarocks.or… 680kB
3bc72a09b6c5 2 minutes ago /bin/sh -c curl -R -O -L http://www.lua.org/… 1.18MB
e42272df760b 2 minutes ago /bin/sh -c apt-get update && apt-get ins… 295MB
4963bf334d3c 3 minutes ago /bin/sh -c #(nop) ENV LUAROCKS_GPG_KEY=3FD8… 0B
bc0bdbf8f45b 3 minutes ago /bin/sh -c #(nop) ENV LUAROCKS_VER=3.13.0 0B
2427405de735 3 minutes ago /bin/sh -c #(nop) ENV LUA_CHECKSUM=57ccc32b… 0B
05651577c77b 3 minutes ago /bin/sh -c #(nop) ENV LUA_VER=5.5.0 0B
0b1ebe5dd426 3 weeks ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0B
<missing> 3 weeks ago /bin/sh -c #(nop) ADD file:8ce1caf246e7c778b… 78.1MB
<missing> 3 weeks ago /bin/sh -c #(nop) LABEL org.opencontainers.… 0B
<missing> 3 weeks ago /bin/sh -c #(nop) ARG LAUNCHPAD_BUILD_ARCH 0B
<missing> 3 weeks ago /bin/sh -c #(nop) ARG RELEASE 0B
» docker image history lua-wolfi:latest
IMAGE CREATED CREATED BY SIZE COMMENT
994cfc4619ff 5 minutes ago /bin/sh -c #(nop) ENTRYPOINT ["/opt/test-ru… 0B
00019ae023d8 5 minutes ago /bin/sh -c #(nop) WORKDIR /opt/test-runner 0B
bad6e972582f 5 minutes ago /bin/sh -c #(nop) COPY dir:0fa31b623bf21c5ef… 53.6kB
4f5842079a1b 5 minutes ago /bin/sh -c #(nop) COPY dir:a4e4d1da9a6322663… 4.26MB
eded546afbd2 5 minutes ago /bin/sh -c apk add --no-cache bash coreutils 8.06MB
4cb8351fb6e9 3 days ago apko 20.8kB wolfi-base by Chainguard
<missing> 3 days ago apko 10.6kB wolfi-base by Chainguard
<missing> 3 days ago apko 26.1kB wolfi-base by Chainguard
<missing> 3 days ago apko 119kB wolfi-base by Chainguard
<missing> 3 days ago apko 199kB wolfi-base by Chainguard
<missing> 3 days ago apko 219kB wolfi-base by Chainguard
<missing> 3 days ago apko 226kB wolfi-base by Chainguard
<missing> 3 days ago apko 381kB wolfi-base by Chainguard
<missing> 3 days ago apko 692kB wolfi-base by Chainguard
<missing> 3 days ago apko 6.93MB wolfi-base by Chainguard
<missing> 3 days ago apko 7.19MB wolfi-base by Chainguard
Note layer e42272df760b in the Ubuntu build. That’s 295MB for the apt-get update && apt-get install. If I’m reading things correctly, the biggest size difference here is not coming from the Ubuntu/Wolfi swap but from using a new test-runner layer which only takes the needed parts from the build environment.
If I use the prior Ubuntu build process but add a copy layer:
diff --git a/Dockerfile b/Dockerfile
index c5387a9..7332a99 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,4 +1,4 @@
-FROM ubuntu:24.04
+FROM ubuntu:24.04 AS builder
ENV LUA_VER="5.5.0"
ENV LUA_CHECKSUM="57ccc32bbbd005cab75bcc52444052535af691789dba2b9016d5c50640d68b3d"
@@ -34,6 +34,9 @@ RUN curl -R -O -L https://luarocks.org/releases/luarocks-${LUAROCKS_VER}.tar.gz
RUN luarocks install busted
+FROM ubuntu:24.04
+COPY --from=builder /usr/local /usr/local
+
COPY . /opt/test-runner
WORKDIR /opt/test-runner
ENTRYPOINT ["/opt/test-runner/bin/run.sh"]
$ docker image ls
IMAGE ID DISK USAGE CONTENT SIZE EXTRA
lua-ubuntu-layered:latest 7d9ad1fc595f 82.4MB 0B
lua-ubuntu:latest 926caa2a3564 382MB 0B
lua-wolfi:latest 994cfc4619ff 28.4MB 0B
Dropping the build layer saves us those 300MB or so, which is unique per-track since (almost) each layer installs unique dependencies. That’s the bulk of the savings.
That said, the Wolfi version is still only a third of the size of the Ubuntu image, which is nice.
Separating build and final layer is definitely important.
Getting rid of all ubuntu bases and moving them to at least debian-slim seems doable. The two are very similar and ubuntu doesn’t provide anything important over debian.
Moving everything to alpine / wolfi could be more difficult, since they’re so different. I’m just saying that if we want to capture this opimization, we have to consider the cost of migrating all tooling images.
For example, on the Rust track, I have a bunch of code copy-pasted from the official Rust docker files. This assumes debian-slim as base. I imagine moving that to Wolfi would be very tedious and harder to update in the future. (Currently I can just copy-paste again from upstream.)
Note, the ubuntu:24.04 image is only 78MB. It would be nice to get rid of it, but that’s not a huge space savings. We don’t need to migrate everything. Looking at the Lua builds, using the Wolfi base saves 54MB. Do that with two runners and we save more space than dropping the Ubuntu image would save. We probably shouldn’t focus too much on dropping the base image.
We might be able to just use a pre-built rust image, too, eg https://images.chainguard.dev/directory/image/rust/overview. Note, Chainguard only offers the :latest versions for free so using the Chainguard images means we can’t pin the Rust versions.
There’s also pre-built Rust images that are built on Alpine that may work. See https://hub.docker.com/_/rust
What? If the space saving is coming from switching the base, then the saving is an illusion until you’ve switched all tooling images. The lua image specifically might show up as 50MB smaller or whatever, but those 50MB are still there, because the other tooling images still need it. That’s the point I’m trying to make: The space saving of switching the base image is exactly zero as long as other images still use that base.
I think the original reason for not doing that was that we need the nightly toolchain for JSON test output and the nightly tag changes every day. So we didn’t want the cache to be invalidated daily. But now I’m thinking it should be possible to pin the base image to a hash, instead of a tag.
Huh. I mistakenly thought this difference was not due to the base layer… but it seems you’re right. There may be no space savings from switching from Ubuntu to Wolfi here. It might be entirely due to the base change.
None the less, a smaller image does mean faster loads and less memory. Those are still improvements even if the disk usage doesn’t change!
Are you sure? A quick local benchmark shows basically identical performance. It looks to me like container startup time is dominated by other factors.
hyperfine --warmup 3 --runs 100 --shell none \
--prepare "podman rm -f test || true" \
--command-name \
"podman run --name test ubuntu true" \
--command-name wolfi \
"podman run --name test cgr.dev/chainguard/wolfi-base true" \
--export-markdown -
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
ubuntu |
100.1 ± 4.2 | 90.4 | 114.5 | 1.00 |
wolfi |
102.1 ± 3.8 | 93.4 | 110.8 | 1.02 ± 0.06 |
As for RAM consumption, I believe that podman / docker don’t load entire images into RAM at startup. A test runner would have to scan the entire filesystem for some reason, in order to make image size impact RAM consumption. I’m not sure if this is a meaningful way to measure RAM usage, but an LLM suggested time -f "%M" podman run --rm $image true. This also shows the same results for ubuntu and wolfi, about 50kB ± 1kB. (Much smaller than and unrelated to the image size.)
Maybe it doesn’t matter much for Lua. I believe I shrunk the bash-test-runner by 160MB by switching out Ubuntu for Alpine. The apt-get update && apt-get install ... layer in the bash build process added 166MB. The Alpine apk add layer write 5.6MB.
I was curious so I checked the old ubuntu-based bash-test-runner image. The 160MB layer was pulling in a lot of packages that were seemingly not necessary (python, systemd…). So I tried adding --no-install-recommends to that apt-get install command. That reduced the size of that layer to 4.5MB. 1MB smaller than the equivalent layer of the new alpine-based image.
Maybe I should try going through all the apt based runners and add that flag 
Chapel is a WIP track but its test runner is available on the Exercism website. That might be a candidate for using a builder image since it’s currently 624 MB compressed according to Image Layer Details - exercism/chapel-test-runner:bfc3df31ecbd6d191ba6948cb4fcd0ab545cf014. The Dockerfile is at chapel-test-runner/Dockerfile at main · exercism/chapel-test-runner · GitHub.
Chapel seems like it has a long list of requirements, which seems to be responsible for most of the space:
clang
libclang-cpp-dev
libclang-dev
libedit-dev
libgmp-dev
libgmp10
llvm
pkg-config
python3
Removing any of those appears to make Chapel stop working, so I’m not sure we can get much further ahead.