I’ll poke at the Scheme test runner since it was mentioned upthread. The image was built before the latest Ubuntu LTS so I think it pulled in ChezScheme 9.5.8 which is three years old now. 26.04 has ChezScheme 10.0 which is two years old, but ChezScheme 10.4 is a few days old and currently available from GitHub.
GitHub’s action runners don’t yet have Ubuntu 26.04; bear that in mind if you plan on validating exercises both on GHA and as a test runner.
FWIW, total uncompressed test image sizes, sorted by size, as reported when I built them.
5.62MB exercism/objective-c-test-runner
9.87MB exercism/forth-test-runner
9.87MB exercism/func-test-runner
9.87MB exercism/generic-test-runner
9.87MB exercism/harbour-test-runner
9.87MB exercism/jai-test-runner
9.87MB exercism/sed-test-runner
9.87MB exercism/squirrel-test-runner
12.7MB exercism/jq-test-runner
13.6MB exercism/sqlite-test-runner
14.6MB exercism/bash-test-runner
27.8MB exercism/wren-test-runner
28.4MB exercism/lua-test-runner
34.1MB exercism/nim-test-runner
46MB exercism/tcl-test-runner
52.6MB exercism/sml-test-runner
53.1MB exercism/common-lisp-test-runner
56MB exercism/vimscript-test-runner
59.3MB exercism/erlang-test-runner
59.5MB exercism/uiua-test-runner
66MB exercism/awk-test-runner
91.5MB exercism/python-test-runner
92.2MB exercism/clojure-test-runner
98.1MB exercism/lfe-test-runner
108MB exercism/haxe-test-runner
120MB exercism/gleam-test-runner
120MB exercism/phix-test-runner
123MB exercism/php-test-runner
129MB exercism/euphoria-test-runner
145MB exercism/rescript-test-runner
148MB exercism/rexx-test-runner
177MB exercism/coffeescript-test-runner
179MB exercism/factor-test-runner
190MB exercism/abap-test-runner
198MB exercism/arm64-assembly-test-runner
198MB exercism/x86-64-assembly-test-runner
205MB exercism/elm-test-runner
213MB exercism/futhark-test-runner
216MB exercism/powershell-test-runner
217MB exercism/gdscript-test-runner
241MB exercism/cfml-test-runner
243MB exercism/elixir-test-runner
247MB exercism/c-test-runner
267MB exercism/racket-test-runner
270MB exercism/prolog-test-runner
299MB exercism/raku-test-runner
303MB exercism/r-test-runner
314MB exercism/pyret-test-runner
326MB exercism/fortran-test-runner
373MB exercism/unison-test-runner
387MB exercism/cairo-test-runner
387MB exercism/javascript-test-runner
390MB exercism/dyalog-apl-test-runner
408MB exercism/java-test-runner
431MB exercism/cobol-test-runner
440MB exercism/scheme-test-runner
465MB exercism/free-pascal-test-runner
468MB exercism/groovy-test-runner
495MB exercism/nushell-test-runner
498MB exercism/reasonml-test-runner
520MB exercism/purescript-test-runner
529MB exercism/crystal-test-runner
537MB exercism/cpp-test-runner
538MB exercism/kotlin-test-runner
561MB exercism/ballerina-test-runner
576MB exercism/emacs-lisp-test-runner
700MB exercism/odin-test-runner
708MB exercism/arturo-test-runner
714MB exercism/pony-test-runner
716MB exercism/pharo-smalltalk-test-runner
742MB exercism/clojurescript-test-runner
782MB exercism/batch-test-runner
868MB exercism/vbnet-test-runner
898MB exercism/dart-test-runner
929MB exercism/go-test-runner
1.02GB exercism/vlang-test-runner
1.04GB exercism/idris-test-runner
1.04GB exercism/scala-test-runner
1.29GB exercism/chapel-test-runner
1.37GB exercism/swift-test-runner
1.47GB exercism/julia-test-runner
1.67GB exercism/lean-test-runner
8.1GB exercism/haskell-test-runner
1 Like
Do we know if this Batch test runner image is being deployed to the test runner machines? The track doesn’t use it currently:
ClojureScript was deactivated a few years ago so I’m not sure we need to load its test runner either at this point:
Edit - Pony is a WIP track that hasn’t been touched in four years. It also doesn’t have its test runner active. Is the image deployed to the machines?
I have no idea what’s deployed where. I just ran docker build in all the Exercism $track-test-runner repos.
1 Like
Probably something Jeremy can enlighten us on later then. 
You can see that for yourselves by looking at this file: terraform/terraform/main.tf at main · exercism/terraform · GitHub
1 Like
I’m always a bit scared by loosing the option to keep things running for a while without major updates:
Non-latest versions of packages in these repositories are
retained for 12 months until June 8th, 2026, and then for
6 months by default.
6 month are really short sometimes…
Once a test runner image is built, it can continue being used indefinitely. Building new images would pull in a new version.
Speaking of which, our various runners would only be using the same base layers if we actively sync them to the same hash/build. Going through all the runners and applying the same hash might be a fairly significant space saving effort. I might give that a go.
Went back over my Dockerfiles again. Did some more pointed cache cleaning and flag setting. Also upgraded to python3.13-alpine3.23. Turns out, these small changes saved about 25MB+ per image! The three were originally 91MB, 81MB, and 81MB:
REPOSITORY TAG IMAGE ID CREATED SIZE
exercism/python-analyzer latest 09d5ceb81f68 19 hours ago 61.3MB
exercism/python-test-runner latest caf43723118a 19 hours ago 58.6MB
exercism/python-representer latest 03ff2214c930 19 hours ago 58.7MB
Was tempted to play with pre-compiling the tooling to omit docs and other things, but decided that would send me down a rabbit hole with questionable results. I am more or less happy with these sizes at the moment.
3 Likes
Can a Guardian do a quick sanity check on these? Want to make sure I didn’t do something stupid. 
https://github.com/exercism/python-test-runner/pulls
Many thanks!!
I left some nitpicky comments; feel free to ignore them. This looks fine.
1 Like
I looked through unlaunched tracks in terraform to see if they might be close to launch. Here are my impressions.
chapel
A PR discussion suggests build times might have been the launch blocker. I experienced time-outs when submitting the reference meetup and yacht solutions, and narrowly avoided a timeout when submitting the reference forth solution.
Perhaps we could omit some test cases, or the test data for some exercises could be in an array or text file instead of myriad test procs.
dyalog-apl
The track repo has no exercises. The test runner repo has an open issue consider a testing framework
If we remove --read-only then bin/run-tests-in-docker.sh successfully runs the test runner tests.
gdscript
All 23 exercises pass bin/verify-exercises-in-docker
Some open PRs but no insurmountable open issues.
haxe
There is one open test runner issue, reproduced as follows:
If we introduce a bug in an example solution, bin/verify-exercises-in-docker still passes. When we change xit to it in the tests, then bin/verify-exercises-in-docker correctly rejects the solution attempt.
j
I failed to build the test runner image
> [4/7] RUN /opt/j9.5/bin/jconsole -js "load'pacman'" "'update'jpkg''" "'install'jpkg'convert/json'" "'install'jpkg'general/unittest'" "exit 0":
0.196 Load library /opt/j9.5/bin/libj.so failed: /opt/j9.5/bin/libj.so: cannot enable executable stack as shared object requires: Invalid argument
The downloaded test runner image exercism/j-test-runner:latest works well.
phix
The official website is http://phix.x10.mx (no https)
Today I successfully built and tested the test runner image. Yesterday
12.27 950K .......... .......... .......... .......... .......... 9% 88.4M 67s
12.27 1000K .......... .......... .......... ...... 10% 67.3T=7.3s
912.3
912.3 2026-05-17 12:03:45 (142 KB/s) - Read error at byte 1061004/10412596 (Connection timed out). Retrying.
912.3
913.3 --2026-05-17 12:03:46-- (try: 2) http://phix.x10.mx/phix.1.0.5a.zip
913.3 Connecting to phix.x10.mx (phix.x10.mx)|198.91.81.15|:80... failed: Connection timed out.
1049.7 Resolving phix.x10.mx (phix.x10.mx)... failed: No address associated with hostname.
1052.7 wget: unable to resolve host address 'phix.x10.mx'
I have not checked if it would be feasible to download the source from github and build a chosen commit.
The downloaded test runner image exercism/phix-test-runner:latest works well.
pony
With 11 exercises, there is opportunity for more.
Everything looks good.
2 Likes
Here’s a fix for J. I’m not sure why it’s requesting an executable stack but this fixes it. With this fix, this passes all the tests and passes 49/51 example solutions. dnd-character and robot-simulator fail; I did not investigate.
For Chapel, I recently looked at bumping the tracker from 2.0 to 2.6, but build times seem to get a lot worse inside the Docker image. So maybe this can be one of the rare tracks without an online editor? That’d save quite a bit of space.
For Haxe, I recall the track being pretty close to launch besides the test runner and lots of stale concept docs. Nobody has been working on it so the track is targeting Haxe 4.2, which is five years old. Similarly, the buddy BDD testing framework hasn’t been updated since then so we’ll probably want to swap testing frameworks if we bump the track to the current Haxe 4.3.7 release.