One issue that has been in the back of my mind for a while, is that (for most tracks) the CI workflow that uses Docker to verify the test runner’s behavior does not apply proper caching. This meant that each time the CI workflow runs, the entire Docker image is rebuilt from scratch. Is is not only wasteful, but also very slow for many tracks.
I’ve worked on a solution to fix this, and have come up with one that seems to work well. You’ll only need to add two steps to your workflow, as can be seen in this PR: ci: allow caching of docker image by ErikSchierboom · Pull Request #83 · exercism/generic-test-runner · GitHub
The first change is to add a step to setup docker buildx, which has many nice improvements including good caching support. We pass the install: true argument to the workflow, which makes sure that any calls to docker build will actually use docker buildx build, this allowing any cache to be used whilst still being able to use docker build.
The second step is to build the Docker image. The image is just built, not pushed, as we’re only really interested in the Docker build cache that it produces.
With these two steps, running the tests themselves (using Docker) also uses the build cache, which greatly speeds-up the workflow.
Note: you have to ensure that the tag specific in the build-push-action workflow matches the one used in the bin/run-tests-in-docker.sh script.