For CI on the Tcl track, I want to run tests in the tcl-test-runner. This will be similar to what 8th does. The reason is the base Ubuntu image provides an old package for Tcl.
The main thing to look out for is that if you use the test runner image to test the exercises, updating to a new language release can be a little involved. The updated runner image needs to be deployed first so it can be pulled in by the track CI, and as a result, you might be playing with fire if something goes wrong upgrading the track repo.
For the Vim script track, that hasn’t been an issue to date, but for LFE, I had to scrap this approach when I went from 2.1.3 to 2.2.0. Now we build LFE in the CI, which is slower but then I can do the upgrade PRs in parallel.
AFAK - yes. The run directive will take place inside:
The workflow context (which is something along the lines of __w/home/<mapped drive>/., and home is the repo you are running the CI in. So for the Python runner, I ended up with /__w/python-test-runner/python-test-runner/. The extra level in the directory got me.
The container/image, if one is mapped.
For the current changes to the Python test runner, I ended up calling the run tests in docker script directly … which built the runner container from the cached exercism/python-test-runner image and ran tests. But when it came to then running content tests, I had to be aware of where actions was building things.
Since I am fairly pathetic with these things, my main.yml is sort of a Frakenstein. I am thinking on it before deploying, so I might find a way to make it cleaner. But here is what I have working so far. Note that we run a build and cache action for the runner image first, then do the rest down the line:
ci yml file for Python runner
---
name: CI
on:
pull_request:
branches:
- main
paths-ignore:
- '.gitignore'
- 'LICENSE'
- '**.md'
jobs:
build:
name: Test Runner
runs-on: ubuntu-24.04
steps:
- name: Checkout code
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435
with:
install: true
- name: Build Docker image and store in cache
uses: docker/build-push-action@2eb1c1961a95fc15694676618e422e8ba1d63825
with:
context: .
push: false
load: true
tags: exercism/python-test-runner
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Run Tests in Docker
run: bin/run-tests-in-docker.sh
housekeeping:
name: Python Content Validation
runs-on: ubuntu-24.04
container:
image: python:3.13.5-alpine3.22
steps:
- name: Checkout Runner Code
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- name: Checkout Problem Specs
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
with:
repository: exercism/problem-specifications
ref: main
path: 'problem-specifications'
- name: Checkout Exercises
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
with:
repository: exercism/python
ref: main
path: 'python'
- name: Install Dependencies
run: |
pip install -r requirements-generator.txt
- name: Test Templates Status
continue-on-error: true
run: |
cd python
../bin/template_status.py -v -p ../problem-specifications
- name: Generate Tests from Templates
run: |
bin/generate_tests.py --verbose -p ./problem-specifications --check
- name: Lint Exercises and Data with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 ./python/exercises/practice --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 ./python/exercises/practice --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Check Exercise Examples against Tests
run: |
cd python
../bin/test_exercises.py --runner test-runner
Aup. This is the negotiation currently happening with Python - and why I am updating the runner first. But really, all tooling first needs to be done and merged, then the content. It’s extra fun when you have language syntax that changes …
One thing I like to is have separate workflows for PRs and for merging into main. For a PR, I only want GHA to test the exercise(s) that have changed. I have a little jq recipe for that: The script to run takes zero or more exercise slug names (zero means test all exercises)
- name: Run tests for changed/added exercises
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
# example: https://api.github.com/repos/exercism/tcl/pulls/390
pr_endpoint=$(jq -r '"repos/\(.repository.full_name)/pulls/\(.pull_request.number)"' "$GITHUB_EVENT_PATH")
gh api "$pr_endpoint/files" --paginate --jq '
# extract the unique exercise slugs from the files in this PR
map(
select(.status == "added" or .status == "modified")
| .filename
| capture("^exercises/(?:concept|practice)/(?<slug>[^/]+)/.+\\.(tcl|test)$")
| .slug
)
| unique | .[]
' | xargs -r bash bin/verify-exercises-in-docker
Hmm, that’s a good idea. I’d like to test all exercises for every PR to surface any unforeseen issues. For most of my tracks, the CI would be done in about two minutes so it’s not a problem. However, I could adapt your approach to run any added/modified exercises first and then the rest of the exercises separately. That lets me know up-front whether it passes, and I don’t need to fix breakages exercises one by one until the CI reaches the added/modified.
# Run all tests for commits to master
name: tcl / ci-test-all
on:
push:
branches:
- main
jobs:
ci:
runs-on: ubuntu-24.04
container:
image: exercism/tcl-test-runner:latest
steps:
- name: Checkout repository
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- name: Run tests for all exercises
run: bin/verify-in-test-runner.tcl
The workflow for a PR is the same except for the run command:
For a lot of reasons (even though I assisted in launching the Delphi Pascal track) we should probably think twice about having proprietary languages, unless sponsored. Then the “lot harder” becomes the burden of the commercial interests.
Not my monkey though, just first thing I thought when I read this in e-mail.
- and why I am updating the runner first. But really, all tooling first needs to be done and merged, then the content. It’s extra fun when you have language syntax that changes …