I’ve been working on a GitHub Actions workflow that automates the configlet sync process for:
filepaths
metadata
docs
tests
The workflow is currently scheduled to run once a month, but the frequency can easily be tailored to suit each track’s needs or how often you’d like to stay in sync with the problem-specifications repo.
What It Does:
Automatically raises a Pull Request if filepaths, metadata, or docs are out of sync.
Automatically raises an Issue (with details) if test files are outdated or mismatched.
Uses configlet sync behind the scenes, powered by GitHub Actions.
Supports both scheduled and manual (workflow_dispatch) runs.
This is already being implemented on the Java track, and @kahgoh has been testing it to confirm everything runs as expected.
If your track is interested, I’d be happy to help you set this up or open a PR to get it started.
Thanks for reading, and looking forward to your thoughts!
Seems useful as an org-wide GHA tracks can opt into using. Common Lisp and Emacs Lisp both already have a similar scheduled workflow that creates an issue when docs, metadata, or tests are out-of-date.
As a GHA that doesn’t fire frequently, this could be useful for keeping unmaintained repos up-to-date on docs, filepaths, and metadata. Tests would still need to be manually updated of course, and this would help surface what tests still haven’t been updated.
I’d love to help implement this across other tracks. If I can get a sense of which tracks already have something like this in place (or don’t need it), and which ones would be interested in adding it, I can start raising PRs accordingly to add the workflow.
Happy to coordinate with any track that wants this set up, just let me know!
If it’s an Exercism GHA, you wouldn’t need to re-implement it per track and tracks can opt in on their own.
You’re referring to this, right? I’ll go ahead and raise a PR to make it an official Exercism GHA (edit- only after a mutual consensus is achieved). But since each track needs to opt in individually, they would still have to raise their own PRs, correct?
If that’s the case, is it okay if I raise the PRs on behalf of interested tracks, like yours, as well as JavaScript, TypeScript, and any others that want to opt in?
I’m not sure how you’d auto sync the tests as changes to the test metadata typically require manual edits to the corresponding test files. I’d be open to auto sync for metadata and docs for the tracks I maintain (awk bash jq sqlite), though my co-maintainers would need to sign off.
We should discuss it here and come to a consensus before any PR may or may not be made there. One thought is tracks may want an option to have an issue created instead of a PR for the docs, filepaths, and metadata. That lets them choose how they want to deal with the situation.
That would make the action slightly less useful for that particular track, since they already have the opportunity to review the files before merging the PR, which also acts as a final checkpoint to ensure everything looks good before accepting the changes from the GHA.
That said, if a track prefers issues over PRs, the change can be made quite easily!
I generally see a bit of an issue with the metadata sync. Since that will sync the toml file. Toml files are really good for keep track of whether a test file is updated or not. I have seen tracks which just have synced them so there is you there is no easy way of telling that the tests are out of sync. You could go back in the commits and see that the toml file was updated and not the test file, but that is annoying and a lot of just extra work (or the test file will just stay outdated). I suppose it is here that the issue system comes into the picture. But for that to work well someone actually has to take care of those issues; otherwise, the test might be unsynced to the docs, or issues stack on top of each other. I am thereby generally against this being implemented exercism-wide, but if individual maintainers want this, it is up to them. I think it is better to let it be out of date until someone takes a real sweep and updates the test files also.
The tests.toml file is only affected if one uses configlet sync with the --tests [mode] option. When I (and possibly Jagdish) refer to syncing metadata, I mean configlet sync --metadata which would sync the .meta/config.json files for each exercise.
Syncing tests would be almost certainly out of scope out of necessity. Not all tracks have generators in the first place, and those that do don’t necessarily have a standard API for the GHA to hook into.
Sure, but if you don’t know things are out of sync for a given repo, would you stop and check if it is? It’s a multiple-step process to fork the repo, fetch configlet, check if something needs to be synced, and then sync it. I’m not sure I would think to do it unless I already had the repo locally.
I mean if you do it like once every 6 months, there is very likely something out of sync. I have never done a sync and there hasn’t been any changes so I can’t say I have experienced that. I also often just spin up a codespace for this kinds of stuff. But if it doesn’t update the toml file i guess it is fine for unmaintained tracks.
As @BNAndras said, when I refer to the automatic PR, this command is used in the background: configlet sync --docs --metadata --filepaths -u -y. Hence the test.toml files are not updated with the automatic PR but rather an issue is raised mentioning all the exercises that have missing or reimplemented tests, so it’s up to someone to come and manually update the test cases after updating the test.toml for each exercise.
What It Does:
What It Does:
