Introducing: The Universal Test Runner!

Hey folks!

I wanted to share with you my latest open source project, the Universal Test Runner!

When I’m switching between languages (especially common when working on 12in23), I never remember how to run that language’s unit test suite. Sometimes it’s an npm test, othertimes it’s cargo test or go test or… you get the idea.

Instead of losing my muscle memory every month, I wrote a program that uses an advanced AI a series of if-else statements to determine what command to run (and then runs it).

You can see it in action here:

It already supports the following languages:

  • python
  • rust
  • javascript/typescript
  • clojure
  • elixir
  • go
  • makefiles

That mostly just covers my personal 12in23 journey, but I’ll be adding more languages and frameworks over time. Feel free to comment here or open an issue to add support for your favorite!


Mine isn’t nearly as well developed, but I accomplished something similar-ish in my editor. I use vim for all my editing and I can run tests directly from vim. I set up :E to save-and-test across languages:

autocmd FileType go command! E w|!go test
autocmd FileType python command! E w|!pytest
1 Like


This is supercool. Nice work! :slight_smile:

If you’d like to move this into the exercism namespace and maintain it there (to get extra visibility etc) then let me know, and I’m happy to set that up for you!

Thanks Jeremy! That’s a very generous offer.

I think I want to keep the main project where it is (since I’m planning things that are out of scope for exercism, such as django & rails support).

That said, I’d be happy to fork it into your org and release a version that’s a more Exercism focused. Since that’s a much more finite problem space, I could probably support any language which exercism already supports for local development. It could release as the exercism-test-runner or something.

Happy to chat more if that’s something you’re interested in.


Cool. That makes lots of sense.

I’d love to ship this as part of the CLI, but it would need to be an executable without dependencies to achieve that. So maybe we do a Nim or Go port (most of our tooling is in one of those two languages!).

Let’s see what @ErikSchierboom thinks when he’s back off holiday in ~10days! :slight_smile:

1 Like

I haven’t written Go in a bit but I think I could put together a Go version of this…

We could pull the track from the $CWD ($exercise_home/$track/$exercise) then just call a subprocess to run the tests. I’ll wait to hear back from you and/or Erik before attempting anything, though.

Alternatively … this could be part of the Exercism CLI tool! exercism test could run that logic.

That’s pretty sweet. I do something similar in my (bloated) fish wrapper for exercism

I love this idea! I’m fairly handy with go, so I’d love to spec this out. Pulling data from the local environment (probably .exercism/[config|metadata].json) is a great idea!

this is very cool! I love these custom wrappers.

Ok, I’ve got a basic PR for exercism test up here: Add `test` command to run any unit test by xavdid · Pull Request #1092 · exercism/cli · GitHub

I think I’ve got the important bits down. It works in go, rust, and ruby so far, and more will be easy to add.

Looking forward to hearing your thoughts over there!

If / when that ships, I can add an exercism option to my universal test runner which will plug into that, which will give everyone the best of both worlds. :grin:

1 Like

I quite like the idea of being able to more easily run all the tests via exercism test. As for the implementation, I think we have two options:

  1. Run the tests via the track’s suggested test command
  2. Run the tests via the track’s test runner

They have their own pros and cons:

Local command:

  • Pro: Fastest option
  • Pro: Could allow for configuring the actual command that is being run (not sure we would need that)
  • Pro: would need to account for potentially unskipping tests
  • Con: would still need the user to correctly setup any tooling on their machine

Test runner:

  • Pro: Would work for any track with a test runner (which almost all tracks do)
  • Pro: Would not require any local setup, apart from having Docker installed
  • Pro: would automatically handle the unskipping of tests
  • Con: memory intensive (Docker images can be quite big)
  • Con: require downloading the test runner

Maybe we should discuss these two options first before committing to one.

My sense is that I prefer the first:

  • The cons of downloading docker and test runner images (often multi-gb) are quite big in my eyes. I think if people want to not install tooling locally, then we have the editor for that, but I think if people want to dig into a language, they’re going to need to install some stuff locally anyway.
  • I also don’t think we’d want to have automatic unskipping of tests, so we’d need to find a way to stop all the docker images from doing that.

On the flip-side, if we were trying to achieve “run tests before upload” as part of the CLI workflow, then I’d be more pro-docker for the auto-unskipping and the consistency. But I don’t think that’s what we’re trying to achieve here (at least in this phase).

I think most are a couple of hundred megs, but many are indeed very big.

Maybe so. I’m not necessarily in favor of one option or the other, but I did want to point it out.

What Jeremy said. People developing locally usually have the local commands set up. Setting up Docker to run tests seems like a big ask to me.

As far as unskipping tests, in many tracks that’s handled via an environment variable. Those can be set via the tool when forking the child process that will run the tests. I vaguely recall other tracks may have other approaches but I’m not familiar with the details.

I can’t name one that does it like that :smiley: But we’re probably doing different tracks.

Yeah, I think this is true. Let’s go with local.

  1. agree, let’s go local
  2. skipping tests: I think that’s a personal preference that people can implement on their own
    • it will get tricky quickly if we try to do this on multiple OS’s
  3. think about how to structure the CLI source code for it with *nix/Win/Mac in mind.
  4. I’ve done this a lot, so I’ll have contributions, or just look at the various __exercism__test__*.fish scripts in

bash and Tcl do it.

1 Like

I agree on going local first- folks doing local dev will probably have (or be interested in) the requisite platform tooling.

re: skipping. Some tracks like rust instruct the user to edit the test file to remove the skip instruction. This approach will have the widest compatibility at the risk of a user changing or breaking a test file accidentally.

A safer option is ensuring wide support for environment and flag passthrough. This is how universal-test-runner already works and it brings a lot of compatibility for free. For instance, Rust supports the --ignore-skipped flag to cargo test. Each language has their own version of this, so we could amend instructions to say

run exercism test -- --ignore-skipped to run all your rust tests

Lastly, configuration. I think we launch the initial version of this feature without it. While some people will want to do per-track configuration, the core value prop (IMO) is It Just Works. I think if we ship an unconfigurable version, that’s still a huge win for most users. We can follow up with some of the discussed configuration options in a later release; I think we should start (and really nail) the most common case.

If that all sounds good, I can clean up the PR and add more tracks. I’ll likely need help writing the unit tests though, since mocking the filesystem and system calls in go isn’t something I’m particularly familiar with.

1 Like

I think for a lot of languages it wouldn’t be a simple thing to do. For C#/F# for example, it would require me to create a custom attribute. It’s possible, but might be more work than you envision.

That said, I do like the idea. I would make the command be something like:

exercism test --include-skipped (positive assertion) or something like that.

But for now, let’s focus on getting the most simple version merged.
One complication that you might not yet have considered is that the command to run might be different across os-es. For example having to run test.ps1 vs running
I think we need to support this at the very least.

Fair! I was using cargo as an example of a situation in which it would be useful to pass through flags to the underlying command. I think we could get that mostly for free, but given that some tracks require a local user to edit the test file, it’s fine to have that be the focus for everyone. Let’s keep it simple to start.

Agree! Do you have examples of languages that have platform-specific test commands? I would guess the two major cases are “windows” and “macos/linux”, right?