I remember raising this issue when this project was started, if I can recall was the idea that a user wanted to run things locally they would have to run the test runner through the docker container (or they could chose to not do that on Linux (maybe also mac)).
The fundamental issue I think is that the testing happens with a custom made testing framework and that may need surrounding scripts to work. But again I haven’t looked at this for more than a year, so this is just what I recall.
I agree, and this seems to be where the thread left off on the topic of non-Linux systems.
So if we can get some help testing the Docker version on Windows and Mac once it’s set up, great.
For now, the more I think about it, it seems clear that my above “made for the user, intended to run one exercise’s tests only” script should live in the gdscript-test-runner repo.
Here’s a version with some clearer intentions, especially in error messages. Let me know what you think:
#!/usr/bin/env bash
set -e
# Script for testing a student's local solution to a single GDScript exercise.
#
# This script is intended to be run from any exercise subdirectory from the
# Exercism GDScript track, but can live one directory higher than the
# exercise subdirectories for convenience.
slug=$(basename "$(pwd)")
missing_gd_msg="Normally, the exercise subdirectory created by the exercism download command should contain this file."
general_help_msg="Please see https://exercism.org/docs/tracks/gdscript/tests for details."
if [ ! -f "${slug//-/_}_test.gd" ]; then
echo "Missing test file: ${slug//-/_}_test.gd"
echo $missing_gd_msg
echo $general_help_msg
exit 1
fi
if [ ! -f "${slug//-/_}.gd" ]; then
echo "Missing solution file: ${slug//-/_}.gd"
echo $missing_gd_msg
echo $general_help_msg
exit 1
fi
if [ ! -f "/opt/test-runner/bin/run.sh" ]; then
echo "Missing test runner file: /opt/test-runner/bin/run.sh"
echo $general_help_msg
exit 1
fi
solution_dir="$(pwd)"
output_dir="${solution_dir}/.test-output"
results_file="${output_dir}/results.json"
mkdir -p "${output_dir}"
(cd /opt/test-runner && bin/run.sh "$slug" "$solution_dir" "$output_dir") || {
echo "Test runner script failed."
exit 1
}
test_status="$(jq -r '.status' "${results_file}")"
if [ "$test_status" != "pass" ]; then
echo "Tests for $slug have failed:"
cat "${results_file}"
exit 1
else
echo
echo "Tests for $slug passed!"
fi
Sorry, I missed this earlier. That’s a good point about jq. @Meatball is there any reason we couldn’t change test_runner.gd to output in a non-JSON format that we can grep instead?
Or maybe JSON makes sense for CI, but we could have a separate student option that outputs in TAP or something? Students would be interested in seeing test output as it happens, anyway, rather than only at the end and only if there are errors.
You need the json output for the online test runner interface.
Well my initial idea of how this infrastructure would be built up is by using something like bitwes/Gut: Godot Unit Test. Unit testing tool for Godot Game Engine., I have not tested this framework, but I belive that you have to make a project and you can’t run it as a single script. But if you just can get that working, then I belive user testing would work without docker and you could export the result into XML and then use some Python (or any other languages) to process that into a json. I think there are several benefits from this kind of approach, I don’t know from a technical point how feasible something like this is to run in docker however.
The option we have chosen at the moment is a custom solution that is quite tailored around the exercism interface. In practice, nothing is wrong with that; most users will anyway use the online platform. And there is also nothing that hinders further development of this solution to be more “friendly” towards local development in the future as well. Like supporting different kinds of outputs and adding a PowerShell script and likely other stuff aswell.
OK, cool, so I could add a student-friendly, non-JSON option? I guess that would also fit @BNAndras 's comment I linked to above:
maybe the test runner should have an optional mode for students. They navigate to their solution folder and run “docker test runner whatever --offline”. The test runner picks up the current folder, parses the slug out, and then uses the current folder as the output directory for the results.json. Then we just need to highlight what the results.json is and how to read it (preferably with examples).
So mostly with the same idea, just removing jq dependence (only for the student option) so it’s easier to run outside Docker as well.
If students are going to need to install Docker (a pretty big dependency), then requiring jq inside Docker shouldn’t be a big deal. If you can avoid Docker, that helps reduce the dependencies a lot.
If you want shell script feedback, I’d be happy to help with improvements there (as one of the bash track maintainer) once y’all figure out the details of what you want to run and how Though at that point, working through the finer details via a PR might be easier.
So earlier in this track’s history, it seems the conclusion for local student testing was to (at least for now) support only Linux (though maybe Mac) natively, and support Docker for Win/Mac/Linux (I guess under Linux you might prefer Docker if you don’t want to put the test runner in /opt/ just to be an Exercism student). I’m ignorant of the state of built-in bash support on Windows nowadays — if there’s a reasonably easy way to support all three natively, that’s super!
On this track, once the godot executable is installed, the test infrastructure that needs to go under /opt/ is a combination of GDScript and bash code. The main test runner that outputs JSON is written in GDScript. The helper scripts then are for running that main test runner (on all tests, or in my script above, just on the current exercise) and using jq to parse the output.
There’s WSL which lets Windows users get access to a Linux environment, but it adds some work to set up.
There’s nothing wrong with tooling landing in /opt … but how are people supposed to get that all set up? /opt/test-runner is pretty generic. If this is a gdscript specific runner, should it have gdscript in the path somewhere? Imagine if other tracks (or unrelated software) also wanted to install a test runner; how quickly would there be path conflicts? When you’re talking about a Docker image which is used for one specific purpose, you don’t have to worry about that stuff nearly as much. When you’re talking about setting up software on the main OS, you need to consider that there’s other users of the OS.
Excellent point. I’m certainly up for changing that. (This is definitely an example of where what’s in the current test runner was created more with CI in mind than with local usage.) In fact, since it’s a custom test runner, I’d say it should live under /opt/exercism-gdscript-test-runner.
I would discourage of the usage of wsl here. I am generally for wsl, but since GDScript isn’t really a standalone language so can’t you really use it outside of Godot. Using Godot in wsl isn’t really a thing (you more or less need GUI to do most things), yes wsl does have GUI support but it is very meh, according to myself.
At the same time it could be argued that no one is going to do game development in a docker container, however I don’t really see how wsl makes the experience better/easier, considering that you will likely not use godot in wsl later anyways and wsl requires a bit of a setup aswell.
Ah…I think I misunderstood. Isn’t it possible to use WSL just for bash scripting, i.e., to launch the (Windows) godot binary in headless mode and parse the output? That’s what I’d been picturing, so if it’s not possible, I guess the Windows options are just Docker or full native, i.e., providing the equivalent PowerShell script. Or cygwin ;)
No wsl isn’t like wine it is more of a virtual machine. So you actually have to install kind of an operating system for it to work (and setting it up).
Now docker is quite similar from a technical view, it also runs on wsl. But it has another interface which I think can be more friendly.
In the end if we are talking about a bash script, is likely the best course of action to explore if it can be rewritten in PowerShell
Meanwhile, I noticed that Resistor Color Duo/Trio/Expert all come before the Resistor Color exercise in terms of the order they’re recommended to students. It seems to be based on the order in config.json. OK if I change that around? I guess having them one after the other (rather than interspersed with unrelated exercises) makes sense, though that part doesn’t matter to me — what doesn’t make sense is putting the easiest variant last.
Will do… OK if I make an exception for “expert,” which comes alphabetically before “trio,” though?
Or we can keep them alphabetical and just make them prereqs of each other? (I was mistaken: I originally wrote that apparently JS and Haskell do that, but that’s false. Just the description implies it in the JS track.)
Though at that point, working through the finer details via a PR might be easier.