Yeah, I think a custom solution may be the best here. I will see if I can do some digging around to get the error message of an error run.
Test code, should more or less be just these "test_name": "Test One", "args": [1, 2], "expected": 3, {"test_name": "Test Two", "args": [10, 20], "expected": 30} and binding them to their test in the json file. Thereby shouldn’t be too hard, and I think that is the best approach. Output is not a requirement, a lot of tracks don’t have it and I personally really love having an output, but it could be quite a big nightmare to implement. So perhaps a skip.
But personally, I have not looked too much at godot 4, but since it has been released we should try to make so the track isn’t too hard to move over to that version (meaning we don’t build up something and then it has to be totally rewritten). I am personally not up on the “breaking” changes but just try to keep that in mind.
But otherwise, I think creating a docker file is a good next step.
Nice 
I was also thinking about Godot 4 … Now with an official release, and with some fixes coming up shortly, I think it might be a good idea to start with v4 instead of v3.5.1. This would have an additional aspect of helping people get started with the new version of the engine. Everyone in the community is super excited about this release and they are porting everything they have to Godot 4 now (I do the same 
).
The syntax is a bit different, but I think it’s mostly stuff like exports, signals etc. that are not going to be used in these exercises.
1 Like
I was actually thinking of suggesting starting of with godot 4. I think that it may be the best path, and will make this track have an easier time to grow.
Good news! I just checked Godot v4.0.1 and the runner works with 2 small changes:
-
JSON.print() needs to be replaced with JSON.stringify()
- there is no headless version of the binary, instead there is a
--headless flag
So, I would say we can proceed with Godot 4 ;) Who do I ping to create a test runner repository?
Sounds good, I have been sick for the last few days, so been very unproductive. I am on my way to getting better though and should be able to return to my usual schedule at the end of the week.
I think pinging @ErikSchierboom or @kytrinyx is the most suited person for creating a repo.
Sorry to hear about your sickness @Meatball , I hope you are feeling better now ;)
@ErikSchierboom @kytrinyx could you please create a repository for GDScript test runner ?
Fantastic, thank you! I will not be available this weekend, so I’ll start working on the test runner next week ;)
Hi again! Sorry for the delay, there is a lot going on right now for me ;) I’ve pushed some code changes today, it’s basically the test runner prototype that I’ve posted here earlier with some minor changes. I discovered a small issue with callv in GDScript, but a workaround was found. Currently, the “example-success” test is passing with a method that accepts 2 arguments.
I will be adding other test cases based on the Python test runner, and refactoring the runner’s code as I go (currently it’s one big file with 3 methods). Please forgive me if I take some time with this, as I mentioned there is a lot going on, and my availability is not perfect at the moment. However, I’m still interested in making this track happen! :)
Hi, sorry for not responding for a while, I have also been busy. But my schedule will soon be more free, if you would like so can I help you out with writing those test and I could also start writing some drafts of track docs to help out?
Hi! I also apologize for not contributing to this in the last 3 weeks, I expected to have a bit of time, but I ended up overwhelmed with other obligations … But my situation is also stabilizing, so I hope that soon I’ll get back to working on the test runner ;) If you have a moment until then, it would be great to have some help with the docs (both track and exercises), as you seem to be already familiar with the required structure. That way I can focus on the GDScript runner. Does this work for you? :)
Yeah, I will write a draft for installation.md. I although don’t have a mac, so that part may miss some sections.
Sorry for a huge delay, my life was very busy for the past few months. I hope to have a bit more time now. I’ve pushed another test today, once I get a few more done I’ll start refactoring the runner. Currently it’s very messy, and Godot’s error handling doesn’t help
2 Likes
I’m back! :) Today I’ve added a method that saves results in a .json file directly, without using the output of the running script. This will make things a lot easier, since there is no way to control the error (and greeting) messages printed by Godot ;) I’ve also fixed the 2 tests that were causing problems in CI.
I will proceed with adding more tests soon. I don’t have as much time for this task as I’d like to, so progress is slow, but nevertheless the task is moving forward! ;)
1 Like
I have access to a Mac if you still need help with the OS-specific install documentation.
@BNAndras yeah, it’s a good idea to check if the Docker setup works correctly on Mac, just to be sure :D If you could confirm that, I would be grateful :)
The point of docker is that, what runs on one machine will also run on another machine, so that should be rather universal, the question is should really the local test runner be using docker?
My experience with Docker is that there are some things (configuration mostly) that only work on Mac or Windows. That being said, I hope there would not be any problems with my Docker setup, and (as @Meatball said) we should focus on the local test runner ;) Ideally, if there is time, we could check both ;)
I mean Generally have I found that the exercism (test runners) docker containers can’t run on WSL for example, so I just use a pure linux enviroment every time I work with docker. And that is sorta all that matters, since I think @iHiD, isnt planning on running these docker containers in Windows servers (or?)?
Ran the PR code on a M1 Mac and had some issues with lib64/ld-linux-x86-64.so.2 missing. This seems to be a Docker issue with Apple Silicon.
run-tests-in-docker log
./bin/run-tests-in-docker.sh
[+] Building 79.5s (14/14) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 528B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 213B 0.0s
=> [internal] load metadata for Docker 1.9s
=> [1/9] FROM Docker 1.6s
=> => resolve Docker 0.0s
=> => sha256:ec050c32e4a6085b423d36ecd025c0d3ff00c38ab93a3d71a460ff1c44fa6d77 1.13kB / 1.13kB 0.0s
=> => sha256:cf3cc0848a5d6241b6218bdb51d42be7a9f9bd8c505f3abe1222b9c2ce2451ac 424B / 424B 0.0s
=> => sha256:a2f229f811bf715788cc7dae1fbe8f1d9146da54d3fbe2679ef6f230e38ea504 2.32kB / 2.32kB 0.0s
=> => sha256:db76c1f8aa176de7f2698c761088ac1e18cdbbafa6210e405f310221c7a9ea6a 27.35MB / 27.35MB 0.7s
=> => extracting sha256:db76c1f8aa176de7f2698c761088ac1e18cdbbafa6210e405f310221c7a9ea6a 0.8s
=> [internal] load build context 0.0s
=> => transferring context: 48.04kB 0.0s
=> [2/9] RUN apt-get update 6.0s
=> [3/9] RUN apt-get install -y jq coreutils wget zip libfontconfig1 6.3s
=> [4/9] RUN wget https://downloads.tuxfamily.org/godotengine/4.0.1/Godot_v4.0.1-stable_linux.x86 61.3s
=> [5/9] RUN unzip Godot_v4.0.1-stable_linux.x86_64.zip 1.4s
=> [6/9] RUN mv Godot_v4.0.1-stable_linux.x86_64 /usr/bin/godot 0.6s
=> [7/9] RUN rm Godot_v4.0.1-stable_linux.x86_64.zip 0.1s
=> [8/9] WORKDIR /opt/test-runner 0.0s
=> [9/9] COPY . . 0.0s
=> exporting to image 0.2s
=> => exporting layers 0.2s
=> => writing image sha256:a76bcefdbb63e347f122ad81238332f6250146c931b29303810336333ab01110 0.0s
=> => naming to docker.io/exercism/test-runner 0.0s
example-all-fail: testing…
qemu-x86_64: Could not open ‘/lib64/ld-linux-x86-64.so.2’: No such file or directory
example-all-fail: done
example-all-fail: comparing results.json to expected_results.json
diff: /opt/test-runner/tests/example-all-fail/results.json: No such file or directory
example-empty-file: testing…
qemu-x86_64: Could not open ‘/lib64/ld-linux-x86-64.so.2’: No such file or directory
example-empty-file: done
example-empty-file: comparing results.json to expected_results.json
diff: /opt/test-runner/tests/example-empty-file/results.json: No such file or directory
example-incorrect-return-type: testing…
qemu-x86_64: Could not open ‘/lib64/ld-linux-x86-64.so.2’: No such file or directory
example-incorrect-return-type: done
example-incorrect-return-type: comparing results.json to expected_results.json
diff: /opt/test-runner/tests/example-incorrect-return-type/results.json: No such file or directory
example-partial-fail: testing…
qemu-x86_64: Could not open ‘/lib64/ld-linux-x86-64.so.2’: No such file or directory
example-partial-fail: done
example-partial-fail: comparing results.json to expected_results.json
diff: /opt/test-runner/tests/example-partial-fail/results.json: No such file or directory
example-success: testing…
qemu-x86_64: Could not open ‘/lib64/ld-linux-x86-64.so.2’: No such file or directory
example-success: done
example-success: comparing results.json to expected_results.json
diff: /opt/test-runner/tests/example-success/results.json: No such file or directory
example-syntax-error: testing…
qemu-x86_64: Could not open ‘/lib64/ld-linux-x86-64.so.2’: No such file or directory
example-syntax-error: done
example-syntax-error: comparing results.json to expected_results.json
diff: /opt/test-runner/tests/example-syntax-error/results.json: No such file or directory
I was also thinking about Godot 4 … Now with an official release, and with some fixes coming up shortly, I think it might be a good idea to start with v4 instead of v3.5.1. This would have an additional aspect of helping people get started with the new version of the engine. Everyone in the community is super excited about this release and they are porting everything they have to Godot 4 now (I do the same 
).