Yeah, I will write a draft for installation.md. I although don’t have a mac, so that part may miss some sections.
1 Like
Sorry for a huge delay, my life was very busy for the past few months. I hope to have a bit more time now. I’ve pushed another test today, once I get a few more done I’ll start refactoring the runner. Currently it’s very messy, and Godot’s error handling doesn’t help 
2 Likes
I’m back! :) Today I’ve added a method that saves results in a .json file directly, without using the output of the running script. This will make things a lot easier, since there is no way to control the error (and greeting) messages printed by Godot ;) I’ve also fixed the 2 tests that were causing problems in CI.
I will proceed with adding more tests soon. I don’t have as much time for this task as I’d like to, so progress is slow, but nevertheless the task is moving forward! ;)
2 Likes
I have access to a Mac if you still need help with the OS-specific install documentation.
2 Likes
@BNAndras yeah, it’s a good idea to check if the Docker setup works correctly on Mac, just to be sure :D If you could confirm that, I would be grateful :)
1 Like
The point of docker is that, what runs on one machine will also run on another machine, so that should be rather universal, the question is should really the local test runner be using docker?
1 Like
My experience with Docker is that there are some things (configuration mostly) that only work on Mac or Windows. That being said, I hope there would not be any problems with my Docker setup, and (as @Meatball said) we should focus on the local test runner ;) Ideally, if there is time, we could check both ;)
1 Like
I mean Generally have I found that the exercism (test runners) docker containers can’t run on WSL for example, so I just use a pure linux enviroment every time I work with docker. And that is sorta all that matters, since I think @iHiD, isnt planning on running these docker containers in Windows servers (or?)?
1 Like
Ran the PR code on a M1 Mac and had some issues with lib64/ld-linux-x86-64.so.2 missing. This seems to be a Docker issue with Apple Silicon.
run-tests-in-docker log
./bin/run-tests-in-docker.sh
[+] Building 79.5s (14/14) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 528B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 213B 0.0s
=> [internal] load metadata for Docker 1.9s
=> [1/9] FROM Docker 1.6s
=> => resolve Docker 0.0s
=> => sha256:ec050c32e4a6085b423d36ecd025c0d3ff00c38ab93a3d71a460ff1c44fa6d77 1.13kB / 1.13kB 0.0s
=> => sha256:cf3cc0848a5d6241b6218bdb51d42be7a9f9bd8c505f3abe1222b9c2ce2451ac 424B / 424B 0.0s
=> => sha256:a2f229f811bf715788cc7dae1fbe8f1d9146da54d3fbe2679ef6f230e38ea504 2.32kB / 2.32kB 0.0s
=> => sha256:db76c1f8aa176de7f2698c761088ac1e18cdbbafa6210e405f310221c7a9ea6a 27.35MB / 27.35MB 0.7s
=> => extracting sha256:db76c1f8aa176de7f2698c761088ac1e18cdbbafa6210e405f310221c7a9ea6a 0.8s
=> [internal] load build context 0.0s
=> => transferring context: 48.04kB 0.0s
=> [2/9] RUN apt-get update 6.0s
=> [3/9] RUN apt-get install -y jq coreutils wget zip libfontconfig1 6.3s
=> [4/9] RUN wget https://downloads.tuxfamily.org/godotengine/4.0.1/Godot_v4.0.1-stable_linux.x86 61.3s
=> [5/9] RUN unzip Godot_v4.0.1-stable_linux.x86_64.zip 1.4s
=> [6/9] RUN mv Godot_v4.0.1-stable_linux.x86_64 /usr/bin/godot 0.6s
=> [7/9] RUN rm Godot_v4.0.1-stable_linux.x86_64.zip 0.1s
=> [8/9] WORKDIR /opt/test-runner 0.0s
=> [9/9] COPY . . 0.0s
=> exporting to image 0.2s
=> => exporting layers 0.2s
=> => writing image sha256:a76bcefdbb63e347f122ad81238332f6250146c931b29303810336333ab01110 0.0s
=> => naming to docker.io/exercism/test-runner 0.0s
example-all-fail: testing…
qemu-x86_64: Could not open ‘/lib64/ld-linux-x86-64.so.2’: No such file or directory
example-all-fail: done
example-all-fail: comparing results.json to expected_results.json
diff: /opt/test-runner/tests/example-all-fail/results.json: No such file or directory
example-empty-file: testing…
qemu-x86_64: Could not open ‘/lib64/ld-linux-x86-64.so.2’: No such file or directory
example-empty-file: done
example-empty-file: comparing results.json to expected_results.json
diff: /opt/test-runner/tests/example-empty-file/results.json: No such file or directory
example-incorrect-return-type: testing…
qemu-x86_64: Could not open ‘/lib64/ld-linux-x86-64.so.2’: No such file or directory
example-incorrect-return-type: done
example-incorrect-return-type: comparing results.json to expected_results.json
diff: /opt/test-runner/tests/example-incorrect-return-type/results.json: No such file or directory
example-partial-fail: testing…
qemu-x86_64: Could not open ‘/lib64/ld-linux-x86-64.so.2’: No such file or directory
example-partial-fail: done
example-partial-fail: comparing results.json to expected_results.json
diff: /opt/test-runner/tests/example-partial-fail/results.json: No such file or directory
example-success: testing…
qemu-x86_64: Could not open ‘/lib64/ld-linux-x86-64.so.2’: No such file or directory
example-success: done
example-success: comparing results.json to expected_results.json
diff: /opt/test-runner/tests/example-success/results.json: No such file or directory
example-syntax-error: testing…
qemu-x86_64: Could not open ‘/lib64/ld-linux-x86-64.so.2’: No such file or directory
example-syntax-error: done
example-syntax-error: comparing results.json to expected_results.json
diff: /opt/test-runner/tests/example-syntax-error/results.json: No such file or directory
1 Like
That’s not good :( I will continue working on tests, hoping that the issue with Apple will be solved. If I reach the end of my work with the test runner and the issue is still there, I’ll see if I can do anything about it.
1 Like
Hey there @pfertyk, I hope everything is going well.
I saw you mentioned that you are an indie dev, then I’m sure you must have saw the recent atrocious Unity announcement. I’m sure that move will drive a lot of people to Godot and having a track on exercism would be really awesome.
Whenever the track is ready, if you need people to help porting over exercises I would be gladly to help whenever I can.
3 Likes
Hi @glaxxie ! Thank you very much for your offer ;) I agree that Unity’s decision might push a lot of people toward other engines, including (hopefully) Godot. I also agree that it would be great to have an Exercism track ready to help them on their new path :) I was a bit overwhelmed recently, but given the circumstances I’ll put in some effort to focus on this task. I think the runner looks promising already (with some refactoring and extra tests coming up soon), and the initial exercises could be mostly ported from Python track (the syntax is very similar). There might be some other issues (like the one found by @BNAndras ), but hopefully we’ll find a workaround.
2 Likes
Let me know if you need any help porting exercises. I’m almost finished with a backlog of exercise PR reviews for Haxe so I’ll have some time.
1 Like
@glaxxie @BNAndras @Meatball I’ve realized that the test runner needs to support also checking variables (not only methods) so I’ve started adding this functionality. I’ll also add an ability to test multiple methods (both of these requirements come from the “lasagna” exercise that I keep seeing in multiple tracks ;) ).
Meanwhile, is it possible for you to start adding the exercises? The exact format of test files will be known once I finish with the runner (there is no unit testing in Godot, so I’m making this up from scratch), but perhaps the initial 20 exercises can be already selected? Assuming, of course, that the list is not fixed for new tracks ;) Maybe you could already copy the descriptions from existing tracks (using Python as an example), add proper .json configuration required by Exercism etc? Just to save some time and start this track faster ;)
What do you think? ;)
1 Like
Lasagna is a concept exercise and those can always be added later. The general recommendation is to focus on practice exercises which are easier to implement since we have canonical test cases and data for most of them. You’lol also need the test runner to be v2 or v3-compliant before you need to worry about concept exercises. Practice exercises work fine at v1.
Having gone though a lot of exercise PRs for Haxe and Pyret recently, I don’t recall a practice exercises that tests a variable. They tend to test methods, properties, functions, and those kinds of things. Being able to test multiple methods though will be necessary for several exercises (clock, grade-school, list-ops, etc.) so that’d be a good feature to add.
As to which exercises to port, there are a few suggestions here. leap, resistor-color series, space-age, and two-fer are good ones to add (likely without much difficulty) and lend themselves nicely to mentoring opportunities for students.
If the CI uses the Docker test runner image to test PRs before allowing a merge, then we could do partial implementations and then once the test runner is ready, add the test files. Then we can merge the PRs when the CI is eventually happy. However if Godot doesn’t have an unit testing framework, that’ll make it harder to check our exercise solutions while we work on them.
1 Like
@BNAndras Thank you for the feedback! Today I’ve added a test for multiple methods (I just moved the const METHOD_NAME to individual test cases). The code is already pushed ;)
With the info you provided, I now know what to focus on :) Tests for variables will be added later, multiple methods work already, I will add some other test cases (e.g. a method that raises an exception, some of the multiple methods not found etc). But I think the format of test files will not change a lot (especially with variables out of the way). You can check the examples in my PR, like this one:
const TEST_CASES = [
{"test_name": "Test Add 2", "method_name": "add_2_numbers", "args": [1, 2], "expected": 3},
{"test_name": "Test Add 3", "method_name": "add_3_numbers", "args": [1, 2, 4], "expected": 7},
{"test_name": "Test Hello", "method_name": "return_hello", "args": [], "expected": "Hello!"},
]
Please bear in mind that GDScript doesn’t have any built-in testing frameworks, so this is created from scratch. But, in earlier discussion, we decided that a simple format like this will be enough for GDScript exercises ;)
Knowing that format, is it OK to start adding the docs, configuration, and metadata for initial exercises? From my side, I would really like to see “Grains”, “Leap”, “Protein Translator”, and “Hamming” (but I won’t insist on these ones ;P ).
1 Like
The canonical data for Hamming, Grains, and Protein Translation all include error handling to various levels. Does the test runner have a way of testing whether an error occurred? Then, you could run a test case with an expected error message and see if the method causes an error with the same message. If you can’t compare the messages, you could just check if an error occurred for those tests expecting one. If an error occurred for tests not expecting one, they’d error out normally so I don’t think you need to assert that those won’t raise an error.
In a pinch, the tests expecting errors in Grains and Protein Translation can be excluded since they’re for input validation. For Grains, they make sure you can’t pass in a number too high or too low. For Protein Translation, they make sure incomplete / gibberish sequences can’t be translated, and the readme makes it clear the tests are skippable. Leap on the other hand doesn’t have any tests expecting an error so that’d be an easy one to add.
1 Like
I made draft PRs for leap, pangram, resistor-color, resistor-color-duo, and triangle since those are fairly straightforward and don’t have any error handling. Once the test runner is working nicely, we should be able to use that Docker image in the CI to test the exercises as they’re being merged in. That’ll speed up the exercise porting.
1 Like
Thanks for the PRs, I’ll review them soon ;) As for error handling in Godot, today I’ve added a new example to the test runner that shows a function validating its input and returning an error in some cases. Unfortunately, Error in Godot is just an enum, so effectively my test checks if the function returns a specific integer (in this case, 31). I’ll see if it makes sense to change the format of test runner files to check if the return value is an actual error, or if we can keep it as it is.
1 Like
I’ve added more tests, including some that are checking for “invalid” scenarios (e.g. the method is called with a wrong number of arguments, or the method throws an exception when it’s executed). Because of GDScript’s limited error handling mechanisms, I ended up wrapping the actual execution of user defined method in several levels of functions/assertions, and if something goes wrong, that execution returns “null”. So, I came up with the following rule: if a tested method returns “null”, it means that something went wrong and the test runner shows a generic message (without any details).
Is this OK? There is no “try … catch …” in GDScript, so this seems like the best thing I can achieve for our purpose. It is possible to filter out the error output from Godot’s execution, and maybe put this output somewhere in the test runner’s output files. However, that would require mixing up the GDScript code that saves the “results.json” files with the bash code that executes our test runner script, which might get dirty … Also, my solution doesn’t make sense if there are any exercises in Exercism that expect a method to return “null” in valid scenarios (are there any?). So, I was wondering if this part of the test runner meets the standards?
1 Like