Seeking maintainers to test the GDScript track

Programming GDScript
1

The GDScript track is available for testing to maintainers. Feedback appreciated if you can help find any issues before we go live!

1 Like
2

Some thoughts:

  1. The first paragraph of the About doc ends midsentence.

  2. Hello World test failure in the online editor:

    Expected output was ‘Hello, World!’, actual output was ‘Goodbye, Mars!’.

    It’s all on one line which might be hard to read as strings get longer.

  3. After submitting hello-world online, https://exercism.org/tracks/gdscript/exercises/hello-world returns error 502, bad gateway. Subsequently, the page renders fine.

  4. Installing doc

    • could use a direct link to the download page.
    • alternate script to install into /opt/test-runner:
      sudo mkdir -p /opt/test-runner
cd /opt
sudo chown $(id -u) test-runner
git clone https://github.com/exercism/gdscript-test-runner.git test-runner
    • provide copy-and-paste-able instructions to download the test runner script
      exercism download -t gdscript -e hello-world
cd "$(exercism workspace)/gdscript"
curl --remote-name https://raw.githubusercontent.com/exercism/gdscript-test-runner/refs/heads/main/bin/test-local-gdscript-solution.sh 
chmod a+x test-local-gdscript-solution.sh
    • the Godot install process is “download a zip file and extract”. That gives me an executable named “Godot_v4.7.2-stable_linux.x86_64”. The test runner script requires the command to be named “godot”. You’ll need to connect those dots, perhaps with ln -s Godot_v4.7.2-stable_linux.x86_64 godot

  5. Testing doc could just link to the Installation doc instead of repeating.

  6. Testing locally: the test runner script needs a fix:

    $ ../test-local-gdscript-solution.sh
Godot Engine v4.7.2.stable.official.ed1daf0bf - https://godotengine.org

ERROR: This runner requires being run with STDERR redirected to /tmp/stderr to capture errors
at: push_error (core/variant/variant_utility.cpp:1023)
GDScript backtrace (most recent call first):
    [0] check_stderr (res://bin/test_runner.gd:106)
    [1] _init (res://bin/test_runner.gd:31)
Test runner script failed.

    $ diff test-local-gdscript-solution.sh.orig test-local-gdscript-solution.sh
33c33
< (cd /opt/test-runner && godot --headless -s bin/test_runner.gd -- "$slug" "$solution_dir") || {
---
> (cd /opt/test-runner && godot --headless -s bin/test_runner.gd -- "$slug" "$solution_dir" 2>/tmp/stderr) || {
2 Likes
3

Thank you for the feedback!

  • The first paragraph of the About doc ends midsentence.

Fixed

  • Hello World test failure in the online editor:

It’s all on one line which might be hard to read as strings get longer.

I could put the expected and actual on their own lines. Tests with large output may require I write some custom pretty-print helper … but that’s lower on the priority list.

I’m pretty sure that’s not track related.

  • Installing doc
  • Testing doc could just link to the Installation doc instead of repeating.
  • Testing locally: the test runner script needs a fix:

I’ve been so focused on the web runner, I haven’t touched the local running issue, nor the install docs. I’ll update the Godot install docs.

I’m not sure how to best approach the local runner issue. The test runner/framework is some ~500 lines of GDScript spread over 3 files. Having a “shared_files” would help here :slight_smile: The best solution here might be to simply package those files with each exercise so a exercism download fetches the runner code.

The runner currently writes directly to JSON. I’m thinking I could add a --json <FILE> flag to the runner. With the flag, it will write JSON to <FILE>. Without the flag, it will write to STDOUT. The --json mode will require captured STDERR. The regular mode will not.

1 Like
4

The test docs still need updating, but the runner should work better locally now. The runner needs to be pulled from the test runner repo. Once I polish it, I’ll copy it to all exercise directories.

Running --all tests:

» godot --headless -s ./bin/test_runner.gd -- --all example-partial-fail ./tests/example-partial-fail/
Godot Engine v4.7.2.stable.official.ed1daf0bf - https://godotengine.org

-> test_add_1_and_2
-> test_add_10_and_20
-> test_add_0_and_3
-> test_add_0_and_0

Exercise has at least one failed test:
 test_add_1_and_2 passed ✔
 test_add_10_and_20 failed: Expected output was 30, actual output was 3.
 test_add_0_and_3 passed ✔
 test_add_0_and_0 failed: Expected output was 0, actual output was 3.

Default, fail fast.

» godot --headless -s ./bin/test_runner.gd -- example-partial-fail ./tests/example-partial-fail/
Godot Engine v4.7.2.stable.official.ed1daf0bf - https://godotengine.org

-> test_add_1_and_2
-> test_add_10_and_20
-> test_add_0_and_3
-> test_add_0_and_0

Exercise has at least one failed test:
 test_add_1_and_2 passed ✔
 test_add_10_and_20 failed: Passes 2/4 tests. First failure: Expected output was 30, actual output was 3.
 test_add_0_and_3 passed ✔

Usage:

» godot --headless -s ./bin/test_runner.gd -- example-partial-fail
Godot Engine v4.7.2.stable.official.ed1daf0bf - https://godotengine.org

Usage:
test_runner [--all] [--json filename.json] <slug> <solution_directory>

Expecting 2 positional args but got 1
1 Like
5

Install and testing docs updated and simplified. I reused your bash snippet to fetch the test runner code. I did away with needing to clone the test runner repo and using /opt; instead, it’s just one .gd script and a somewhat messy godot invocation.

The test runner now supports running without redirects and prints nicely to STDOUT.

Long values are not yet pretty printed.

6
  • installation doc: links not rendering on https://exercism.org/docs/tracks/gdscript/installation
  • testing doc: “The test runner expects two arguments: the slug and the path to the solution directory.” – can’t the slug be derived from the path? Or is this a snake-case vs kebab-case situation?
1 Like
7

Suggestion for local testing: supply the test runner and the launch script for each exercise, so when you download you get

./
├── atbash_cipher.gd
├── atbash_cipher_test.gd
├── HELP.md
├── lib/
│   └── test_runner.gd
├── README.md
└── tester*

The (pending?) bin/add_practice_exercise script can download it to the exercise directory, and there can be a bin/refresh_test_runners when changes are made in gdscript-test-runner repo.

My main concern in getting students to download it once is that it will get stale when changes are made in the repo.

And having the test script (tester) right there is just a convenience. It can be pretty small:

#!/usr/bin/env bash

die() { echo "$*"; exit 1; }

slug=${PWD##*/}

[[ -f "${slug//-/_}.gd" ]] || die "Error! solution file is missing"
[[ -f "${slug//-/_}_test.gd" ]] || die "Error! test file is missing"

godot --headless -s lib/test_runner.gd -- "$slug" "$PWD" || die "Test runner script failed."

I wouldn’t worry too much about the size of the runner, it’s smaller than bats-extra.bash

2 Likes
8

I’m not sure I can reliable and easily resolve relative links. I can accept just the path and derive the slug, but that will fail if a relative dir . is used. I can also simply document that an absolute path is expected.

9

Fixed.

Fixed … but won’t handle .

Done.

10

Any idea why this is failing the test runner? collatz

I have found an API endpoint that returns details about the last test run, but it's not very revealing... 
{
  "test_run": {
    "uuid": "6cc7de60-0d47-4dfe-8ea9-1c14f4888033",
    "submission_uuid": "39c4a5e72d1c4ca1ad09d5070661147c",
    "version": 0,
    "status": "ops_error",
    "message": "An unknown error occurred",
    "message_html": "An unknown error occurred",
    "output": null,
    "output_html": null,
    "tests": [],
    "tasks": [],
    "highlightjs_language": "gdscript",
    "links": {
      "self": "https://exercism.org/api/v2/solutions/2641cd61fa104f8abe14d2667fb30afd/submissions/39c4a5e72d1c4ca1ad09d5070661147c/test_run"
    }
  },
  "test_runner": {
    "average_test_duration": 3,
    "status": {
      "exercise": true,
      "track": true
    }
  }
}
11

sigh

The common test runner pattern is to mount the solution into /solution in the Docker container. It’s hard to extract the slug from that.

Revert: https://github.com/exercism/gdscript-test-runner/pull/78

1 Like
12

Test reran and passes

13

image
image2467×502 72.5 KB

What went wrong?

14

Does GDScript support list comprehensions?

15

Got this error this morning:

$ bash run_tests
Godot Engine v4.7.2.stable.official.ed1daf0bf - https://godotengine.org

Usage:
test_runner [--all] [--json filename.json] <solution_directory>

Expecting 1 positional args but got 2

Need to refresh the test runner for the exercises?

Also, can you add execute permissions to run_tests please.

16

Yeah. Please refresh the files. I set them as executable in the repo. I guess the CLI doesn’t copy that over.