I don’t have (real) previous experience with GitHub workflows or anything like that, so I have a few basic questions:
How I know that my implementation of a CI is correct? My implementation just stopped to send me emails about fails, looks like it’s done tests as expected but I don’t know if it is really “done”.
According to what I understand, the Test Runner is needed to run the environment on the website but I couldn’t understand if it’s really needed to write on from scratch or not.
Sorry if the docs already cover this questions, I must have missed it and, anyway, I need some advices/opinions
According to what I understand, the Test Runner is needed to run the environment on the website but I couldn’t understand if it’s really needed to write on from scratch or not.
There is a generic test runner that can often be used as a base, but we also have many test runners some from scratch, others vastly diverging from this generic base. It is an essential tool to be able to run the track in the web editor, and you do not need to write it from scratch (but you could).
In general the steps are the same though:
get the solution prepared
run the test suite
collect the results
output in a compatible format
How I know that my implementation of a CI is correct? My implementation just stopped to send me emails about fails, looks like it’s done tests as expected but I don’t know if it is really “done”.
This has already clarified some of my doubts. I hope the generic test runner will be enough for my case.
In my case, I use TAP format output for tests, so if necessary, I need to adapt it to some other format, like json, right?
This really scared me now. I never would have imagined that I could do something like this alone. I have started working on the track about two or three days ago… Maybe it was me who provoked the incident.
You may at any time invite people via the forum to point you to existing solutions (surely some track already has a TAP-to-JSON converter), look at Pull Requests and comment on code. There is a helpful community here, and most people only need to know where they may help.
Thinking in using it as test, I have tried to enroll free-pascal track (it’s correctly configured "active": false and there’s no test-runner at the moment) and got a “422 An error occurred” page, is this error due to the track being disabled or is it due to the absence of the test runner? If it’s because the test runner, I could use it as test before active the track?
I’m pretty sure some internal plumbing needs to be connected by the staff before it’s visible on the site to you and other track maintainers. It shouldn’t be related to whether you have a test runner or not since we do have tracks without them for very specific reasons.
I have checked the awk, bash, perl, and raku tracks and all these tracks have a separate repository for the test runner. Is it really necessary to request another repo to implement it? If not, in what cases could I implement a test runner in a single repository?
Another related question: How can I verify that my test runner is really working?
Yeah, it’s always a separate repository since it’s track tooling, not track content. You verify the test runner works by running it against existing test cases and comparing the output JSON to the expected JSON for each test case. Common scenarios are that no tests passed, some tests passed, all tests passed, the solution file was empty, or that there were syntax errors. Some test runners may have additional tests, especially if they’re a v2 or v3 test runner since the functionality is more complex than a v1 test runner.
And if something doesn’t make sense to you after reading those docs, don’t be afraid to ask about it. There’ll likely always be things we could clarify better in the docs and then more people will benefit from it.
