Well, I have to do something. I can’t have people getting false positives. I’m not about to write my own unit test tool and no one is volunteering.
Yes, it kills valid solutions. So the student will just have to resubmit without the stop run/goback.
Someone’s going to have to be offended and that’s all there is to it.
Bruce, the current version of the the test runner does not detect that a solution skips tests. This is the problem. Use of “stop run” or “goback” has very little to do with it.
As I have already explained, this mechanical problem can be easily fixed by modifying the test runner slightly. The test runner only needs to examine existence of the four-line test report at the end of the output from the executed solution.
You certainly don’t have to offend any students.
My understanding is that you’re saying we can patch the test runner to fail if any tests are skipped. That seems sensible to me.
Could you PR this please so we can understand the solution. Then we can be talking in specifics. Right now, I think everyone’s a little lost in conceptual ideas. Code helps firm things up.
Hi Jeremy,
Yes, that’s what I meant. I think that I can work on the patch pretty soon. Once I submit the PR, I will let you know here.
Hi Friends,
I have just submitted the PR I promised to work on. Please find it at
PR. Here is a summary of the PR.
The revised version of bin/run.sh examines if the last four lines of the output from the test program show the summary of the test results. Because absence of the summary implies that the program prematurely teminated skipping some or all tests, bin/run.sh fails the solution.
To verify that the modified test runner behaves in the intended way, one example case “example-premature-termination” has been added to the tests directory.
The docker build command seems to fail without a “–netowrk host” flag. The flag has been added to the docker build command in bin/run-in-docker.sh and bin/run-tests-in-docker.sh.
If I could reopen the cobol/test-runner#30 PR I would. But I have faith it will be reopened soon by a track maintainer or the guardians.
@borderite Thanks for taking this forward! Your editor seems to produce backups of the files edited (*~), which are part of your PR. You should remove them and commit the deletion, then add *~ to .gitignore and commit that. Changing .gitignore first makes it impossible to delete the files from the repository!
@mk-mxp Thanks for your review. I’ve removed *~ and addid *~ to .gitignore of the COBOL test runner.
Editor is “user choice” so does not belong in the project .gitignore but belongs in your excludes file, as indicated in ~/.gitconfig.
@kotp I see your point and will remove *~ from the track’s .gitignore. I don’t have *~ in my global .gitignore, because I sometimes want to keep the backup files in my github repositories.
You can ignore your “user choice” things in general and when you change your mind for that specific situation, you can force track them (one time work) to keep them tracked for that/those specific repositories.
I do this commonly for docs/ directories, while some repositories use that directory to generate and store their documents, others expect them to be generated on install or locally, or perhaps never. By default, I know I do not want them (tracked), but always defer to the project choice, and that is how I manage that.
Thanks for the info.
Thank you.
Could you split out any unrelated changes into a separate PR please? Else we’ll end up having a million discussions to get anything moving forward. If we could have something that just has your new test case, and whatever code is essential to make that test case pass, that would be ideal.
We’ll then be grateful for any follow on PRs that help improve things.
Hi Jeremy,
That is exactly what I intended to do. My PR only contains minimal changes to attain the simple goal I described. I have not incorporated any changes suggested by people, unless they were directly related to the PR. Do you have something else in mind?
you are absolutely right. I think the idea was fine and I have taken many such approaches in the past until I or someone else had the time and energy to fix it.
In this case Borderite had the time and energy! I have just checked the PR myself and the idea is sound, which will still allow people to use the jumping / goback.
KOTP isn’t a Cobol track maintainer. Typically PRs on maintained tracks are merged by a track maintainer. If you want to approve the PR via the forum and have someone else click merge, that can be done.
I do not have the magic dust for this track.
Okay, I’ll do the honours.
I can’t do anything.
Review required
Code owner review required by reviewers with write access.
All checks have passed
1 successful check
Merging is blocked
Waiting on code owner review from exercism/guardians.
I am totally out of my depth here.