Custom test-runner cases

Exercism Support
1

Can someone talk me through how to create a custom test case in the cobol-test-runner. If students put STOP RUN or GOBACK in their code, the 3rd-party test runner tells them that they’ve passed all tests. This is … um … not good. It’s been suggested to me to create another test on top of example-fail etc. I’m not entirely sure how that’s to work as these aren’t syntax errors so the compiler will treat them as example-success and we’re using the compiler for the other example-* cases.

@KTSnowy @0xE282B0 @glennj @kotp

2

Would it not be sufficient to do a grep check to see if the code exists in their program and exit with with a non-zero status?

Click here for rough draft of bash script to check for `GOBACK` and `STOP RUN` statements. 
#!/bin/bash

# Function to search for GOBACK or STOP RUN in a file
search_file() {
    local file="$1"
    if grep -qiE '\b(GOBACK|STOP\s+RUN)\b' "$file"; then
        echo "Found GOBACK or STOP RUN in file: $file"
        return 0
    fi
    return 1
}

# Initialize a flag to track if any matches were found
found_match=0

# Check if any files or directories were provided as arguments
if [ $# -eq 0 ]; then
    echo "Usage: $0 <file_or_directory> [<file_or_directory> ...]"
    exit 1
fi

# Iterate through all arguments
for arg in "$@"; do
    if [ -f "$arg" ]; then
        # If it's a file, search it directly
        if search_file "$arg"; then
            found_match=1
        fi
    elif [ -d "$arg" ]; then
        # If it's a directory, search all .cob and .cbl files in it
        while IFS= read -r -d '' file; do
            if search_file "$file"; then
                found_match=1
            fi
        done < <(find "$arg" -type f \( -name "*.cob" -o -name "*.cbl" \) -print0)
    else
        echo "Error: '$arg' is not a valid file or directory"
        exit 2
    fi
done

# Exit with status 0 if no matches were found, 1 if matches were found
if [ $found_match -eq 0 ]; then
    echo "No GOBACK or STOP RUN statements found."
    exit 0
else
    echo "GOBACK or STOP RUN statements found in one or more files."
    exit 1
fi

This will exit status of 1 if it is found, 0 otherwise.

So, it may be a pre-test check style, looking for things that will stop COBOL outright, resulting in the no failures found for the running tests.

3 Likes
3

You start by just copy-pasting on of the existing directories in the tests directory with another name. You’d then modify the code in that new directory to contain the code you want to test (which in your case will include “STOP RUN”). Then you change the expected_results.json to match what you want it to return (likely fail) and then finally you modify the code the test runner executes to handle this special case.

1 Like