While investigating a small bug in one of the Java concept exercises and wondering why this wasn’t caught in CI, I noticed that the current CI jobs are not running the tests properly 
- On the Java track the script bails completely because it fails to parse
config.json, probably introduced when the concept exercises were added and the structure changed: sync saddle-points exercise (#2329) · exercism/java@d944d6e · GitHub - On the Kotlin track it is unable to extract the concept exercise slugs from
config.json, so only the practice exercises are tested: Fix markdown link not displaying for secret handshake · exercism/kotlin@ad36534 · GitHub
I already opened a PR to make the Java test script match the one from the Kotlin track, since it seems to be shared between them. Link to PR: Fix CI silently not running exercise tests by sanderploegsma · Pull Request #2339 · exercism/java · GitHub
However, another pressing issue is why the CI script is happily ignoring the failure, probably the reason that this went unnoticed. This is probably caused by the error occuring in a pipe while the script doesn’t set -o pipefail.
