Incorrect solutions are shown as passing

Programming 8th
1

See for example the luhn exercise. There are currently two solutions, submitted via the editor, which have just the empty function declaration. But they are shown as passing the latest version of the tests.

2

cc @axtens and @ErikSchierboom and anyone else that fancies looking at the 8th test runner :)

3

My guess is that we’re not trapping the

Probable missing ; or : in <filename>

error.

4

It also “passes” if I put the single word bye as the solution, or have bye anywhere in the solution definition.

5

(Thinking out loud) Hmmm … that tricky. I might have to readdress the testing regime, and have a “number of tests” setting at the top the way the TAP testing framework does it. The simple framework given there isn’t helpful for Exercism tests which contain a descriptive name. Also TAP wants things to resolve to true or false.

6

I don’t know anything about the exercism testing setup, but would it be possible to rewrite test-words.8th so that prove -e 8th test.8th does exactly what is needed (using TAP directly)?

7

So I’m considering adding this to the top of all test-words.8th

0 var, test-count

: !test-count \ n -- 
    test-count ! 
;

: -test-count \ --
    test-count @ n:1- test-count !
;

Into each test word, e.g. test_eq, I’ll put -test-count and at the end of test-words.8th

( test-count-zero? 
  !if 
  con:red con:onBlack
  "FAIL - not all tests completed" . 
  con:white con:onBlack
  cr then ) onexit

Into each *-tests.8th file I’ll put the number of tests e.g.

13 !test-count

Thus a stray bye will show a “FAIL”.

Handling the absence of a ; I’m still working on.

2 Likes