Skipping tests in the 8th track

Programming 8th
1

Some background:

It’s common practice in many Exercism tracks to have all tests (except the first one) skipped.
This enables students to focus on the TDD approach: get one test to pass before working on the next one.
I’d like to apply this to the 8th track.

My proposal is to introduce RUN and SKIP words, so that a test suite might be transformed from

4 tests

"Black" ( "black" color-code ) 0 test_eq
"White" ( "white" color-code ) 9 test_eq
"Orange" ( "orange" color-code ) 3 test_eq
"Colors" ( colors ) ["black", "brown", "red", "orange", "yellow", "green", "blue", "violet", "grey", "white"] test_eqa

end-of-tests

to

4 tests

RUN "Black" ( "black" color-code ) 0 test_eq
SKIP "White" ( "white" color-code ) 9 test_eq
SKIP "Orange" ( "orange" color-code ) 3 test_eq
SKIP "Colors" ( colors ) ["black", "brown", "red", "orange", "yellow", "green", "blue", "violet", "grey", "white"] test_eqa

end-of-tests

The students can be instructed to change each SKIP to RUN one at a time to enable the tests.

The test status line will be updated to show how many tests were skipped:

$ 8th test.8th
test-words.8th
resistor-color.8th
resistor-color-tests.8th
Black  ... OK
White  ... SKIPPED
Orange  ... SKIPPED
Colors  ... SKIPPED
4 tests planned - 1 passed - 3 skipped - 0 failed

Additionally, an environment variable can be set to ignore the SKIP directives and run all the tests, without having to edit the test suite.

$ RUN_ALL_TESTS=true 8th test.8th
test-words.8th
resistor-color.8th
resistor-color-tests.8th
Black  ... OK
White  ... OK
Orange  ... OK
Colors  ... OK
4 tests planned - 4 passed - 0 skipped - 0 failed

@axtens @softmoth, can I get your feedback?

2

Proof of concept implementation:
https://github.com/exercism/8th/commit/c0c7b542226bada4aa1582b03b057c1d30b00fa1

3

Thumbs up from me. An alternative would be to have a set of skip words, like:

SKIP-NEXT
SKIP-REST
5 SKIP-MANY

I imagine the implementation being a #p:skip-num variable that is checked and decremented by the test_foo words.

[Additional comment] One nice thing about this would be there’s no need for a RUN word. It would be handled by the fact that skip-num 0 =.

I’m not sure which version is more ergonomic. Let me know if my idea isn’t clear.

4

One reason why RUN/SKIP is easy to implement is that each test word knows it has a boolean value on the stack. I don’t have that much experience working in forth, so thinking with variables feels a bit unnatural.

SKIP-REST does feel like it would be a great candidate. Students can just move it down the tests: it would set a boolean variable that the test words can easily examine.

false ,var skip-test

: SKIP-REST true skip-test ! ;

: run-test? \ -- T
    skip-test @ !if true
    else "RUN_ALL_TESTS" getenv n:>bool \ the env var controls it
    then ;

: test_eq
    run-test? !if ... ;; then \ handle skipping the test
    ... \ execute the test
    ;
5

Hi glennj, I have not really thought about this but took a quick look on your code.

You use code like this a lot:

tests-failed @ n:1+ tests-failed !

You could write addition into var contents like this instead:

1 tests-failed n:+!

You can also replace:

;; then

by:

;then

To make code a bit more compact.

1 Like
6

Thank you so much! Committed.

7

Here’s what a SKIP-REST-OF-TESTS implementation looks like, for the acronym exercise:

I like this approach better than each test having a RUN or SKIP directive.

8

A pull request for review:

1 Like
9

I made a follow-up PR here: test-words: fix test_eqa2 to skip tests, normalize test-words.8th by softmoth · Pull Request #100 · exercism/8th · GitHub

Thanks!

10

I also found that RUN_ALL_TESTS=1 wasn’t working, because the getenv returns a string, and n:>bool doesn’t work on "1". Adding >n to convert it to a number first solves the problem.

11

Well, I did document to use RUN_ALL_TESTS=true. If you’re going to change the code, please also update the docs: docs/TESTS.md and exercises/shared/.docs/tests.md