Running pending Scala tests

Exercism Support
1

All but the first test in a Scala track exercise are marked with pending, which causes those tests to be skipped when ran locally. However, these tests run when submitted, creating a discrepancy, and often, test failures. Running these tests locally requires manually removing the word pending from each and every test, which seems like unnecessary manual work, and error-prone, since it’s easy to forget/miss.

ScalaTest runner has an argument -q that accepts a list of suffixes for test classes to be discovered. Instead of marking tests as pending, these tests could be put in a separate class named Additional or something like that, and the user can run these sbt "testOnly -- -q Additional" without needing to modify the test code. See this SO thread for details.

2

Hello @bruce-wayne!

I also noticed that the current usage of pending in tests is problematic.

I believe the idea was to remove them manually one by one, so you can focus only at single test at the time, but in the scala test runner we are stripping the pending all at once anyway, and as you mentioned - there is no easy way to run all pending tests.

I’m not sure if the separate class is the best approach here though.

I see two alternatives:

  1. We are committing to the pending tests, but we are using the ScalaTests tags instead of the actual pending method, which seems to be misused here.
    Using tags will give us ability to easily run all tests, or only non pending ones.
  2. We are giving up on the pending tests approach entirely

The pending tests pattern is used in other tracks too, so to keep it consistent, the option 1. might be preferred.

I’d want to revisit the problem in the scope of: Run exercises tests in a single sbt session · Issue #842 · exercism/scala · GitHub

3

@grzegorz-bielski I think using a custom Scalatest tag would work just fine. The goal is to have a way to skip/run some tests, and either putting them in a different test class or marking with a custom tag both solve that purpose. As you mentioned, the pending tag is misused here, since it’s meant for tests that have yet to be implemented, not for tests that are supposed to be run optionally.

1 Like
4

So we could use ignore then instead of pending?