Fsharp test runner seems to fail more often than before

Programming F#
1

Hi!

I’ve had 2 students in 2 days that both experienced the test runner failing on an iteration with the message

Your tests timed out. This might mean that there was an issue in our infrastructure, but more likely it suggests that your code is running slowly. Is there an infinite loop or something similar?

Please check your code, and if nothing seems to be wrong, try running the tests again.

The tests finish immediately when running local and none of them contains any heavy recursion or anything that indicates that the code is in risk of being slow.

Not sure if this is how it has always been but I have not noticed this frequency before so something might have happened.

Someone else reported the same thing 2 days ago Tests frequently appear to timeout on code that later passes with no functional changes · Issue #185 · exercism/fsharp-test-runner · GitHub

Anyways… just wanted to report this. Cheers!

2

Hey. Thanks for the report. Are they using the editor or the CLI?

1 Like
3

I am one of those two, so here’s what I’ve done. I’ve solved the exercise locally, got all tests passing. Then, submit via cli. Now the test runner online marks the tests as failing with the mentioned error message. If I now run the tests again from the online editor, everything is fine. But I’ll then need to submit an additional iteration…

2 Likes
4

OK, thanks. I think there’s something going on via CLI submissions. There should be absolutely no difference, and so I’ve not really investigated this route, but I just can’t replicate via the editor, and everyone that’s having the issue seems to go via the CLI, so I’m guessing it must be something to do with that route.

I’ll investigate. Could you let me know the exercise please?

5

It was yacht.

6

In @denismaier 's case it was the exercise “Yacht” and the other one I encountered having the same problem had it on the exercise “Interest is Interesting”.

8

Hmm, solutions time out after 20s so a few ms difference shouldn’t matter.

What’s the memory usage like for this? I presume not too much? But that’s another fail condition.

9

Does this include compile time or just running time? How long (clock time) does it take from you hitting “Run tests” to getting the response on this?

11

The issue seems to be with the test runner itself. I’ll wait for @ErikSchierboom to come back off holiday to look at it.

12

Bumping this since I’ve been running into this too. It happened 3 times when trying to submit solutions for Log Levels and it happened once on Lucian’s Luscious Lasagna.

13

Erik’s still on holiday, but will look into this next week :slight_smile:

1 Like
14

It might be a coincidence (or my imagination), but it seems like if I wait a minute after I submit a solution before I refresh the exercise’s page, it’s less likely to timeout.

15

(cc @ErikSchierboom)

16

After working on some more F# exercises, I’m pretty confident that it’s opening an exercise page too soon after submitting a solution from the command line that’s triggering these spurious timeout errors.