I’ve submitted the same exercise several times. In each case the item shows “Processing” and then “Passed” and then about 20 seconds later, without re-submitting, “Passed” changes to “Failed” all by itself! When I look at the test output in the iteration page it says they timed out. Something is very wrong here…
This problem has been seen for a number of other tracks.
The work-around I use it to copy-paste my local solution into the online editor, run the tests and submit. I’ve seen the tests timeout using the online editor too, but if you try again I find that the tests will pass … eventually. The main upside is that you don’t have failed iterations littering your solution history.
Alright, I think I have an idea of what has happened. What I think is causing this behavior is what I like to call “compile spikes” which refers to when the test-runner takes a lot more time than it should. So what is causing this? After my testing that I have got some testing on, is that swift caches some of the compiling so the next time you compile it should go faster. My idea is that when it times out, so are you given a docker container that hasn’t cached that exercise. The problem is that on my local machine + some docker testing, I found that there is an average compile time of 5-6 seconds. But when it “spikes” it can reach 15-50 seconds.
This forum thread:multiple-test-runners-not-working. Was about an issue where the docker container had not enough storage, these issues should be unrelated since this is should be swift specific.
Containers are never reused FYI.
I’m going to spend this evening digging into this more.