It seems that most solutions I submit on the Kotlin track keep timing out. After having experienced this with submissons via the editor, I set up the CLI but it didn’t really help.
Currently I have six published solutions (four submited via editor, two via CLi) and another six ‘in progress’ that have successfully passed all tests locally but timed out on submisson.
The ‘timed out’ exercises are Scrabble Score, Grains, RNA Transcription, Reverse String, Pascal’s Triangle and Perfect Numbers.
Perhaps related to the timeouts I was getting in Rust? Rust: Why does this code take a long time to run?
Altho for me the timeouts are infrequent, and resubmitting a timed out exercise (with white space changes to make the submission runner think it’s a new solution) always passed the second time.
To maintainer: The issue might not be passing/short-lived. I had the same experience while doing 20+ exercises on the Kotlin track in the span of 16-20 days ago, most often they failed on submission (CLI), but as soon as I clicked publish on a failed test, the test suit would run again with what I assume is a higher timeout, because they mostly passed then. If not, I would make a minuscule change and resubmit. It did not seem to be correlated with the time complexity of executing the tests.
Thank you! I didn’t realize I could publish a solution even though it had failed. Now, after publishing all my solutions, I got several of them to pass successfully as a second (unchanged) iteration.
@ErikSchierboom Sorry to say that this is an issue this morning. Been trying to submit an exercise in the web editor with code that passed locally for Matching Brackets. Last time Matching Brackets seems to have passed successfully was about 16 hours ago.
The Kotlin test runner is still quite slow unfortunately :( I’ll see if I can do anything
Same issue here, unfortunately. I started using Exercism about a month ago for Kotlin and I don’t think I had a successful first-time submission. I just tried the C track to see if it was an Exercism-wide issue and everything seems to be working fine there, so it looks Kotlin-specific.
Tests run in less than a second locally, then timeout when submitted.
This is really interesting. So from the moment you press “run tests” to the moment you get your output, less than one second passes locally? If so, we should be able to reproduce your local setup on our systems. I do want to confirm though that it’s actually one second from the button click to the results - rather than just the system saying 1s but you waiting ~10s for the result?
Using Matching Brackets as an example, which is the one I was having timeouts with a couple weeks or so ago, running locally it is
D:\Exercism\kotlin\matching-brackets>gradlew.bat test
Starting a Gradle Daemon, 2 incompatible and 1 stopped Daemons could not be reused, use --status for detailsBUILD SUCCESSFUL in 18s
3 actionable tasks: 3 up-to-date
D:\Exercism\kotlin\matching-brackets>gradlew.bat testBUILD SUCCESSFUL in 1s
3 actionable tasks: 3 up-to-date
D:\Exercism\kotlin\matching-brackets>
You can see that it takes 18 seconds to spin up the daemon for the first time, but a subsequent run takes 1 second. This is using vscode in Windows, running gradlew.bat test with what comes downloaded with the exercise.
Yeah, so 19s total, which is more than the 15s we allow per test run 
I am getting faster results on my machine, running from Linux. I kill the Gradle daemon and clean the project (word-count). It takes ~8s in that situation:
[12:40:42] ng@desktop:~/exercism/kotlin/word-count$ ./gradlew clean
Starting a Gradle Daemon, 2 stopped Daemons could not be reused, use --status for details
BUILD SUCCESSFUL in 3s
1 actionable task: 1 up-to-date
[12:40:50] ng@desktop:~/exercism/kotlin/word-count$ ./gradlew --stop
Stopping Daemon(s)
1 Daemon stopped
[12:40:52] ng@desktop:~/exercism/kotlin/word-count$ time ./gradlew test
Starting a Gradle Daemon, 3 stopped Daemons could not be reused, use --status for details
> Task :test
WordCountTest > numbers are allowed PASSED
WordCountTest > one of each word PASSED
WordCountTest > multiple occurrences of a word PASSED
WordCountTest > heading substring PASSED
WordCountTest > quotations are allowed PASSED
WordCountTest > case insensitive PASSED
WordCountTest > cramped list PASSED
WordCountTest > punctuation is ignored PASSED
WordCountTest > various separators PASSED
WordCountTest > apostrophes are allowed PASSED
WordCountTest > expanded list PASSED
WordCountTest > one word PASSED
WordCountTest > multiple spaces PASSED
BUILD SUCCESSFUL in 7s
3 actionable tasks: 3 executed
real 0m8.177s
user 0m1.601s
sys 0m0.112s
Might it help if we compile everything in the test-runner images. Then the only compilation we have to afterwards are students files. Maybe something like this?
Well, the thing is: we don’t use Gradle in the test runner, instead we use Maven. This has resulted in a nice speed up, as Gradle is frustratingly slow. The main problem is that Gradle seems to be optimized for incremental compilation, where future changes to the code compile way faster, but we only ever have one compilation per submission.
Ideally, we’d move to something like this: Upgrade to test runner v3 by sanderploegsma · Pull Request #61 · exercism/java-test-runner · GitHub, where we directly compile without the overhead of any build system. I’ve done the same for the C# test runner and that helped a lot.
@sanderploegsma Do you know if it is possible to directly compile code for Kotlin too, just like your Java test runner PR?
I haven’t been able to look into it further, but I did a bit of searching a while ago. In general, most solutions I could find seem to leverage the fact that you can evaluate a Kotlin Script at runtime, but I’m not sure that will work in our case as we need to invoke the testing library with a class loader containing the compiled classes.
I found one blog post talking about actually compiling Kotlin code at runtime using kotlin-compiler-embeddable, that might be a route worth investigating: Hannos Blog: Programmatically compile Kotlin code
Has this been solved or improved? I ran tests for a few versions of the same exercise without issue! 
Not yet, but we are slowly making progress!