Since I seem to be the first one that tried to solve the (quite) new PasswordChecker exercise, I think I should communicate the problem I had. Although I had all the tests passing in my IDE (Visual Studio Insiders), when submitting my solution to exercism via the CLI it failed (I tried it 5 times) with the message “An error occurred while running your tests. This might mean that there was an issue in our infrastructure, …”
I cannot find anything undesirable in my code (you can watch it in the Community solutions, it is the first one published there), so perhaps the issue is really in the infrastructure.
If I can help with anything like more information (sorry, I don’t know anything about exercisms infrastructure or where I would have to look to see if something is broken or not set up correctly) or if the problem has been solved, please contact me.
I got the same result when I ran the tests on the unmodified exercise source. Expected test failures, but got the error instead.
My Mac is too old to run Powershell, so I don’t have any practical way to run the test scripts to see what is going on. Can someone please try them on Windows or on a Mac capable of running Powershell?
The exemplar solution times out in the editor so I don’t believe it’s specific to the OP’s code.
The exemplar works fine when tested locally against the test suite so that suggests the issue may be with the test runner. When I cloned locally the test runner repo and ran bin/run-in-docker.sh, I got the following error.
[2026-02-24 22:54:29Z] Running test runner for 'password-checker' solution...
Unhandled exception. System.IO.DirectoryNotFoundException: Could not find a part of the path '/solution/TestResults/tests.trx'.
at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
at System.IO.File.OpenRead(String path)
at Exercism.TestRunner.FSharp.Testing.TestResults.parse(ISourceText originalTestCode, ParsedInput originalTestTree, TestRunContext context) in /app/Testing.fs:line 208
at Exercism.TestRunner.FSharp.Testing.DotnetCli.runTests(ISourceText originalTestCode, ParsedInput originalTestTree, TestRunContext context) in /app/Testing.fs:line 266
at Exercism.TestRunner.FSharp.Testing.runTests(TestRunContext context) in /app/Testing.fs:line 300
at Exercism.TestRunner.FSharp.Program.runTestRunner(Options options) in /app/Program.fs:line 43
at Exercism.TestRunner.FSharp.Program.main(String[] argv) in /app/Program.fs:line 52
Aborted
fsharp/reference/implementing-a-concept-exercise.md at 21743d79f69b1a28a4cff48585b2dce267c57b88 · exercism/fsharp · GitHub says Theory should be avoided, but there’s a test using that tag in the test suite at fsharp/exercises/concept/password-checker/PasswordCheckerTests.fs at 21743d79f69b1a28a4cff48585b2dce267c57b88 · exercism/fsharp · GitHub. It looks like the test runner parses an XML output file generated by XUnit using src/Exercism.TestRunner.FSharp/Testing.fs. My shot in the dark is that maybe Theory is represented in the XML such a way that it crashes the test runner during this parsing stage. If so, we just need to create separate test cases for the inlined data and remove the Theory.
As to why the F# track repo CI didn’t catch this, it doesn’t use the test runner setup so the issue wouldn’t have happened.
@BNAndras , are you able to run the test runner manually? If so, would you please try some experiments to test your theory? I would do it myself if not for the limitations I mentioned earlier.
If you can run Docker, you should be able to clone GitHub - exercism/fsharp-test-runner locally and run the test runner directly. Running the test script at fsharp/bin/test.ps1 at main · exercism/fsharp · GitHub isn’t really the same thing because that just tests whether the exemplar will pass the test suite. The test runner also has to parse the results and report them, and it seems the issue might be in those later stages.
But regardless of that, I should investigate my earlier guess and run locally the test runner on the test suite once the Theory tag is removed. I can probably do that tonight.
Ok, if that turns out to be the case, I’d be happy to do the legwork to modify the tests accordingly.
But didn’t a solution pass the tests in the end? I think the problem has nothing to do with the tests but just suffer from the slowness of the test runner. But I might be wrong
Yeah, the solution passes locally. Other exercises pass both in the test runner and locally. Is the bit in implementing-a-concept about avoiding still correct? It seems a likely culprit at the moment.
Well, I tried again today but it’s still failing on the platform. Couldn’t one just remove the [<Theory>] tag in the tests, push and then try to submit some solution to sort out if it’s like BNAndras suggested? It would be great if it could be solved in that simple way…
The source can be found here: fsharp/exercises/concept/password-checker/PasswordCheckerTests.fs at main · exercism/fsharp · GitHub PRs are most welcome
Ok, I just removed the theory tag and wrote four single tests instead, see changed the tests in password-checker, removed theory tag by Teilzeitdenker · Pull Request #1372 · exercism/fsharp · GitHub
Successfully submitted my solution, tests are passing now on Exercism platform! Thanks for the tip with the problematic theory tag!
@BNAndras, @ErikSchierboom, FWIW, I managed to successfully run the test runner in a GitHub codespace, and sure enough, I managed to replicate the test runner crash on the exercise source at Add missing link; improve formatting (#1370) · exercism/fsharp@21743d7 · GitHub. Would it make sense to add a CI job to run the test runner on the exercise source? Maybe it could be focused to run the test runner only on the exercise(s) that was/were modified in the given PR.
I just noticed another CI gap: the password-checker concept exercise was never tested, because the exercises/Exercises.sln file was not updated to reference it. What would folks think about changing .github/workflows/test.yml to auto-generate the exercises/Exercises.sln and exercises/Refactoring.sln files from the config.json file?
Or, maybe adding a simpler check to ensure that exercises/Exercises.sln and exercises/Refactoring.sln are both up-to-date with respect to config.json?
Silly me. I had assumed that .sln files were XML-based, like .fsproj files. But since that is not the case, I believe neither auto-generating the .sln files nor ensuring that they are up-to-date would be very straightforward or reliable.
Another half-baked idea: remove both .sln files and modify bin/test.ps1 to derive the list of exercises to test from config.json. Since test.ps1 is already parsing config.json, I think maybe it should be fairly straightforward.
Other thoughts?
Well, they can be! With .NET 10, solution files can be in an XML format: The New .slnx Solution Format (migration guide) I’ve not yet done the work to support .NET 10, but updating the solution is something we could be doing right now.
I really like this!
I’ll work on upgrading the track to .NET 10. Maybe you would be interested in then writing the CI script?