Something strange is happening with this new exercise (it went live yesterday), which I am still struggling to diagnose.
- Local testing works perfectly, with both Guile and Chez.
- CI on the GitHub repo also works perfectly.
- In Exercism’s online editor, testing just gives a deeply unhelpful “syntax error” message, with no details. I can see this, and @Borderite has also reported it.
- I saw this yesterday with the old test runner, and again today with the new one. The problem seems to be with the exercise, not the change in test runner. I submitted both of them, so it’s my problem in any case.
It may be best to avoid this exercise temporarily, while we try to fix it.
Apologies!
One more detail, for those not familiar with the inner workings of Exercism. Both locally and in CI, we routinely test proposed changes against all non-deprecated exercises: currently 39 on this track. Nothing about this exercise looked unusual.
I also tried several exercises in the online editor, and List Ops is the only one showing a problem. Please let us know if you see other problems.
I’ve pushed some small syntax changes in PR #360. Nothing that matters to local testing, but it seems the Docker tests (both old and new version) hit a problem.
This showed up a separate problem, which needs fixing for local testing in Docker. The Scheme track has been unmaintained for a few years, I became a maintainer about 10 days ago, and catching up slowly…
I don’t know if this is a helpful clue, but scheme-test-runner bin/run-tests-in-docker.sh also fails for me.
Backtrace:
2 (primitive-load "/opt/test-runner/tests/example-syntax-…")
In ice-9/eval.scm:
223:20 1 (proc #<directory (guile-user) 790ee8120c80>)
In unknown file:
0 (%resolve-variable (7 . @import) #<directory (guile-use…>)
ERROR: In procedure %resolve-variable:
Unbound variable: @import
Non-fatal error: (error #f "~A" "syntax error")
(I also think tests/prime-factors/expected_results.json should be deleted from the test runner repo, and example and run-example in the root Makefile )
1 Like
> `(,(lambda (x) (odd? x)))
(#<procedure>)
> '(,(lambda (x) (odd? x)))
(,(lambda (x) (odd? x)))
Your lambda is getting evaluated immediately and written. eval is later given your test cases as in-memory code so a procedure in memory isn’t a problem. Tests run okay locally and in Docker, but the issue comes when the utility code parses the test code. You’re using the test cases in-memory so the lambda is represented by the #. The breakage then occurs since the parser sends the parsed code back to the test runner. #<procedure> is treated correctly as a syntax error at this stage because Scheme sees #< and that’s not one of Scheme’s special literals like #t and #f.
1 Like
The run-tests-in-docker script is a job for tomorrow, I think. This track was set up with great skill and professionalism nearly a decade ago, but a recent period of neglect left it a bit adrift from Exercism’s 2026 standards.
So you need to change the quasiquote on the test case list that I linked to a single quote. Then the lambdas need to be changed because you no longer need the unquote (comma). At that point, we get a list with lambda as a symbol, not a procedure. The last bit is to wrap it inside list so the lambda gets evaluated and we get our procedure at runtime.
(,(lambda (x) (odd? x)) ...)
to
(list (lambda (x) (odd? x)) ...)
You don’t think the PR is going to work?
Not at a computer, but is the test CI passing in the Docker argument? If not, you’re likely running the regular execution which only checks if the test suite ran okay and not whether the results.json was checked. That’d explain why the CI was green on the earlier list-ops PR so we didn’t catch there was a defect in the parsing. I think perhaps the CLI should run the tests and check that a valid results.json was created with an overall “pass” status. Only then does an example solution pass.
After making your PR’s proposed changes locally, the tests do pass where they didn’t previously.
$ ./bin/verify-exercises-in-docker list-ops
Using default tag: latest
latest: Pulling from exercism/scheme-test-runner
Digest: sha256:ab9b8ab63a8fac1d2eb36141000c3426b3165ce0c782bd27573bc1a017ee1749
Status: Image is up to date for exercism/scheme-test-runner:latest
docker.io/exercism/scheme-test-runner:latest
Verifying list-ops exercise...
I believe It’s testing whether a valid results.json is created after a fatal error is encountered. Otherwise, the test runner wouldn’t produce anything and the diff with the expected results should fail. That’s similar to example-syntax-error where the fatal error is in the student’s code, not the test runner setup, so the tests complement each other. However, we probably want to rename the fixture like example-missing-test-suite so this intent is clear if I’m right about it.
I’m also noticing that the test files in the fixtures are stale and needed to be synced with our current test file setup. Several only have a test.scm to define the tests and run them, but test.scm should only have the test definitions and defer to test-util.ss for running them. Others have both test.scm and test-util.ss, but we should make sure both match how versions the track uses given the age of these files.
Lots still to do in other parts of the track, but I’m happy to report that this exercise now passes all tests online.
1 Like
As @BNAndras mentioned, the problem seems to be # appearing in the output from test.scm, which comes from the lambdas used in test.scm. I have made a quick fix to test.scm and gott a pass by submitting it together with my solution.
; These tests are auto-generated with test data from:
; https://github.com/exercism/problem-specifications/blob/main/exercises/list-ops/canonical-data.json
; File last updated on 2026-10-09T22:38:09+00:00
(load "test-util.ss")
(define test-cases
`(
(test-success "1. append: empty lists"
equal? my-append '(() ())
'())
(test-success "2. append: list to empty list"
equal? my-append '(() (1 2 3 4))
'(1 2 3 4))
(test-success "3. append: empty list to list"
equal? my-append '((1 2 3 4) ())
'(1 2 3 4))
(test-success "4. append: non-empty lists"
equal? my-append '((1 2) (2 3 4 5))
'(1 2 2 3 4 5))
(test-success "5. concat: empty list"
equal? my-concatenate '(())
'())
(test-success "6. concat: list of lists"
equal? my-concatenate '(((1 2) (3) () (4 5 6)))
'(1 2 3 4 5 6))
(test-success "7. concat: list of nested lists"
equal? my-concatenate '((((1) (2)) ((3)) (()) ((4 5 6))))
'((1) (2) (3) () (4 5 6)))
(test-success "8. filter: empty list"
equal? my-filter `(,odd? ())
'())
(test-success "9. filter: non-empty list"
equal? my-filter `(,odd? (1 2 3 5))
'(1 3 5))
(test-success "10. length: empty list"
= my-length '(())
0)
(test-success "11. length: non-empty list"
= my-length '((1 2 3 4))
4)
(test-success "12. map: empty list"
equal? my-map `(,1+ ())
'())
(test-success "13. map: non-empty list"
equal? my-map `(,1+ (1 3 5 7))
'(2 4 6 8))
(test-success "14. foldl: empty list"
= my-foldl `(,* 2 ())
2)
(test-success "15. foldl: direction independent function applied to non-empty list"
= my-foldl `(,+ 5 (1 2 3 4))
15)
(test-success "16. foldl: direction dependent function applied to non-empty list"
= my-foldl `(,/ 24 (1 2 3 4))
64)
(test-success "17. foldr: empty list"
= my-foldr `(,* 2 ())
2)
(test-success "18. foldr: direction independent function applied to non-empty list"
= my-foldr `(,+ 5 (1 2 3 4))
15)
(test-success "19. foldr: direction dependent function applied to non-empty list"
= my-foldr `(,/ 24 (1 2 3 4))
9)
(test-success "20. reverse: empty list"
equal? my-reverse '(())
'())
(test-success "21. reverse: non-empty even-length list"
equal? my-reverse '((1 3 5 7))
'(7 5 3 1))
(test-success "22. reverse: non-empty odd-length list"
equal? my-reverse '((1 3 5 7 9 11 13))
'(13 11 9 7 5 3 1))
(test-success "23. reverse: list of lists is not flattened"
equal? my-reverse '(((1 2) (3) () (4 5 6)))
'((4 5 6) () (3) (1 2)))
))
(run-with-cli "list-ops.scm" (list test-cases))
See Validate in CI that test source code can be parsed by BNAndras · Pull Request #363 · exercism/scheme · GitHub. We don’t run the Docker image directly so there’s no results.json but we parse the Scheme data that goes into the results.json so that’s the next best thing. That fails on the old list-ops test suite but passes on the fixed suite.
1 Like
I completely overlooked ticks before the lamdas, because the commas before the lambdas made me believe that the preceding ticks must be backquotes. So, I think that PR #360 has fixed the problem. The test runner now lets my solution pass.
I don’t think that what @keiraville saw was an error of the test runner. The particular test is for verifying that the test runner does not crash even when a student’s solution contains syntax errors. While error messages are shown in reflection of the stduent’s errors, the test runner completes the job comparing results.json and expected_results.json.
1 Like
