In the Aider Python track’s ‘grep’ exercise, I’ve found an issue relating to the test implementation of the built-in open() function. The test suite employs a mock for open():
Python
def open_mock(fname, *args, **kwargs):
try:
return io.StringIO(FILE_TEXT[fname])
except KeyError:
raise RuntimeError(
"Expected one of {0!r}: got {1!r}".format(list(FILE_TEXT.keys()), fname)
)
This specific mock implies that the solver’s code is compelled to directly attempt to open() a file (and handle the resulting RuntimeError from the mock, or FileNotFoundError in a real scenario) rather than first checking for file existence using os.path.exists().
However, the problem specification for the exercise does not explicitly describe this constraint. From the perspective of standard Python programming practices and a literal interpretation of the problem instructions, using os.path.exists() to verify a file’s presence before attempting to open it is a perfectly reasonable and common approach.
When evaluating LLMs/agents using the Aider benchmark, this mismatch between the test’s implicit requirement and the problem’s lack of explicit guidance leads to a failure in a one-shot attempt for this specific case, as some models default to the more intuitive os.path.exists() strategy, which then fails the mock-based test.
Proposed Solutions
To address this ambiguity and improve the fairness of the benchmark, I propose the following solutions:
Direct File Provision: Similar to the JavaScript track, consider providing actual .txt files in the testing environment rather than fully mocking the open() function with io.StringIO.
Comprehensive File Operation Mocking: Implement a more extensive mock for file system operations that also covers functions like os.path.exists(), ensuring it responds predictably to simulate file existence or non-existence.
We generally encourage people not to use AI on Exercism. Using AI to solve Exercism exercises defeats the whole purpose of Exercism. Any test setup with causes AI to fail is, if anything, possibly a good thing as it encourages people to solve the exercises themselves.
Can you share with us what it currently describes? Ignoring all the AI stuff, we may be able to help improve the description which accidentally also helps the AI stuff.
That said: why does this need to one-shot when benchmarking? Wouldn’t it be a good thing that it fails and then “interprets” the error and applies the change? Checking an existence vs using exception catching isn’t idiomatically decided which is preferred or better. Both have their case.
But Aider clones exercism exercises to use as benchmarks:
The only Python track exercise that uses mocks (to my knowledge) is grep, which uses mock to avoid having a file to open … it fakes things out by having text in the test look like its being opened from a file. We’ve considered re-writing the exercise to use files, but its really really low on the priority list and we have a general ban on doing much file IO within exercises as interacting with the file system is out of scope for much of what we are teaching.
Because we consider the tests part of the spec, and different tracks implement their tests differently, we do not amend the descriptions unless there is a large deviation. The method here was considered “good enough”, and amendment was not needed.
But it is not clear to me how Aider is using the files and the tests within their system.
Barring that, I’m with DJ in his comment above: share what the problem description is that you are working on and what your code is that is causing the issue and we might be able to help with the non-ai assisted bit.
Thanks for the context regarding Exercism’s stance on file I/O and considering tests as part of the specification. I understand the priority constraints and the general ban on extensive file I/O within exercises.
However, let’s revisit the grep exercise from the perspective of a Python learner or beginner. （ignore anything of AI）
If I, as a Python learner, were to implement a solution for the grep exercise, adding a file existence check like:
Python
if not os.path.exists(file_path):
continue
…would feel completely reasonable and idiomatic based on common Python practices and the problem description itself. It’s a fundamental step for robust file handling.
The issue arises because, as you mentioned, the current mock implementation implicitly makes this perfectly valid approach incorrect for the test. Learners are effectively compelled to infer that they must directly call open() and handle a FileNotFoundError (or in this case, the RuntimeError from the mock) without any explicit guidance in the problem statement.
Furthermore, this contrasts with the JavaScript track, where similar file existence checks are allowed because it provides a more “real-world” simulation with actual files. This inconsistency can be confusing.
Such an experience can be frustrating and counterproductive for a beginner trying to learn Python. It forces them to “code to the test’s implementation details” rather than to the problem statement or idiomatic Python.
… and then you run the tests which tells you that these tests don’t allow for that. You read the test output, modify your code and rerun, following the test-driven development paradigm and solve the exercise, and all is well
Cross-track consistency isn’t the goal here. Fluency and TDD is the goal. If some inconsistencies force people to read the tests more carefully, this helps develop those TDD skills we try to build.
Test driven development is explicitly what Exercism asks students to practice. This is the goal, not an unintentional side effect! It sounds like this exercise is a success story in that regard ;)
To me, the trouble AIs have with such a simple issue is the benchmark for practical application. If an AI cannot figure out what’s going wrong here, it does not deserve a good mark…
