Hello. While testing exam tests for an AI coding benchmark, we found that the poker example solution in the JavaScript track (exercises/practice/poker/.meta/proof.ci.js, as on main today) returns the wrong winner in some high-card and flush comparisons, and the current test file (37 tests) passes it.
In compareHandValues, the FLUSH / HIGH_CARD case starts with:
if (handValue1.card1 < handValue2.card1) return 2;
if (handValue1.card2 > handValue2.card2) return 1;
There is no if (handValue1.card1 > handValue2.card1) return 1;, so when the best hand so far has the higher top card, the comparison falls through to the second card.
The example/proof isn’t meant to be an exemplar solution; it is simply meant to show the exercise can be solved. Making them bug proof is not a goal.
Wouldn’t that be captured by the second test in the file? If not, why is that test different from the existing high tests?
Thanks, that’s fair, and we read the note on entropy.
Why the second test doesn’t catch it: the solution keeps the best hand so far and compares each next hand with it. In “highest card out of all hands wins”, every next hand has a higher top card than the one held (8, then 10, then J). So the comparison never meets the case the missing line covers: the held hand has the higher top card but a lower second card. ['2C 4D 6H 7S AS', '3D 4H 6C 8C QD'] is the smallest case of it: ace over queen, but 7 under 8.
So the gap is in the tests as well as the example: a solution with the same omission passes all 37 tests. For learners that may matter little. We noticed it because these test files are also used to grade AI-written solutions, since Aider’s polyglot benchmark copies the JavaScript track’s tests. There, a solution with this fault is counted as solved. One test with that case would close it. If problem-specifications is the right place, we can raise it there instead, or leave it with you.
We found similar gaps in go-counting and queen-attack: one-line faults that pass every test. We’ll only post those if you’d find them useful.
Given that the tests come from the problem specs, any proposed changes should go to the problem specs first, not to JS.
The purpose of the tests is to guide students (not AI) to a correct solution without necessarily testing every single edge case. If you can make the argument that a human is likely to implement this incorrectly without a test being added, ideally with a pointer to a human written solution with this bug, you can make a case to add this test to the problem specs.
If this is unlikely to be an issue in human written code, it probably doesn’t make sense to add the test.