All tests give the buffer of the same length as the input string. This is bad. People just return buffer without checking its length. Which is a bug.
You can fix it in line 7 of tests file:
var buffer: [s.len+42]u8 = undefined;
This makes the buffer longer and forces the student to return the correct slice.
What’s wrong with using the same buffer size for the return as the input string? That seems like a valid approach if it returns the expected string.
Imagine input string “hello” and a long buffer of length 10 initialized to “0123456789”. Currently many students return “olleh56789” and the tests do not notice the bug.
I mean the instructions never mention buffer length being the same as the length of the string. And the example explicitly checks the length of the buffer.
The tests drive the requirements, not the instructions. The instructions are there to explain the problem. The tests set the requirement.
The example is meant purely to show the problem can be solved and is there as a sanity test. It is not meant to be used as a guide for the requirements nor for the expected approach. See docs: example implementation.
Zig tests for the exercises below use constant buffer sizes that don’t depend on the input. I will change reverse-string to be similar. (Various other Zig exercises pass in an allocator, but changing reverse-string to use an allocator would break existing community solutions.)
bottle-song
food-chain
house
raindrops
run-length-encoding
twelve-days
two-fer
I do not believe this is always true. For example egg counting exercise instructions restrict using standard library fuctions and tests can not possibly enforce it.
Restrictions
Keep your hands off that bit-count functionality provided by your standard library! Solve this one yourself using other basic tools instead.
The instructions guide how you should go about solving it. The tests are the requirements. This is true of the egg counting exercise, too.
Some tracks do enforce that built in/standard tools cannot be used. Others do not and the exercise can be (and in many cases, is) completed using the built in/standard tools.
I am sorry. I am not intending to be rude. I am new here and I may not know how this thing work.
Can you give me a link to some documentation that says what you say? You told me the same thing twice which means you are confident that this is not just your opinion. Even if you were the inventor of Exercism you should have put that important information in documentation somewhere.
Until then I will keep my opinion that any contradiction between instructions and tests is a deficiency of tests.
Also please show me how to restrict the use of standard library or builtins in Zig specifically. I do not believe I saw it in tests of any of 60 exercises on Zig track I completed.
Those descriptions come from a repository shared by all tracks. So, no matter how hard the maintainers try to polish the text, it’s possible that it can’t account for every track’s specifics.
For instance, regarding eliud’s eggs, in one language there might be a bit-count function in a standard library, in other it might be an operator and there are many languages without any built-in popcount.
It’s generally expected that the student should interpret those warnings, and the description, in a way that makes sense for the specific track, rather than taking them literally. As far as I know, this is not something codified anywhere, it’s more a matter of common sense (the technical term is charitable interpretation).
I believe that what Isaac is trying to say is that, when in doubt about an exercise’s specific details, one should check the tests. If a solution passes the tests, it’s valid. Even when they contradict the description, and even more so if the description can’t be realistically enforced, as with eliud’s eggs in Zig.
Of course, the tests might have some flaw or you might have a different opinion on how they should be implemented. So there’s room for discussion. But it’s probably best to focus on how a specific change impacts the overall quality of an exercise, making it better.
By the way, in the x86-64-assembly track, the usual behaviour is to pass a constant-sized buffer larger than required by any of the tests. Then the student should return the size for the buffer, so the tests check if this size is equal to the size for the expected answer and if the values are the same up to this size.
For strings, however, it’s usually not necessary to return the size, since strings are NULL-ended and therefore their size can be clearly determined and checked by the test runner.
Also, many solutions for eliud’s eggs just return popcnt. And many solutions for square root just use sqrtss/sqrtsd. This is not the intention for the exercise, but people are free to ignore the instructions and do what they think it’s best for them.
I didn’t think it was in the docs. It took a fair bit of digging, but now I know where to find it 
Here’s said documentation.
On Exercism, the tests are the requirements!
All Practice Exercises you work on (those ones that don’t teach you a new concept) will have some instructions describing in general terms what you need to do. By design, these instructions do not account for programming-language-specific implementation details because they are shared by all of Exercism’s 70+ language tracks. Some language tracks will append more specific details for you, but not all of them do.
You have solved an exercise when all the provided tests run and pass. In other words, your solution is not just an interpretation of the instructions that “looks right”, your solution is a program that satisfies the given tests. The tests represent the complete requirements for the exercise.
With over 70 tracks, Exercism is heavily built by volunteers and maintainers. Exercism was built by @iHiD and @kytrinyx but there’s a lot of people here who are very involved with Exercism and can be very certain about how things without having invented it!
A track is a language. I’m not familiar with Zig. I know Python lets you mock out built in functions using the unittest.mock library. I’ve seen other tracks (languages) mock out various built in functions; I can’t recall which track, exercise or function (eg square root, bit count or list ops). We also generally don’t worry too much about students cheating themselves out of solving an exercise. If students want to take shortcuts, there’s very little we can do to control that.
@IsaacG has given some documentation nearby, but it is also documented in over a decade worth of conversations in various places. Isaac has been around long enough to have the cultural memory.
The README/Instructions should never contradict, and while the tests are not meant to be exhaustive, they are meant to provide disambiguation where the document that explains or introduces the problem may not be fully clear.
Also, there are enough of us monitoring that if Isaac were to state something that is not true, we would definitely step in and correct the statement(s).
That said, if there is a contradiction between the instructions and the tests, we would definitely want to correct that.
Thank you. That was mildly terrifying.
I agree.
This is exactly why I started this talk in the first place. There is something we can do. After I completed all Zig exercises I noticed the option to mentor other students. There is an option to use automation to review many similar solutions at once. What should I do there? Give a celebratory review to cheaters that solved “Eliud’s Eggs” with @popcount() builtin? Or should I tell them that it is essential to produce a solution without @popcount()? Maybe I should calm down and stay away form mentoring?
Thank you for keeping me honest
If you’re comfortable with the language, please do mentor 
I generally recommend:
If I was mentoring someone who was “cheating”, I would point out that the purpose of the exercise is to implement that functionality themself without using any built-in or standard library helpers. If I were filling in automated feedback, I might even mark it “Essential” and write something like the following:
That solution is perfect for production code. Using built-in/standard libraries keeps code simple and reduces bugs.
However, the whole purpose of this exercise is to give you an opportunity to reinvent that bit counting logic yourself! The instructions mention not using any standard functionality (which includes using built-in functions). Can you solve this without using the built-in
@popcount()function?
This one might not be simple in Zig track. There were dozens of unanswered requests for months before I answered them. I will try anyways.
Thank you.
Yeah. That can make things tricky. You could skip that step, but I think it’s generally super valuable to go through the process from the mentee side a few times. One options here are to specifically ask if track maintainers would be willing to mentor you to help you get familiar with the process in order to mentor others. Alternatively, if you’re at all familiar with another more popular language, you can try getting mentored on another track, too.