We are working on the new Odin track and the question came out of adding benchmarks to some of the exercises. One of the worry is that adding benchmarks may cause the Test Runner to time out, frustrating students that are simply trying to learn the language. There is also the question of turning them off by default or not.
Here are some of the solutions we are discussing internally:
Solution 1 (Additional custom practice exercises)
Stay away from including benchmark in the normal practice exercises
Add standalone “optimized versions” of existing exercise (under a different but related name, like "Fast Pascal’s Triangle)
Make it clear that these are challenging exercise and that students should solve the regular version first (make it a prerequisite).
Solution 2 (optional benchmark in existing practice exercises)
Have the benchmarks in the exercise but with the @(test) attribute commented out.
Indicate in the instructions that benchmarks are optional and how to turn them on (not sure how to do this from UI though).
Create an approach document explaining the different possible optimization techniques usable for this problem(see Go pangram approach)
Add an article discussing performance in the context of the specific exercise (again see Go Pangram Performance article for an example)
Solution 3 (reserve benchmark for concepts)
Add specific optimization techniques concepts to the concept map
Pick an exercise where the optimization benefits are obvious
I like the Go track’s approach where you can optionally run benchmarks when working locally.
I feel like too much focus on benchmarking often leads to optimizing the wrong thing and leading to unreadable, unmaintainable code. Benchmarking is a nice add on and runtime optimization is important in a small number of places, but I don’t think it needs to be front and center.
I don’t know the language. Is it focused on speed or on “constrained” hardware? In other words, is optimization a core part of the language?
I think a great deal of work on creating concepts goes into designing things. Deciding:
What is fundamental and should be taught first before anything else. I think that ideally all of the fundamentals should be taught before making the syllabus live for students.
What is most important and must be taught at some point after the fundamentals. And
What is an useful or interesting extra and should be left for later, when the other concepts are already covered.
All of this of course depends on the language and where it is used.
I think that benchmarking might be center in languages where speed is center. But not in general.
Another thing of note is that on most cases speed is more about the algorithm than the language. And it is not the focus of exercism to actually teach algorithms and data structures, this lies beyond (or besides) language proficiency.
To me, Solution 2 is the best solution for the most people.
Solution 1 is the weakest because that effectively doubles the maintenance work for the maintainers. Despite whatever you end up doing to distinguish them, students may end up doing the benchmark version without meaning to, leading to the confusion you were trying to avoid in the first place. Personally, I wouldn’t make custom practice exercises. They’re surprisingly hard to get right, and the existing ones are battle-tested for the most part. Ballerina added a bunch of custom practice exercises, and in my personal opinion, they aren’t bad but they stick out like a sore thumb to the exercises sourced from the problem-specifications.
I strongly prefer Solution 2 since it works for the most people. Folks can engage with benchmarking locally if they want to, but they’re not forced to. There’s no confusion over which version of an exercise to make. If folks want to benchmark, everything’s right there in the exercise they already started on so they don’t need to copy and paste their code from the non-benchmark exercise to the benchmark one.
I don’t have a strong feeling on Solution 3 since I’d need to see an example first. I don’t believe I’ve seen optimization techniques as a standalone concept on other track syllabi. If benchmarking is only in the concept exercises, then that’s a missed opportunity to surface it in the practice exercises. Not everybody necessarily does all the concept exercises so they might miss out under Solution 3.
Oops, I forgot to add that the test runner needs to run all implemented tests. I wouldn’t expect it to also run the benchmarks since you’re penalizing the student for an unoptimized solution that does pass all the other usual tests.
So you could just comment-out the benchmark tests and then have a comment above it to uncomment that section if working locally to run those tests as well.