Report community solution's performance and timing

Most of the time I wonder which community solution has the best performance. It would be great if we could “sort” community solutions by the time it takes for the tests to complete. I know it depends on the Hardware, etc. But I am interested in at least a rough comparison!

I imagine such a feature wouldn’t be very helpful, because the tests for an exercise are rarely big enough for any performance differences to show. And most exercises aren’t really optimization problems either. In that context, I expect the time for tests to complete to be dwarfed by setup and teardown work, e.g. compilation, interpreter startup time, etc.

I have done benchmarking for some exercism puzzles for fun, but I had to set them up in a proper way and create custom, large inputs to see any difference between implementations.

While I agree that it could be useful, it’s just very hard to do. For example, our test runners’ performance depends on how busy the server is. Ideally you’d want to measure things in a controlled environment.

One thing we can do though is to write articles that go into the performance of various approaches to an exercise. Here is a C# example: Explore the 'Performance deep dive' article for Reverse String in C# on Exercism

Something that appears less problematic is to sort Community Solutions by the number of lines in addition to starred/newest. Without wanting to turn every exercise into code-golf, I find balanced abstractions to be more concise and I often enjoy these solutions which temper brevity with readability. YMMV.

1 Like

Indeed the articles about code performance and different approaches are intriguing. I do appreciate the authors and editors.

It looks like it can’t be helped for now, since those articles are not generated automatically. Maybe one can create a program that takes in various community solutions and run it under the virtually same conditions.

Note, most approaches to benchmarking tend to also be computationally expensive. Exercism runs on cloud VMs so computationally expensive operations have a financial cost associated with them, too, which may play a role.

I am also interested in benchmarking different community solutions. One way of tackling this could be to run the benchmarks in a VM locally or via action on someones own github repo.

Enhancing the CLI to get all the community solutions for a specific task, like exercism download ... --community-solutions=True could be a first step for making people working towards this goal.

Just in general: would it be (theoretically) possible to get the community solutions automatically? Or is there a show stopper?