POC for autogenerated Tests for PHP Track

Programming PHP
1

Hi,

After talk with some of you about how to improve the PHP Track, I thought about, how to ease some of the work.

@mk-mxp pointed me to the repository GitHub - exercism/problem-specifications: Shared metadata for exercism exercises. as we talked about improving the tests for the Nucleotide Count Exercise.

I then came up with the idea to automate this, so I created a small POC for that.

I cannot take all the credit for this https://twitter.com/VotrubaT helped me kickstart this.

Text from README.md

This is a small POC on how we could auto generate tests for the PHP track based on the GitHub - exercism/problem-specifications: Shared metadata for exercism exercises..

How to test it:

git clone https://github.com/tomasnorre/exercism-tests-generation.git
cd exercism-tests-generation
composer install
bin/console app:create-tests
vendor/bin/phpunit src/Command/NucleotideCountTest.php

If you now make a git status you will see that the src/Command/NucleotideCountTest.php and you can now inspect the auto generated tests.

It’s all based on the nikic/php-parser and the GitHub - exercism/problem-specifications: Shared metadata for exercism exercises. repository, I have made a local copy of that on file for now to spare the http-requests.

I have allowed to add some people in cc that I think this could be of interest for:
cc: @neenjaw @iHiD @glennj @ErikSchierboom

Let me know what you think.

1 Like
2

cc @joohan Thought you might find this cool too ;)

3

That really looks great so far! Is it possible to generate PHPDocBlocks, too? These would help to link the UUIDs and test descriptions to the generated code.

There, of course, is much more complexity in some of the problem specifications. But it is a promissing POC.

4

Do you mean docblock like?

/**
  * uuid: 3e5c30a8-87e2-4845-a815-a49671ade970
  * https://github.com/exercism/problem-specifications/blob/main/exercises/nucleotide-count/canonical-data.json#L5
  */
  public function testEmptyStrand(): void
  {
      $this->assertEquals(['A' => 0, 'C' => 0, 'G' => 0, 'T' => 0], nucleotideCount(''));
  }

We can in principle generate what we want, as long as it’s well structured.

5

Awesome work! We call these “Generators” in Exercism’s nomenclature btw. Many tracks have them if you’d like to look around to be inspired. C# is a good example: https://github.com/exercism/csharp/tree/main/generators/Exercises/Generators

6

You are totally right. Generators is a better word for it, didn’t put must thoughts into naming yet.

I’ll have a look at the csharp link you send. I’ll not be able to do the work yet though, for personal reasons, but would like to chime in with ideas.

Furthermore, I’ll be happy to help later on though, but I think I need to know more about the infrastructure etc. to ensure the best implementation.

7

Because I stumbled upon it right now: Advice for writing a test generator from the forum. It is not “documentation”, but contains helpful information about all the stuff around taking the JSON to output a test file. Many of your questions might already be answered there.

And be assured: Having a test generator in 6 months is better than having none forever.

8

One thing I would suggest the generator do is to be as “dumb” as possible. I went with a very complex setup for the C# generator, but that is much harder to write and maintain.

If I were to start again, I would go a different route, where the test generator is just a small shell around reading and pre-processing the JSON data from prob-specs and rendering that data via some templating system. I quite like what Crystal has done, where an exercise template looks something like this:

require "spec"
require "../src/*"

describe "<%-= to_capitalized(@json["exercise"].to_s) %>" do
<%- @json["cases"].as_a.each do |cases| %>
    <%= status()%> "<%-= cases["description"] %>" do
        <%= to_capitalized(@json["exercise"].to_s) %>.<%= cases["property"].to_s.underscore %>("<%= cases["input"]["phrase"].to_s %>").should eq("<%= cases["expected"].to_s %>")
    end
<% end %>
end

Just my 2 cents :)

9

I must have missed this in all the emails.
Looks good! :slight_smile:

1 Like