Making a test generator for Groovy track

Exercism Building Exercism
1

As @BNAndras mentioned, reports on test generator tool should be in separate discussion.

2

Report on building test generator for Groovy track

Introduction

As advised by Glenn Jackman, I downloaded canonical data for largest-series-product exercise.

These data was downloaded from:

https://github.com/exercism/exercises/largest-series-product/canonical-data.json

…to the following folder:


C:\Users\Aksima\Exercism\building-exercism\canonical-data\largest-series-product.json

But I don’t know how Glenn uses his editor’s (vim) recording feature to automate the repetitive stuff. So a decision was made to write a simple Groovy generator script for such automation.

Studying existing solutions

Before writing a program, it is a good idea to study existing solutions. Because no generator scripts was found in Groovy repository, I started searching other repositories.

It turns out that C# track have such a tooling. Generators are located in csharp/generators folder. Instructions are located in csharp/docs/GENERATORS.md file. The most interseting part in generators folder is csharp/generators/Output/Rendering folders, where one can clearly see the functions and templates used for rendering the tests.

I guess that I will try to do something like this in Groovy track.

Preparing the Groovy repository for generator development

For generator development, a “tooling” folder was created on local machine:


Aksima@DESKTOP-NIMQE87 MINGW64 ~

$ mkdir exercism/building-exercism/tooling && cd exercism/building-exercism/tooling

And a new branch:


Aksima@DESKTOP-NIMQE87 MINGW64 ~/exercism/building-exercism/tooling

$ git clone https://github.com/Aksima/groovy.git

Aksima@DESKTOP-NIMQE87 MINGW64 ~/exercism/building-exercism/tooling

$ cd groovy

Aksima@DESKTOP-NIMQE87 MINGW64 ~/exercism/building-exercism/tooling/groovy (main)

$ git checkout -b test-generator-tool

Implementing file structure of C# track

Now we implement a file structure close to what we see on C# track.

Creating documentation file


Aksima@DESKTOP-NIMQE87 MINGW64 ~/exercism/building-exercism/tooling/groovy (test-generator-tool)

$ echo '# Test generators' > docs/GENERATORS.md

Creating folder for generators


Aksima@DESKTOP-NIMQE87 MINGW64 ~/exercism/building-exercism/tooling/groovy (test-generator-tool)

$ mkdir generators

Saving the work


Aksima@DESKTOP-NIMQE87 MINGW64 ~/exercism/building-exercism/tooling/groovy (test-generator-tool)

$ git add .

warning: in the working copy of 'docs/GENERATORS.md', LF will be replaced by CRLF the next time Git touches it

Aksima@DESKTOP-NIMQE87 MINGW64 ~/exercism/building-exercism/tooling/groovy (test-generator-tool)

$ git commit -am "Started work on generators"

[test-generator-tool c6d9276] Started work on generators

1 file changed, 1 insertion(+)

Aksima@DESKTOP-NIMQE87 MINGW64 ~/exercism/building-exercism/tooling/groovy (test-generator-tool)

$ git push origin test-generator-tool

Next steps

The next step planned is to find learning resources on Groovy and study the language a bit. As I am is completely new in Groovy and Java and never programmed anything above “Hello, World” in these languages, it is necessary for me to learn the basics before writing a generator script.

3

Report on building test generator tool, part 2

Resources used

  1. An article on command line builder at groovy-lang.org.
  2. Instructions on building command-line parsers using annotations and an interface.
  3. Official page for Spock test framework.
  4. Examples of test cases built on top of Spock framework.
  5. Picocli official documentation.

Preparing a project for test generator tool

Creating a project

  1. Open IntelliJ Idea.
  2. Click on “New Project”.
  3. Set name: “TestGeneratorTool”.
  4. Set location, for example: “C:\Users\Aksima\Exercism\building-exercism\tooling\groovy\generators”.
  5. Set build system: “Gradle”.
  6. Set latest JDK and Groovy SDK.
  7. Set Gradle DSL: “Groovy”.
  8. Click “Create”.

Adding necessary dependencies

We will use the Spock testing framework to make sure that our command-line interface is implemented correctly.

We replace all “testImplementation” lines in build.gradle with the line highlighted on main page of Spock framework (under Install and with Gradle).

We also add implementation line for the Picocli framework. Search it under Getting Started and Add as External Dependency.

You might also need to choose a groovy implementation compatible with Spock framework, Picocli and CliBuilder. I don’t know if there exists any guidelines on that, so the best solution for you is to make decision based on googling (I did a lot of googling here!).

Removing unnecessary folders

The folders “java” and “resources”, which were autogenerated by IntelliJ Idea, probably should be manually removed from both src/main and src/test folders for consistency with existing Exercism projects, which does not contain these folders.

Create specification for command line interface

Right-click on src/test/groovy folder and choose New → Groovy Class in context menu. Name the new file CommandLineInterfaceSpec.

Add necessary import at top of file:

import spock.lang.*

Mark the new class as extending Spock’s Specifiction class.

Specifying our expectations from command-line interface

Our expectations

We expect our command-line interface to support the following options:

  • -h, --help - to get help on usage;
  • -i, --input=[canonical data] - to define the path to canonical-data.json for the exercise;
  • -d, --repository=[repository directory path] - to define the path to directory where we downloaded the Groovy language repository.

Writing the first test case

Let us write the corresponding test cases. We start with test for ‘-h’ option:

class CommandLineInterfaceSpec extends Specification {
    def "Can get help on CLI usage." () {
        given: "Initialize command-line interface"
        CommandLineInterface commandLineInterface = new CommandLineInterface()

        and: "Parse arguments with -h flag"
        OptionAccessor options = commandLineInterface.parse(["-h"])

        expect: "Parsing -h flag sets the 'help' property to true."
        options.getProperty("help") == true
    }
}

Writing the first implementation

Obviously the test made in previous section failed, as we do not implemented this option. Let us write the implementation in CommandLineInterface.groovy file. We will use these instructions on building command-line parsers using annotations and an interface. To define the interface, a separate ICommandLineInterface.groovy file was created:

import groovy.cli.Option
import groovy.cli.Unparsed

interface ICommandLineInterface {
    @Option(shortName='h', description='display usage') Boolean help()
    @Unparsed() List remaining()
}

Now let us use this interface in CommandLineInterface.groovy file:

    def parse(args) {
        this.builder.parseFromSpec(ICommandLineInterface, args)
    }

Let us look if it passes the test. And it fails…

java.lang.IllegalArgumentException: argument type mismatch

It turned out that parse function, which was initially used, is more lenient than parseFromSpec and can accept any iterable, for example, a Groovy list, but parseFromSpec is more strict, expecting String... as the type of command-line arguments. Moreover, it turned out that parse and parseFromSpec have different return types: parse returns groovy.cli.picocli.OptionAccessor instance, but parseFromSpec returns the instance of class specified in its first argument.

So, the following fixes were made:

  1. The return type of parse method was changed to ICommandLineInterface.
  2. In the test, we replaced ArrayList (["-h"]) with String[] (new String[] {"-h"})
  3. In the test, the type of options variable was changed to ICommandLineInterface.

Let us run the test. And it passes now.

It the similar fashion, the tests for -i and -d command-line options were written and made passing.

What is next?

The next step is the parsing of canonical-data.json file.

4

Report on building test generator tool, part 3

Resources used

  1. The documentation for problem specifiations.
  2. The schema for canonical data.
  3. Groovy language official documentation.
  4. Parsing and producing JSON topic in Groovy language official documentation.

Parsing the canonical-data.json file

Investigating canonical-data.json file structure

The formal definition of canonical-data.json file can be found here. According to that definition, the canonical-data.json file consists of following parts:

  • exercise - exercise’s slug (kebab-case) - required.
  • comments - optional.
  • cases - an array of labeled test items - required.
    • labeledTestItem - either labeledTest or group of description and cases defined above.
    • labeledTest - a single test with following properties:
      • uuid - a version 4 UUID - required.
      • reimplements - optional.
      • description - a short, clear, one-line description - required.
      • comments - optional.
      • scenarios - scenarios to include/exclude test cases - optional.
      • property - letters-only, lowerCamelCase property name - required.
      • input - the inputs to a test case - required.
      • expected - the expected return value of a test case - required.

For simplicity, we want our test generator to just generate one test class with empty test methods (without any assertions in these methods). For that, we need the following parts:

  • exercise - will be used for test class name.
  • cases.labeledTest.description - will be used for test method name.
  • cases.labeledTest.property - as comment to what to call.
  • cases.labeledTest.input - as comment to which data to test.
  • cases.labeledTest.expected - expected result as comment.

Looking for tools to parse the file

Groovy language official documentation containg an interesting reading under Groovy module guides header. In particular, there are topics on Parsing and producing JSON as well on Template engines which we are going to use later.

According to Parsing and producing JSON topic, we can use JsonSlurper for the task of parsing canonical-data.json file. Let us give it a try…

Writing the tests and code for parsing canonical data

As with command-line application, we start by defining tests which our parser must pass. For the tests, we will use an example from problem specifications documentation, rewritten in terms of Groovy’s JsonBuilder:

    @Shared String sample

    def setupSpec() {
        JsonBuilder builder = new JsonBuilder()
        builder {
            exercise 'foobar'
            ...
        }
        sample = builder.toString()
    }

First we want to test if our parser can retrieve exercise’s slug:

    def "Can retrieve exercise slug."() {
        expect:
        new CanonicalDataParser(sample).getExerciseSlug() == 'foobar'
    }

Now we want to retrieve exercise slug from the contents of example JSON.

We start by importing JsonSlurper:

import groovy.json.JsonSlurper

And then we parse with JsonSlurper in the constructor of CanonicalDataParser class:

    LinkedHashMap specification

    CanonicalDataParser(String source) {
        specification = new JsonSlurper().parseText(source) as LinkedHashMap
    }

After parsing, the retrieval of exercise slug is as simple as:

    String getExerciseSlug() {
        specification["exercise"]
    }

The same job was done when writing the code for retrieval of description, property, input and description for each case in specification. To help in keeping data together, a class LabeledTestCase was created:

class LabeledTestCase {
    String description
    String property
    Object input
    String expected

    LabeledTestCase(String description, String property, Object input, String expected) {
        this.description = description
        this.property = property
        this.input = input
        this.expected = expected
    }
}

We also needed to consider recursive nature of test cases, so unlike getExerciseSlug method, the method to get cases in specification is recursive.

Next steps

The next step is to generate test class and test cases based on parsed data.

5

Report on building test generator tool, part 4

Resources used

  1. The documentation on template engines in Groovy.

Generating the test cases

Studying the documentation on template engines

The documentation on template engines is located here. It offer a lot of variants for template engines:

  • SimpleTemplateEngine - for basic templates
  • StreamingTemplateEngine - functionally equivalent to SimpleTemplateEngine, but can handle strings larger than 64k
  • GStringTemplateEngine - stores the template as writeable closures (useful for streaming scenarios)
  • XmlTemplateEngine - works well when the template and output are valid XML
  • MarkupTemplateEngine - a very complete, optimized, template engine

After studying the available engines, it was decided that SimpleTemplateEngine is more than enough for our purposes.

Writing test for the renderer

For our first test case for the renderer, we want a simple check: given a specification without any test methods, can we render just the test class?

This check looks like this:

class TestCasesRendererSpec extends Specification {
    def "It can render an empty test class"() {
        when:
        JsonBuilder builder = new JsonBuilder()
        builder {
            exercise 'foo-bar'
            cases ()
        }
        String sample = builder.toString()
        CanonicalDataParser specification = new CanonicalDataParser(sample)
        String renderedTests = TestCasesRenderer.render(specification)

        then:
        renderedTests == '''import spock.lang.*

class FooBarSpec extends Specification {
}'''
    }
}

What we can see from the above code:

  1. We want to add framework import before definition of class.
  2. We want to define a class extending Specification class. It name must be in Pascal case and should end with “Spec” suffix.
  3. After the class definition there must be curly braces, inside which we will place our test methods. For the first test case, there are no test methods, so there are nothing inside braces.

Now when test is ready, let us try to implement it:

Writing template

We want our renderer to generate output from template. First, we created templates folder at the root of the project. Second, we placed into that folder a TestClass.template file with the following content:

import spock.lang.*

class ${exerciseName}Spec extends Specification {
}

As we can see, this template already addresses a lot of points described above:

  1. It adds framework import before definition of class.
  2. It defines a class extending Specification class. It ends class name with “Spec” suffix.
  3. It places curly braces after the class definition, inside which we later will place the test methods.

Writing renderer

Okay, we have a template, how can we render the template?

To render the template, we will use SimpleTemplateEngine as described in Groovy docs:

class TestCasesRenderer {
    static String render(CanonicalDataParser specification) {
        String testClassPattern = Files.readString(Path.of('templates', 'TestClass.template'))
        LinkedHashMap bindings = [
                exerciseName: specification.exerciseSlug,
                testMethods : []
        ]
        SimpleTemplateEngine engine = new SimpleTemplateEngine()
        engine.createTemplate(testClassPattern).make(bindings).toString()
    }
}

In the code above, we see the following actions:

  1. We load template from templates\TestClass.template file.
  2. We create a map called bindings which binds values to each of the names defined in the template.
  3. We create a simple template engine to render our template.
  4. And finally, we render our template with the provided bindings to string.

Okay, let us check if it passes the test:

renderedTests == '''import spock.lang.* class FooBarSpec extends Specification { }'''
|             |
|             false
|             3 differences (95% similarity)
|             import spock.lang.*\n\nclass (f)oo(-b)arSpec extends Specification {\n}
|             import spock.lang.*\n\nclass (F)oo(B-)arSpec extends Specification {\n}
import spock.lang.*

Oops, it looks like we forgot to implement conversion from kebab-case to Pascal case. Let us fix it!

Writing text case converters

For the case conversion, we will try the following approach:

  1. First, we get list of all words in the name to be converted, taking into account it current casing.
  2. Second, we convert the list of all words to the name with desired casing.

Here are the implementations for each of the steps.

First step:

    static ArrayList<String> fromKebabCase(String text) {
        text.split("-")
    }

Second step:

    static String toPascalCase(ArrayList<String> tokens) {
        tokens.collect({ it[0..<1].toUpperCase() + it[1..-1].toLowerCase() }).join('')
    }

After these functions were written, we can use these in creation of the class name. And the test is passing now.

Rendering test methods

In the same fashion, we wrote the test for test methods generation, the test method template and updated the TestCasesRenderer.render() method. We made all tests passing.

Conclusion

We successfully built a test generator for Groovy track. This was quite an effort!

6

All three required parts are done: command-line interface, canonical data parser and test renderer. So we now have a complete and production-ready test generator for Groovy track.

1 Like
7

The pull request for test generator tool is ready now: