Improvement to test display in the online editor

[andrerfcsantos]

I’d like to share an improvement made recently to the Go test runner. It addresses one of its long-standing and annoying limitations that impacted how some test results were shown in the online editor. As a result of this change, you should now see better test results reports.

A test runner that lies

To better showcase the impact of the change made, let me walk through what the test runner expected behavior looks like.

When you run the tests in the online editor for the Go Track, the go-test-runner is called and reports a response back to Exercism. The info of that response is used, among other things, to populate what appears in the “Results” tab on the right side of the editor. This is an example of an expanded test result in that tab:

This gives you a nice summary of the test. It shows the test case data, and the if statement that was used in the test.

However, this code is a slight lie of what the real test code is! The real test code for the example above looks like:

func TestWelcomeMessage(t *testing.T) {
	tests := []struct {
		name     string
		customer string
		want     string
	}{
		{
			name:     "Welcome message for customer with first letter capitalized",
			customer: "Judy",
			want:     "Welcome to the Tech Palace, JUDY",
		},
		{
			name:     "Welcome message for customer with only lowercase letters",
			customer: "lars",
			want:     "Welcome to the Tech Palace, LARS",
		},
		{
			name:     "Welcome message for customer with dash in name",
			customer: "Peter-James",
			want:     "Welcome to the Tech Palace, PETER-JAMES",
		},
		{
			name:     "Welcome message for customer with only uppercase letters",
			customer: "MJ",
			want:     "Welcome to the Tech Palace, MJ",
		},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			if got := WelcomeMessage(tt.customer); got != tt.want {
				t.Errorf("WelcomeMessage(\"%s\") = \"%s\", want \"%s\"", tt.customer, got, tt.want)
			}
		})
	}
}

Here you notice that:

• The TestWelcomeMessage function has a lot of test cases, not just one.
• Because it has a lot of test cases, the real code has a for loop, that is not present when the test runner presents just one of the test cases.
• The real test code has a call to t.Run(), which also disappeared when just showing one test case

It is the test runner doing this inspection and transformation of the real test code to the simplified version Exercism shows.

The problem

Before this change, the test-runner was only able to analyze and transform the code this way if all the test cases were in the same test function as the test code itself, like in the previous example.

In most practice exercises, that is not the case - test cases are put into a separate file called cases_test.go and the test code lives in a file <exercise_slug>_test.go. This made it so the test runner wasn’t able to recognize the test cases, only the test code. In the Exercism editor, this would appear something like:

This was less than ideal:

• It doesn’t show the data for the test code
• The for loop and if statement are no longer hidden away

The fix

This most recent change makes it so even if the test cases data and code are separated from the test code function (even on different files), the test runner is still able to associate each test case with the corresponding test code.

For the test run shown before, the report in the editor now looks like:

This makes it so the test run reports are more consistent, even if different exercises organize tests differently.

Acknowledgements

This change was possible by PR contribution by @meoconbatu1111 submitted back in 2023. Sorry for taking us so long getting this over the line and thank you for working on this.