Jq: Better error messages when comparing objects

Let’s create better error messages when comparing objects.


Most exercises want the solutions to return a number, a string, or an array.
The tests compare the actual and expected result with assert_equal and print them if they are not equal. E.g.

 ✗ strand with invalid nucleotides
   (from function `assert_equal' in file bats-extra.bash, line 178,
    in test file test-nucleotide-count.bats, line 73)
     `assert_equal "$output" "$expected"' failed

   -- values do not equal --
   expected : Invalid nucleotide in strand
   actual   : some invalid result
   --

I like that as a student, I can see what my function should have returned (Invalid nucleotide in strand) and what it actually returned (some invalid result).

But a handful of exercises want the solution to return an object (not a number, string, nor array).
The tests call assert_objects_equal (defined in bats-jq.bash) which compares the actual and expected result but in contrast to assert_equal it does not display them. E.g.

 ✗ strand with repeated nucleotide
   (from function `assert_objects_equal' in file bats-jq.bash, line 46,
    in test file test-nucleotide-count.bats, line 45)
     `assert_objects_equal "$output" "$expected"' failed

Here I only see that the expected and actual result, I have to read the tests and use debugging techniques like the debug filter to understand why the test failed.


I think the test results would be better if assert_objects_equal
would print the expected and actual result, too. E.g. (mocked)

 ✗ strand with repeated nucleotide
   (from function `assert_objects_equal' in file bats-jq.bash, line 46,
    in test file test-nucleotide-count.bats, line 45)
     `assert_objects_equal "$output" "$expected"' failed

   -- values do not equal --
   expected : {"A":20,"C":12,"G":17,"T":21}
   actual   : {"XYZ":999}
   --

I think improving assert_objects_equal is easy.
We could change the implementation in bats-jq.base with lines from assert_equal (in bats-extra.bash):

 assert_objects_equal() {
     local result
     result=$(
         jq -n --argjson actual "$1" \
               --argjson expected "$2" \
             '$actual == $expected'
     )
-    [[ $result == "true" ]]
+    if [[ $result != "true" ]]; then
+        batslib_print_kv_single_or_multi 8 \
+        'expected' "$2" \
+        'actual'   "$1" \
+        | batslib_decorate 'values do not equal' \
+        | fail
+    fi
 }

That prints test results like this:

✗ strand with multiple nucleotides
  (from function `assert_objects_equal' in file bats-jq.bash, line 52,
   in test file test-nucleotide-count.bats, line 59)
    `assert_objects_equal "$output" "$expected"' failed

  -- values do not equal --
  expected : {"A":20,"C":12,"G":17,"T":21}
  actual   : {"XXX":999}
  --

It would affect the following exercises: shopping, etl, flatten-array, nucleotide-count, , robot-simulator, satellite, spiral-matrix, two-bucket.

4 Likes

This sounds good with the caveat that we may need a mechanism for dealing with large objects, where printing the complete expected and actual values is not helpful.

1 Like

With no attempt to handle large objects: the test runs generally output compact JSON, so it’s likely just long lines. Even so, it’s better for the students to have something to compare.

3 Likes

@siebenschlaefer thanks for the patch. Very helpful.

1 Like

I think I need to set the bats-*.bash files as invalidators. I just played with flatten-array on the website and the errors just say “failed”.

Another PR upcoming that will touch every exercise.

This updates the output but not the test result. Why would this need to rerun tests?

That is a good point.