Enabling debug output in jq for online editor

Other tracks (such as ruby) have special handling to capture debug output and display it to the users, like this:

I had thought this would be too hard in jq due to using bats for the test framework. I have figured it out though:

  1. the jq debug and debug(msg) functions output a debug array to stderr; the first element of the array is the string "DEBUG:". Ref https://jqlang.github.io/jq/manual/v1.7/#debug
  2. bats captures both stdout and stderr as the run command’s output.
  3. However bats provides file descriptor 3 as a mechanism for tests to output uncaptured text. For TAP compliant output, this text must start with a #. Ref Printing to the terminal and File descriptor 3
  4. All test cases for the jq track use
    run jq ...
    

With a jq function that redirects debugging output from fd 2 to fd 3, and a small change to the test runner, I can enable debugging output in the same manner as ruby.

This will be a really nice addition for jq learners working online.

implementation

  1. Each jq exercise gets a new file named bats-jq.bash

    #!/usr/bin/env bash
    
    # TODO some explanatory comments here ...
    
    jq() {
        command jq "$@" 2> >(
            while IFS= read -r line; do
                if [[ $line == '["DEBUG:",'* ]]; then
                    echo "# $line" >&3
                else
                    echo "$line" >&2
                fi
            done
        )
    }
    
  2. Every jq test-*.bats file gets a new “include” statement

    load bash-jq
    
  3. the result of 1 and 2 will not affect existing solutions: those will not contain any debug output (or else the tests would not have passed)

  4. the test-runner is adapted to handle bats TAP output that might look like this

    1..4
    # ["DEBUG:","Input is: 10"]
    ok 1 double a number
    # ["DEBUG:","Input is: foo"]
    ok 2 double a string successfully
    # ["DEBUG:","Input is: foobar"]
    not ok 3 double a string unexpected output
    # (from function `assert_output' in file bats-extra.bash, line 394,
    #  in test file test-with_debug.bats, line 20)
    #   `assert_output '"foofoo"'' failed
    #
    # -- output differs --
    # expected : "foofoo"
    # actual   : "foobarfoobar"
    # --
    #
    # ["DEBUG:","Input is: [1,2,3]"]
    ok 4 double an array is an error
    

    with the debug output before the test result, and error output after.

@IsaacG and other maintainers, what say you?

2 Likes

The test runner’s result.json file would look like this:

{
  "version": 3,
  "status": "fail",
  "test-environment": {
    "jq": "jq-1.7.1",
    "bats": "Bats 1.11.0",
    "OS": "Linux 6.1.0-21-amd64 GNU/Linux"
  },
  "tests": [
    {
      "name": "double a number",
      "status": "pass",
      "test_code": "run jq -f with_debug.jq <<< \"10\"\nassert_success\nassert_output \"20\"",
      "output": "[\"DEBUG:\",\"Input is: 10\"]\n"
    },
    {
      "name": "double a string successfully",
      "status": "pass",
      "test_code": "run jq -f with_debug.jq <<< '\"foo\"'\nassert_success\nassert_output '\"foofoo\"'",
      "output": "[\"DEBUG:\",\"Input is: foo\"]\n"
    },
    {
      "name": "double a string unexpected output",
      "status": "fail",
      "test_code": "run jq -f with_debug.jq <<< '\"foobar\"'\nassert_success\nassert_output '\"foofoo\"'",
      "message": "(from function `assert_output' in file bats-extra.bash, line 394,\n in test file test-with_debug.bats, line 20)\n  `assert_output '\"foofoo\"'' failed\n-- output dif
fers --\nexpected : \"foofoo\"\nactual   : \"foobarfoobar\"\n--\n",
      "output": "[\"DEBUG:\",\"Input is: foobar\"]\n"
    },
    {
      "name": "double an array is an error",
      "status": "pass",
      "test_code": "run jq -f with_debug.jq <<< \"[1,2,3]\"\nassert_failure\nassert_output --partial 'array ([1,2,3]) and number (2) cannot be multiplied'",
      "output": "[\"DEBUG:\",\"Input is: [1,2,3]\"]\n"
    }
  ]
}

This will help CLI users too: debug messages will no longer break the tests. The pretty bats output would look like (with colours)

$ command bats test-with_debug.bats
test-with_debug.bats
 ✓ double a number
   ["DEBUG:","Input is: 10"]
 ✓ double a string successfully
   ["DEBUG:","Input is: foo"]
 ✗ double a string unexpected output
   ["DEBUG:","Input is: foobar"]
   (from function `assert_output' in file bats-extra.bash, line 394,
    in test file test-with_debug.bats, line 20)
     `assert_output '"foofoo"'' failed

   -- output differs --
   expected : "foofoo"
   actual   : "foobarfoobar"
   --

 ✓ double an array is an error
   ["DEBUG:","Input is: [1,2,3]"]

4 tests, 1 failure

Sounds wonderful to me!

1 Like

I was going to mention that in regards to the Ruby track, the debug breaks for anyone that downloads with that debug statement there, and i would be great to not have that happen. The jq language as a debug function that exists and so it should end up being uniform for all users, rather than providing only a benefit for those using the Exercism platform test runner/reporter.

2 Likes

https://github.com/exercism/jq-test-runner/pull/33
and
https://github.com/exercism/jq/pull/244

1 Like

merged

5 Likes

:heart_eyes_cat:

Very very cool! Nice work.

2 Likes

Very cool. Nice work :slight_smile:

Thank you.

@ihid On the track-specific docs page, what determines the order of the left-hand panel? It’s not the same order as the config.json file. I notice the same is try for the Python docs.

I think it’s unordered, so I guess whatever the database returns. I’d have thought it would be primary key but that doesn’t align with the results. We should probably align it to the config.json - one for @ErikSchierboom when he’s back :slight_smile:

PR created to show the docs by their order in the config.json file: Display docs by position in config by ErikSchierboom · Pull Request #6986 · exercism/website · GitHub

1 Like

:blue_heart:

I imagine @BethanyG will also enjoy this PR for the Python docs.

1 Like

We’ve just deployed this change:

image

3 Likes