Non-Deterministic Behavior for REST API tests

Programming PowerShell
1

For this test, RestApi test cases.iou -> borrower has negative balance, I’m getting test failures about half the, time where the entry for “Bob” has a different order for the “owes” item than the test is expecting. In other words, sometimes it appears like this:

@{name = "Bob"   ; owes = @{Adam = 3.0 ; Chuck = 3.0}; owed_by = @{}; balance = -6.0}

and sometimes it appears like this:

@{name = "Bob"   ; owes = @{Chuck = 3.0 ; Adam = 3.0}; owed_by = @{}; balance = -6.0}

Specifically, this is the failure I’m seeing:

 Expected: '...  "owed_by": {\n        "Bob": 3.0\n      },\n      "balance": 3.0\n    },\n    {\n      "owes": {\n        "Adam": 3.0,\n        "C...'
 But was:  '...  "owed_by": {\n        "Bob": 3.0\n      },\n      "balance": 3.0\n    },\n    {\n      "owes": {\n        "Chuck": 3.0,\n        "...'
            ----------------------------------------------------------------------------------------------------------------^

Since the test is comparing the expected value and the actual value using JSON strings, the order is very unforgiving. I managed to get this working well enough if I changed the order in which I process the borrower and the lender. However, I still see the occasional failure. Isn’t there a better way in PowerShell to compare two hashtables than converted them to JSON strings? Wouldn’t Compare-Object give more consistent results?

2

It is not stated explicitly in the instructions, but the tests seem to intentionally ensure that the names are ordered in the object returned. Comparing the JSON adds the ability to compare values and order at the same time.

You will find a solution to ensure the ordering of the properties, I’m sure.

3

If you look at the test, an unordered hashtable is being used for the expected value. I can change my code to use ordered hashtables, but if the test is using unordered hashtables, there will still be non-deterministic behavior.

As a matter of fact, for the one PR I did for PowerShell, I saw a random test failure in REST API. When the GitHub Action was run again by @glaxxie , the tests all passed.

4

I’m willing to do a PR to fix this.

5

Part of the reason I picked the json conversion is because it fit into the theme of the exercise and the ease of compare two complex dictionaries for pester. This test pass 100% locally with pester, however it failed and pass around 50/50 for the test runner (often the ubuntu env). Logically speaking there is nothing wrong with it, since we don’t really care about order at all, as long as the json or the object has correct values. But since json conversion is not keeping order but pester does want order, it’s marked as failed.
It was an issue that I procrastinate because as I said often if you run again, it just pass. So yeah I would very welcome a PR that can solve this problem once and for all if possible.

2 Likes
6
1 Like