Test Runner issue in Python (Found in Meltdown Mitigation)

Programming Python
1

While working in Meltdown Mitigation (Python route) I was performing some troubleshooting by outputting function argument variables with print() statements.

After numerous runs it became apparent that the ‘Your Output’ section of the test runs did not properly coincide with the tests being run.

After working with Discord users, I was asked to present this as a possible bug in the test runner.

Code Run
temp = 10
threshold = 10000
test_data = ((399, 'LOW'), (300, 'LOW'), (1, 'LOW'),
             (0, 'LOW'), (901, 'NORMAL'), (1000, 'NORMAL'),
             (1099, 'NORMAL'), (899, 'LOW'), (700, 'LOW'),
             (400, 'LOW'), (1101, 'DANGER'), (1200, 'DANGER'))

for variant, (neutrons_per_second, expected) in enumerate(test_data, start=1):
    with self.subTest(f'variation #{variant}', temp=temp, neutrons_per_second=neutrons_per_second,
                      threshold=threshold, expected=expected):

        # pylint: disable=assignment-from-no-return
        actual_result = fail_safe(temp, neutrons_per_second, threshold)
        failure_message = (f'Called fail_safe({temp}, {neutrons_per_second}, {threshold}). '
                           f'The function returned {actual_result}, '
                           f'but the test expected {expected} as the return value.')

        self.assertEqual(actual_result, expected, failure_message)
Test Failure
AssertionError: 'DANGER' != 'NORMAL'
- DANGER
+ NORMAL
 : Called fail_safe(10, 901, 10000). The function returned DANGER, but the test expected NORMAL as the return value.
Your Output
1099

There was also an instance where the output was looping, but there is no loop code in my solution.

image
image516×592 40.1 KB

2

Hi @4verage :wave:

Welcome to the Exercism forums!

Could you post the code that you were using to get the buggy output? That would be helpful to diagnose what the issue is.

It would also be helpful if you posted a link to the Discord discussion about this, since I have not seen any discussions today on Discord around this issue.

I have no doubt that there is a bug somewhere. We’ve had issues in the past, and I am actually in the middle of fixing bugs and upgrading our tooling and Python versions. Unfortunately, the most recent version of PyTest (which our test runner uses) has changed the way it captures print() output, so there’s some significant twiddling involved.

Meanwhile, if you really need the printed output, your best bet is to use PyTest locally (or I can do it for you and post the output)

Many thanks!

3

@IsaacG was helping them in a #get-help Discord thread earlier, but I guess it was closed at some point.

4

The Discord post got closed, making it a bit harder to find.

Code which can be used to reproduce the issue:

def is_criticality_balanced(temperature, neutrons_emitted):
    return temperature < 800 and neutrons_emitted > 500 and temperature * neutrons_emitted < 500000


def reactor_efficiency(voltage, current, theoretical_max_power):
    levels = ((80, "green"), (60, "orange"), (30, "red"), (0, "black"))
    efficiency = 100 * (voltage * current) / theoretical_max_power
    for level, color in levels:
        if efficiency >= level:
            return color
            

def fail_safe(temperature, neutrons_produced_per_second, threshold):
    mix = temperature * neutrons_produced_per_second
    print(neutrons_produced_per_second)
    if mix <= threshold * .1:
        return 'LOW'
    if mix < threshold * .9:
        return 'NORMAL'
    return 'DANGER'
1 Like
5

So this is a variant on this test runner issue, and is solely related to how print() output is captured and reported for subtests. For 85% or so of cases, things are fine. Things become significantly less fine when there is more than one print() statement used, or print() is used in a loop.

There was a fix, but it is brittle, and doesn’t account for this particular creativity. When I have some time, I will take the code example and see if there is a quick change I can make to better parse the output that pytest-subtest sends to stdout in this case.

I am working on a better fix that’s rolled up into the tooling and Python upgrade, but its going to be a while since both the plugin and Pytest have had significant internal changes.

We are currently on Pytest 7.2.2, but the most recent version of Pytest that supports Python 3.13 is 8.3.

3 Likes