Editor doesn't use latest version of test module

simisc · August 1, 2023, 11:01am

Continuing the discussion from Sieve exercise: Why does my solution cause errors?:

I’ve made a few changes to how the test module compares and prints arrays in recent PRs. These changes work locally (when re-downloading the exercise via CLI), but they don’t seem to have affected tests run in the editor.

github.com/exercism/fortran

Use latest version of test module in editor

opened 10:42AM - 01 Aug 23 UTC

simisc

This was previously a comment on #227, but I'm moving it to a new issue because …I think it's specific to how the test module is made available when testing in the editor. Changes made to the test module in #226 and #227 are not applied when testing in the editor. (In case relevant: I only tried it on exercises that I had already started before the test module was changed.) For Sieve, there's still the `An error occurred while running your tests. This might mean...` message (when testing an incorrect solution) — this is the issue that #227 was intended to solve. For Saddle Points (with correct and incorrect solutions), the message is: ``` We received the following error when we ran your code: /tmp/saddle-points/saddle_points_test.f::: | character(MAX_RESULT_STRING_LEN) :: s | Error: GNU Extension: Symbol max_result_string_len is used before it is typed at () /tmp/saddle-points/saddle_points_test.f::: | function pa_to_s(p) result(s) | Error: Function result s at () has no IMPLICIT type ``` This suggests that the editor is using the updated version of `saddle_points_test.f90`, but not the latest `TesterMain.f90`. I can reproduce the same error when working locally by using the old `TesterMain.f90` with the new `saddle_points_test.f90`. When testing an incorrect solution to High Scores in the editor, failed tests are double-counted, so the same issue applies to #226 (i.e., the solution is being tested by the pre-#226 version of the test module). _Originally posted by @simisc in https://github.com/exercism/fortran/issues/227#issuecomment-1659794587_

simisc · August 4, 2023, 1:42pm

I’ve been trying to learn a bit about how the test runner works to understand this issue… The run.sh script downloads the test module (testlib/TesterMain.f90) from the track repo if /opt/test-runner/testlib/ doesn’t already exist. But this directory is created (and the test module is downloaded to it) when the Docker image is built, so the test module is not downloaded again for each test run.

Is the issue here that deployment of the Docker image to AWS ECR is only triggered when there is a push to the fortran-test-runner repo? The most recent commit was on 5 July, which was before PRs #226 and #227 were merged in the track repo. Would that explain why tests in the editor are not using the current version of TesterMain.f90?

ErikSchierboom · August 4, 2023, 2:23pm

That is all entirely correct.

simisc · August 4, 2023, 5:57pm

Thanks Erik. Can you manually run the deploy workflow?

ErikSchierboom · August 9, 2023, 11:57am

I could, but that wouldn’t help as it would just re-use the Docker cache. The best way around this would be to introduce a file in the test runner that could be changed in order to bust the cache.

simisc · August 12, 2023, 12:04pm

Like this?