As we had a problem yesterday with PHP on MacOS in the GitHub workflow action, I turned off CI testing with MacOS as a short term solution to the problem. But that made me think about: “Why do we run the CI tests on MacOS?” And the same question I have for Windows in CI testing.
I would prefer to stop running the tests on MacOS and Windows completely. Because PHP has become super independent of the underlying OS except for those well documented cases of direct OS proxying. And I believe, but haven’t proven yet, that Exercism doesn’t and will not teach or use those.
I open this thread to collect thoughts and input about that. Are there points I don’t see, why running CI tests on MacOS and Windows are useful or required for the PHP track?
I open this thread to collect thoughts and input about that. Are there points I don’t see, why running CI tests on MacOS and Windows are useful or required for the PHP track?
I think we run it on Windows and MacOS only to guarantee that students which uses a local exercism cli will be able to solve the exercises on any platform.
Can we guarantee, that tests are working for students on their systems? In my eyes, this is impossible. The most likely issue, why students cannot run the PHP exercise tests, is a problem with the installation on their mache (which is out of CI testability). As the latest issue showed, even an auto-update of other packages can destroy PHP usability.
I think, that our PHP usage and our tests are 100% portable code. That is tested for by PHP itself (and documented for the few exceptions). So we do not need to repeat those tests.
I’m a little split here, I see it from both sides.
Which CI is the Online Editor working with? I expect this to be a Linux host/runner too?
We should at least ensure that our CI can pass what ever we are using in the Online Editor.
My immediate reaction is why we would reduce our test coverage? If they’ve been tested on those OSs for years successfully, what is changing that means it’s harder for us to run them on those now? It doesn’t cost us more to do and if there is ever a weird edge-case, having the CI helps guard. I ask that as a question though, not as a way of expressing a formal opinion.
Thanks everyone for the valuable input! I would love to hear from someone, that has seen more of the exercises on the PHP track than me, if there are any possibly OS-specific parts of PHP used.
@iHiD “Never change a running system” has a cost, too. Running the tests on more OSes → more problems out of our control cost maintainers time. I spent an hour searching for causes / solutions to a problem I could not solve. I unsucessfully tried a handful of options to make it run again. Which is a weird edge-case outcome to me, as I fix failed Linux CI pipelines for money.
The speed of changes outside our visibility has massively increased. This “shivamatur” mentioned, that such a problem happens every few weeks, when Apple auto-updates some random, required libraries. The CI images we use are updated every week, but that doesn’t seem to make them stable for at least a week anymore - there was no change to the underlying image that caused the problems. Thanks for auto-updates at runtime!
I’d prefer to not lose the OS coverage and pin an older version of the PHP setup image if the latest is a problem.
I think that while PHP has made strides to reduce the OS differences, they still exist, and if we can do a small thing to ensure that the exercises work for students using Windows/mac it would be fine.
If the intent is to just validate the exercise solve for our test-runner, then I think it makes sense to only test linux images.
OK, I’m very much alone with what I wanted. I’ll get that up and running again then.
@ErikSchierboom@iHiD Do you know, why macos-latest is still macos-12 while macos-14 is available? Is 12 an LTS version? Or is that not a thing with MacOS? I have plenty of experience with Linux versioning schemes, but not with MacOS. Maybe it is GitHub not wanting to update MacOS runners…
Just to let you know: MacOS again fails to install any version of PHP, with no change to the underlying docker image nor the GitHub action used. I will stop using it for testing.
