Per Adding CFML exercises - #10 by colinleach, it seems the CFML test runner is slow to boot, and I haven’t been able to successfully run any solutions online for weeks now. CFML is featured in week 47 of 48in24 so we should work on improving the test runner performance, turning off the test runner otherwise, or perhaps not featuring CFML. If we don’t try these things, students might get frustrated trying to submit CFML solutions through the online editor, and being so close to the end, I wouldn’t want that to overshadow the 48in24 experience. Thankfully, we’ve got plenty of time to workshop this.
1 Like
Hi all, I originally implemented the CFML track on Exorcism and I’m the lead developer of the CommandBox CLI you’re using and a maintainer of the TestBox testing framework. I haven’t really been involved in Exorcism since the earlier days when people would just run the tests locally. The spool-up time of the CLI is sadly mostly just the CF engine starting up. When I made the test runner back in the day, I would just leave CommandBox running (it has an interactive shell that emulates bash) and then running the tests inside of there would be super fast. Based on the context I gather from the linked thread above, it sounds like you run all the tests now in a Docker container somewhere on your own infrastructure. On a fast PC, CommandBox will start up in about 5-6 seconds, but on a limited environment (and by limited, I’m mostly referring to the CPU power), it can take 10-15 seconds. I have no idea what your timeouts are set to, but it really shouldn’t be taking longer than that.
I also helped add the CFML track to codewars a few years back, and they run tests in a Docker container as well. We did some more “Drastic” workarounds there where I customized CommandBox to skip most of its internal framework and an wrote a custom bootstrap script that just invoked testbox directly with a custom console output. I could help do something similar here. I have no idea what versions of stuff you’re still using, but the ortussolutions/commandbox images are designed to be used as a web server, and are a little “heavy”. I would recommend moving over to foundeo/minibox which uses a scaled down version of CommandBox and its underlying CF engine. While it doesn’t necessarily start faster, the image itself is around 100 megs which is less work for Docker if it’s having to pull an image. CommandBox is up to version 6.0 now and TestBox is at version 5.4.
Anyway, I have no idea where the current code is or how the Docker runners work, but I can try to help. I saw this because I still get notifications from the Github repo.
Thanks for those suggestions. I’m far from expert on the test runner, but as an old-time CF user I would be very happy to see this track running properly. it needs some love, from someone who understands the tooling.
The expert on this sort of thing is @ErikSchierboom. He may be away for the next few days but is normally back on Tuesday (Netherlands time).
I think that would be a good approach for us too.
We’re currently using ortussolutions/commandbox:amd64-jdk8-3.3.0 (see cfml-test-runner/Dockerfile at main · exercism/cfml-test-runner · GitHub) and then run the tests via box task run TestRunner 2>&1 (see cfml-test-runner/bin/run.sh at main · exercism/cfml-test-runner · GitHub).
Help would be greatly appreciated!
Erik linked to the current code, but in terms of how they work: for each test the docker container starts, runs /opt/test-runner/bin/run.sh, which is basically just box task run TestRunner and then destroys itself.