I noticed @iHiD’s PR the other day that introduces a client-side runner. I think this is overall a neat idea, but copying this as-is for tracks that require non-trivial tooling (e.g., compilers) could be problematic. I imagine the artifacts users would have to download into their browsers could become quite big.
Anyway, I was wondering if there was some discussion around this that I missed? If so, it would be great if someone could point me at it.
Also, is any of this already live? I had a brief look at the jq track. Test test of my hello-world exercise seemed to be happening on the server still.
I noticed @iHiD’s PR the other day that introduces a client-side runner. I think this is overall a neat idea, but copying this as-is for tracks that require non-trivial tooling (e.g., compilers) could be problematic.
It’s a non-trivial project that iHiD and Michael (of https://code.onweb.dev/)have been working on a fair bit. It largely doesn’t involve anyone else other than those two. Track maintainers are getting involved to help with the rollout. As this rolls out to more tracks, the process will hopefully become smoother and need less track maintainer involvement.
I imagine the artifacts users would have to download into their browsers could become quite big.
That’s something iHiD and Michael are explicitly keeping an eye on. The jq runner is surprisingly small, though! (Though I don’t recall the exact size.) They’re targeting C/C++ next. They are aware that size is important. On the other hand, having the client download the WASM runner means Exercism isn’t paying to store the test runner images (which are much larger) and Exercism doesn’t need to pay the compute/CPU cost to run the tests. If I recall correctly, those two account for two of the three largest costs of Exercism. Eliminating those costs would make Exercism substantially more sustainable.
Anyway, I was wondering if there was some discussion around this that I missed? If so, it would be great if someone could point me at it.
As mentioned, this is a collaboration between iHiD and Michael. There hasn’t been any public discussions.
Also, is any of this already live? I had a brief look at the jq track. Test test of my hello-world exercise seemed to be happening on the server still.
Where do you see that happening?
The user experience should be similar, but it’s actually running in your browser! If you watch carefully, when you load the editor, it loads boot.json, test-runner.tar and sysroot.tar which are the WASM runner files! When you run the tests, kernel.js is used to actually run things. No remote AWS VMs running Docker are needed.
I looked a bit closer now. It turns out it is much more lightweight than I thought. I expected this to be Linux-in-wasm-qemu, where after qemu everything would be native (as in unmodified x86 binaries). The linked PR looked like it was extracting binaries out of the original Docker image. But that is not the case, it’s only about the runner script. Linux is not involved, this is all native wasm.
That makes sense and is probably the biggest motivation here.
As I said, I was under the impression that it basically takes the contents of the Docker image and runs that in the browser. I know that some images are huge, that’s what got me worried.
Running everything client-side increases the hardware requirements, though. Hardware that is just good enough to do basic browsing would not suffice anymore. But I guess that’s acceptable if otherwise Exercism wouldn’t be sustainable.
Another thing that comes to mind is the CLI. How are submissions via CLI checked for correctness? Is for those cases the Docker version kept around?
I think it would make sense to have that discussion rather sooner than later if track maintainers are to be involved.
You’re right, I was just inferring from the fact that it took a few seconds when running jq locally takes a few milliseconds. I guess there’s still quite a gap between true native and wasm, hence the increased hardware requirements.
Apart from the missing discussion, that is the biggest issue with this effort for me. On the front page it says “Exercism is open source”. If part of the tooling is not available, that claim becomes at least partly false. Also, I wonder how track maintainers are supposed to update a track’s toolchain in that process? My understanding is that Michael’s tooling is required to create wasm binaries. That means only few will be able to upgrade Python to a newer version or upgrade a C++ compiler. Maybe that’s not a big issue in practice, but it for sure goes against Exercism’s spirit, at least for me.
Most of the exercises have pretty light requirements to run. The tests tend to test small problem spaces.
I think so. But I believe those submissions are far fewer than the number of test runs not associated with a submit.
Maintainers aren’t expected to have much involvement with this.
It’s already partially false IIRC one or two tracks use closed source test runners already. But that’s a good point!
I forget the details but track maintainers are not expected to do anything differently. Michael will handle kernel upgrades. The GHA rebuilds the WASM system image automatically somehow when test runner changes are merged.
Not increasing maintainer effort was an explicit consideration/requirement that was discussed as part of the work here. This runner is still somewhat exploratory. If it doesn’t scale or requires maintainers to do something different, there will be a discussion at that point.
Surprisingly this is all quite light-weight. We are using a wasm-based kernel (no emulation or anything like that) that runs wasm binaries compiled for the same platform. Those binaries can get quite heavy, but with optimisation and strategic deployment, even LLVM is just 15 MB over the wire (precompressed with brotli). Code On Web is a live example of all this if you want to poke around with the devtools.
This is true, although there is ongoing work in this area and so far even the heavier workloads like C++ compilation should be good to run on a 4 GB Chromebook. Extra care needs to be taken under Safari too since it places considerably more restrictive limits than Firefox and Chromium based browsers.
This is still a work in progress and perhaps an argument for keeping the Docker containers around since that is the supported open-source route, while the client-side runners are an closed source extension that makes Exercism more sustainable. I have an NDA that I still need to proofread a couple times before sending it over to @iHiD. That will allow upgrades and continued rollout and deployment of client side runners even if I disappear.
Feel free to tag me on Discord if you have any more questions!
IIRC one or two tracks use closed source test runners already. But that’s a good point!