Add mention to the use of Web Worker API on web test runner to Parallel Letter Frequency text

Programming JavaScript
1

Hi, people. I’m iterating over my previous solution for the Parallel Letter Frequency exercise on JS, experimenting with new things. I noticed that currently the description for the exercise has the following text, which seems to indicate that Exercism only has support to node.js Worker threads:

Parallelism in Javascript

Even though Javascript by default is single-threaded, there is a way to execute code in parallel fashion.

If your running javascript in the browser (e.g. in a web app), then the way to achieve parallelism is through the Web Worker API. As described by MDN:

Web Workers makes it possible to run a script operation in a background thread separate from the main execution thread of an application.

On the other hand, if your javascript is running in Node.js, which is Exercism’s target runtime, this same concept is known as Worker threads.

However, the web test runner doesn’t seem to support Worker threads, but the Web Worker API. And local solutions, on the opposite and according to the description for the exercise, only support node.js Worker threads.

So I’m thinking that maybe the description for the exercise should be updated to mention this difference?

Also, on a slightly unrelated note, web based solutions don’t seem to support SharedArrayBuffer. Did some research and it’s possibly because of this security requirement involving some Cross-origin headers.

Not sure if this should be mentioned on the description as well or if it’s overkill.

1 Like
2

If I understood correctly how the online editor works with the web test runner, one has to support both APIs to succesfully submit a solution using the online editor. The Web Worker API to get it to work in the browser and the Worker threads to pass the final submission tests running in the Exercism infrastructure. Sounds like a nightmare to debug.

@SleeplessByte Does the submission process work that way? Should we discourage actual Parallelism?

2 Likes
3

Nah.

We should probably find a way for someone to write the node code and transpiles it so it works in the browser.

This is likely possible because the interface required to complete the exercise is nearly 1-to-1 what you’d need to write for webworkers. I’ll have to play with it.

There is no CSP other than report-only on the website so that should affect anything. As for cross-origin isolation: yeah I’ll check that out. Could be an issue. I’ll find out what’s up with those shared array buffers

Sidenote: I was actually involved turning the exercise into one with parallelism and I think it’s important to try to keep as there are no race conditions in the classical sense or multi threading bugs with JavaScript because of the single execution context and thread message passing.

2 Likes
4

@oxe-b I wrote up the instructions for this exercise before the web test-runner was a thing so it made sense at the time. ¯_(ツ)_/¯

The way I see it, maybe just removing the bit about Exercism's target runtime may be better, but I’m open for suggestions.

@mk-mxp The tests for this exercise don’t enforce actual parallelism in any way so you can pass with a concurrent solution that just returns a promise (and most people do so), so it wouldn’t be a high bar. I never found a way to enforce test for actual parallelism that didn’t feel clunky. Otherwize, I vote to keep this exercise as it highlights one of the unique aspects of JS nicely.

2 Likes
5

Just for clarification:

  • I was fighting for actual parallelism becoming possible in this exercise and wrote the demonstrating solution that it was not possible back then but could be without enforcing parallelism
  • I do not want anyone to test for students using parallelism and never did
  • I am very much for keeping parallelism in this exercise
  • I am also for describing precisely what can work to add parallelism to ones solution (which is hard now)
  • Which made me ask: Should we discourage it? Because I don’t not see a student friendly way to add parallelism now
  • I am very much in favor of adding transpiling, be it web worker to worker threads or the other way around
  • I have a slight tendendcy to keep the current node worker thread based solutions working while I also understand that writing node worker thread code in the browser feels very weird
2 Likes
6

Another possible way to improve the situation for now: We could add a note, that implementations using actual parallelism must be submitted from command line.

2 Likes
7

@mk-mxp yes let’s do that last one so I can have some time implementing the fix. I also need @dem4ron for that to do a deploy which may take time after I finish.

Want to do the PR?

<!-- prettier-ignore -->
~~~@exercism/caution
...
~~~

is probably appropriate

8

Well, I don’t know all the magic inside the tests, but I wrote many iterations for the exercise and the ones usin Web Workers pass on the web test runner and I’ve also submitted those solutions successfully. So I believe that as of now the web test runner has full support to Web Workers (as long as they don’t use Shared Array Buffers).

For reference, this was the last one using Web Workers:

const workerCode = `
  self.addEventListener("message", (event) => {
    const counter = event.data.reduce((acc, letter) => {
      const key = letter.toLowerCase();
      acc[key] = (acc[key] ?? 0) + 1; 
      return acc;
    }, {});

    self.postMessage(counter);
  });
`;

const blob = new Blob([workerCode], { type: "application/javascript" });

export const parallelLetterFrequency = async (texts) => {
  const numCores = navigator.hardwareConcurrency || 1;
  const workers = Array.from({ length: numCores }, (_, idx) => {
    const worker = {
      worker: new Worker(URL.createObjectURL(blob)),
      active: false,
    };
    return worker;
  });

  const results = texts.map((text) => {
    return new Promise((resolve) => {
      const crtWorker = workers.find(({ active }) => !active);
      if (!crtWorker) {
        const result = (text.match(/\p{L}/gu) ?? []).reduce((acc, letter) => {
          const key = letter.toLowerCase();
          acc[key] = (acc[key] ?? 0) + 1;
          return acc;
        }, {});
        resolve(result);
      } else {
        crtWorker.active = true;
        crtWorker.worker.addEventListener("message", (event) => {
          resolve(event.data);
          crtWorker.active = false;
        });
        crtWorker.worker.postMessage(text.match(/\p{L}/gu) ?? []);
      }
    });
  });

  const counters = await Promise.all(results);
  return counters.reduce((acc, counter) => {
    for (const [key, val] of Object.entries(counter))
      acc[key] = (acc[key] ?? 0) + val;
    return acc;
  }, {});
};
9

This one only uses Workers to do the work, so it’s more clear:

const workerCode = `
  self.addEventListener("message", (event) => {
    const counter = event.data.reduce((acc, letter) => {
      const key = letter.toLowerCase();
      acc[key] = (acc[key] ?? 0) + 1; 
      return acc;
    }, {});

    self.postMessage(counter);
  });
`;

const blob = new Blob([workerCode], { type: "application/javascript" });

export const parallelLetterFrequency = (texts) => {
  const promises = texts.map((text, idx) => {
    const letters = text.match(/\p{L}/gu) ?? [];
    return new Promise((resolve) => {
      const workerURL = URL.createObjectURL(blob);
      const worker = new Worker(workerURL);
      worker.postMessage(letters);
      worker.addEventListener("message", (event) => {
        resolve(event.data);
        worker.terminate();
        URL.revokeObjectURL(worker);
      });
    });
  });

  return Promise.all(promises).then((counters) => {
    return counters.reduce((acc, counter) => {
      for (const [key, val] of Object.entries(counter))
        acc[key] = (acc[key] ?? 0) + val;
      return acc;
    }, {});
  });
};
10
3 Likes