Solutions pass locally but fail when submitted to exercism

Hi, I’ve noticed that sometimes when I submit a solution that passes all the tests on my desktop, it fails here on exercism. Then I go to the test runner, recover the last iteration (the failing one), run the tests online and everything passes as they should, with the same code.

It’s happened a few times the last week, with different exercises, but not always. Recently it happened with 3 or 4 different iterations for the Forth exercise. I haven’t been able to find a pattern for this behaviour, something that might be triggering it.

Anyways, just letting you know.

It would be helpful to see the specific code and exact error message.

There’s no error message, the solution just fails when submitted from the desktop. There’s a processing time and then it fails. After it happens, I recover the same code (or copy/paste it from VSCode), run the tests online (instead of submitting using the CLI) and it passes everything as it had already passed locally.

The last time it happened was with this code. This was my 14th iteration for the Forth exercise, which failed. Then the 15th iteration, with the exact same code, passed. The same occurred with the 11th, 5th and 3rd iterations for the same exercise and also with other exercises the last week or so.

export class Forth {
  #stack = [];
  #ops = [
    [/^\s?dup\s?/i, () => this.#unary((x) => [x, x])],
    [/^\s?over\s?/i, () => this.#binary((x, y) => [y, x, y])],
    [/^\s?drop\s?/i, () => this.#unary(() => [])],
    [/^\s?swap\s?/i, () => this.#binary((x, y) => [x, y])],
    [/^\s?-?\d+\s?/i, (x) => Number(x)],
    [/^\s?\+\s?/i, () => this.#binary((x, y) => y + x)],
    [/^\s?-\s?/i, () => this.#binary((x, y) => y - x)],
    [/^\s?\*\s?/i, () => this.#binary((x, y) => y * x)],
    [
      /^\s?\/\s?/i,
      () => {
        return this.#binary((x, y) => {
          if (x === 0) throw new Error("Division by zero");
          return (y / x) | 0;
        });
      },
    ],
    [
      /\s?:\s(.*?)\s(.*)\s;\s?/i,
      (expression) => {
        const { key, ops } = this.#processUserOps(expression);
        const idx = this.#ops.findIndex((crt) => crt.source === key.source);
        if (idx !== -1) this.#ops.splice(idx, 1, [key, () => ops]);
        else this.#ops.unshift([key, () => ops]);
        return [];
      },
    ],
  ];

  #unary = (callback) => (x) => {
    if (x === undefined) throw new Error("Stack empty");
    return callback(x);
  };

  #binary = (callback) => {
    return this.#unary((y) => (z) => {
      if (z === undefined) throw new Error("Only one value on the stack");
      return callback(y, z);
    });
  };

  #consumeOperation = (expression) => {
    for (const [pattern, callback] of this.#ops) {
      const match = expression.match(pattern);
      if (match) return { pattern: match[0], callback: callback };
    }
    throw new Error("Unknown command");
  };

  #processUserOps = (pattern) => {
    const groups = pattern.match(/:\s(.*?)\s(.*)\s;\s?/i);
    if (/^\s*-?\d+\s*$/.test(groups[1])) throw new Error("Invalid definition");
    const key = new RegExp(`^\\s?${RegExp.escape(groups[1])}`, "i");
    const ops = groups[2].split(" ").map((opName) => {
      return this.#consumeOperation(opName).callback(opName);
    });
    return { key: key, ops: ops };
  };

  evaluate(expressions) {
    this.#stack = expressions.split(/(:.*;|\s)/).reduce((acc, op) => {
      if (!op || op === " ") return acc;
      const { pattern, callback } = this.#consumeOperation(op);
      return [callback(pattern)].flat(Infinity).reduce((res, fn) => {
        while (typeof fn === "function") fn = fn(res.pop());
        return [...res, fn].flat(Infinity);
      }, acc);
    }, this.#stack);
  }

  get stack() {
    return this.#stack;
  }
}

When you submit using the CLI, it runs a docker image based on the javascript-test-runner. This infrastructure may fail based on what we currently support. Your solution + exercise config can be PRed to the test folder, so we can find out.

When you submit using the browser, it runs a newly crafted in-browser test runner which will pass or fail based on what your browser supports. It working then but not using CLI is therefore not completely unexpected.

Do you want to PR your solution to the test-runner so we can find out what exactly isn’t supported? My guess would be the private properties.

1 Like

Of course, I just don’t know how to do that :sweat_smile:

Can you point me to some guide?

Yes.

The straightforward way would be to fork the repository first. Then you will work towards proposing a change to the repository.

  1. Create a new branch
  2. Make the following changes:
  3. If everything worked as expected (I will assume step 2.5 fails as it won’t say pass), then you can create a pull request from your branch (on your fork) to the repository’s main branch, and tag me @sleeplessbyte so I can re-open it.

If you get stuck anywhere and want to stop trying and get help, you can use the current topic on the forum to ask for help.

2 Likes

Thank you! I’ll try and get back.

1 Like

Hi! I finally managed to reach step 2.5. The run.sh wasn’t working here for some reason, it was generating all files except results.json even though the message in the terminal kept telling me the file was generated.

In the end I opened run.sh in the text editor and, as all previous steps seemed to work, I just ran the corepack pnpm jest command, with all flags and file names, directly in the terminal and it solved the issue.

I now have my branch with the ./test/fixtures/forth/oxe-b folder. Inside of it, there’re the files forth.js, forth.spec.js, expected_results.json and a folder .exercism with the files config.json and metadata.json.

I wasn’t sure if I should copy the lines from smoke.test.mjs because it seems an assert and my solution isn’t passing. So decided to check with you what I should do next.

Gotta tell me why your solution isn’t passing so we can potentially change the test runner to make it passing!

You can open the PR and we can discuss there.

1 Like

Forth test by oxe-i · Pull Request #950 · exercism/javascript-test-runner · GitHub got your solution to pass!

1 Like

My first PR, I’m very happy! :smile:

4 Likes

Yesss you got some reputation out of it!

1 Like