Sanity check: Required to run "yarn install" and wait over three minutes before being able to run tests on each exercise (Typescript track)

chowie · February 15, 2023, 3:31pm

So the advice in the Typescript track docs under Installing Typescript locally states:

Each assignment needs some tools to run the tests:
They can be installed running this command within each assignment directory:
$ yarn install
You must run install inside the exercise directory before you are able to run the tests.

For me running yarn install inside the exercise directory:

takes around 229.42 seconds
installs 18400 packages
downloads 180 MB

For every, single, exercise.

Is this par for the course, or do I have a seriously misconfigured dev environment?

Any pointers would be appreciated!

bobahop · February 15, 2023, 5:28pm

I think @SleeplessByte has made a suggestion before about installing yarn at a higher level in the directory structure so it doesn’t have to be installed in each subdirectory. I should have paid better attention, because I don’t remember the details, but perhaps he can recount it if it’s still applicable.

SleeplessByte · February 15, 2023, 5:30pm

You are correct. That, however, never progressed further but it’s on the cli roadmap.

That said:

I THINK each exercise will work with pnp, so what you can do is:

yarn set version stable

And remove any line from .yarnrc.yml that says nodeLinker: node-modules.

This should then become pnp and thus only install each version of each package once at the disk.

chowie · February 16, 2023, 3:37am

Thank you @bobahop and @SleeplessByte for your prompt responses.

I’m going to included details on me following this advice in a following reply. After my attempt to follow the instructions I still ended up with a result where running yarn install takes over 100 seconds to verify the cache, and fetch required packages each time I begin a new exercise. Again, I set up this development environment last year with quite a lot of naivety, so it is very possible that the root cause is a poor structure and the remedy is to begin again with more intention.

Prior to receiving the advice yesterday I moved forward with converting the tests to use the deno test runner, and was able to run tests almost instantly whilst completely avoiding the necessary step of yarn-install.

I’ve already got a rudimentary script that converts the provided jest test suite for an exercise into a deno test suite. Do either of you think that using deno as the test runner for the Typescript track would be something the Exercism team would be interested in pursuing?

I’m sure there are more tradeoffs that I haven’t considered, but using this script and then deno test seems to be more efficient for me personally, at least in the few exercises I’ve looked at so far.

chowie · February 16, 2023, 3:39am

Below is the process and results of me following the advice provided by @SleeplessByte :

Summary

So I ran yarn set version stable in my Exercism/typescript directory:

Exercism/typescript [ yarn set version stable                                              master * ] 10:51 AM
➤ YN0000: Retrieving https://repo.yarnpkg.com/3.4.1/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in ../../../../../.yarn/releases/yarn-3.4.1.cjs
➤ YN0000: Done in 1s 62ms

However after doing so I couldn’t find any reference to a file named .yarnrc.yml, nor any matches when using VS Code to search for the string nodeLinker: node-modules

I continued by downloading a new exercise, navigating into the new directory and attempting to run yarn test. I received a usage error, and a question asking if my home directory was intended to be a project (it isn’t):

typescript/dnd-character [ yarn test                                                           master * ] 10:59 AM
Usage Error: The nearest package directory (/mnt/c/Users/Admini_T470s/Documents/Personal/Dev/Exercism/typescript/dnd-character) doesn't seem to be part of the project declared in /mnt/c/Users/Admini_T470s.

- If /mnt/c/Users/Admini_T470s isn't intended to be a project, remove any yarn.lock and/or package.json file there.
...

$ yarn run [--inspect] [--inspect-brk] [-T,--top-level] [-B,--binaries-only] <scriptName> ...

I moved forward by removing the empty yarn.lock and package.json that existed in my home directory.

My next attempt at yarn test gave me an Internal Error:

typescript/dnd-character [ yarn test                                                           master * ] 11:01 AM
Internal Error: @exercism/typescript-dnd-character@workspace:.: This package doesn't seem to be present in your lockfile; run "yarn install" to update the lockfile
    at P0.getCandidates (/mnt/c/Users/Admini_T470s/.yarn/releases/yarn-3.4.1.cjs:435:5145)
    at yf.getCandidates (/mnt/c/Users/Admini_T470s/.yarn/releases/yarn-3.4.1.cjs:391:1264)
    at /mnt/c/Users/Admini_T470s/.yarn/releases/yarn-3.4.1.cjs:439:7693
    at of (/mnt/c/Users/Admini_T470s/.yarn/releases/yarn-3.4.1.cjs:390:11070)
    at ge (/mnt/c/Users/Admini_T470s/.yarn/releases/yarn-3.4.1.cjs:439:7673)

I moved forward by running yarn install:

typescript/dnd-character [ yarn install                                                        master * ] 11:02 AM
➤ YN0000: ┌ Resolution step
➤ YN0032: │ fsevents@npm:2.3.2: Implicit dependencies on node-gyp are discouraged
➤ YN0061: │ @npmcli/move-file@npm:2.0.1 is deprecated: This functionality has been moved to @npmcli/fs
➤ YN0061: │ w3c-hr-time@npm:1.0.2 is deprecated: Use your platform's native performance.now() and performance.timeOrigin.
➤ YN0002: │ @exercism/typescript-dnd-character@workspace:. doesn't provide @babel/core (peae1d), requested by babel-jest
➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
➤ YN0000: └ Completed in 13s 274ms
➤ YN0000: ┌ Fetch step
➤ YN0013: │ yallist@npm:4.0.0 can't be found in the cache and will be fetched from the remote registry
...
➤ YN0000: └ Completed in 57s 456ms
➤ YN0000: ┌ Link step
➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental
➤ YN0007: │ core-js@npm:3.28.0 must be built because it never has been before or the last one failed
➤ YN0000: └ Completed in 41s 118ms
➤ YN0000: Done with warnings in 1m 53s

Now running yarn test works as expected. This is however the same process as before, albeit with a slightly lower wait time for yarn install , and a .yarn folder that is around 3 times smaller than the previous node_modules folder.

Repeating the process with a new exercise:

typescript/dnd-character [ exercism download --exercise=pangram --track=typescript             master * ] 11:08 AM

Downloaded to
C:\Users\Admini_T470s\Documents\Personal\Dev\Exercism\typescript\pangram
typescript/dnd-character [ cd ../pangram                                                       master * ] 11:10 AM
typescript/pangram [ yarn test                                                                 master * ] 11:10 AM
Internal Error: @exercism/typescript-pangram@workspace:.: This package doesn't seem to be present in your lockfile; run "yarn install" to update the lockfile
...

typescript/pangram [ yarn install                                                              master * ] 11:10 AM
➤ YN0000: ┌ Resolution step
➤ YN0032: │ fsevents@npm:2.3.2: Implicit dependencies on node-gyp are discouraged
➤ YN0061: │ @npmcli/move-file@npm:2.0.1 is deprecated: This functionality has been moved to @npmcli/fs
➤ YN0061: │ w3c-hr-time@npm:1.0.2 is deprecated: Use your platform's native performance.now() and performance.timeOrigin.
➤ YN0002: │ @exercism/typescript-pangram@workspace:. doesn't provide @babel/core (p732fb), requested by babel-jest
➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
➤ YN0000: └ Completed in 16s 345ms
➤ YN0000: ┌ Fetch step
➤ YN0013: │ yallist@npm:3.1.1 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yallist@npm:4.0.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yargs-parser@npm:20.2.9 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yargs@npm:16.2.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yocto-queue@npm:0.1.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0000: └ Completed in 14s 58ms
➤ YN0000: ┌ Link step
➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental
➤ YN0007: │ core-js@npm:3.28.0 must be built because it never has been before or the last one failed
➤ YN0000: └ Completed in 42s 718ms
➤ YN0000: Done with warnings in 1m 14s

We get slightly better, but similar results.

Conclusion: Something is probably still a little off here, and a better understanding of both yarn and how my Exercism/typescript directory is structured is required.

SleeplessByte · February 16, 2023, 8:16am

You didn’t get similar results.

This list time it only downloaded a handful of packages!

Subsequent installs will only get faster, but more importantly, smaller!

You do need to make sure that:

It’s using Yarn 3 in each directory (you are from the log)
It’s not using the nodeModules linker (in the rc file).

It will only download new packages if your version of the exercises uses a different version of the packages. You can alleviate this by updating each exercise to the latest version via the Web UI.

Hope it helps @chowie . Please ask more if you want more information.

SleeplessByte · February 18, 2023, 1:01am

Alright. I ran the numbers myself.

C:\Users\Derk-Jan\Exercism\typescript\hello-world>yarn set version stable
➤ YN0000: Retrieving https://repo.yarnpkg.com/3.4.1/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-3.4.1.cjs
➤ YN0000: Done in 0s 420ms

C:\Users\Derk-Jan\Exercism\typescript\hello-world>yarn
➤ YN0000: ┌ Resolution step
➤ YN0032: │ fsevents@npm:2.3.2: Implicit dependencies on node-gyp are discouraged
➤ YN0061: │ w3c-hr-time@npm:1.0.2 is deprecated: Use your platform's native performance.now() and performance.timeOrigin.
➤ YN0061: │ @npmcli/move-file@npm:2.0.1 is deprecated: This functionality has been moved to @npmcli/fs
➤ YN0000: └ Completed in 4s 663ms
➤ YN0000: ┌ Fetch step
➤ YN0013: │ y18n@npm:5.0.8 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yallist@npm:3.1.1 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yallist@npm:4.0.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yargs-parser@npm:20.2.9 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yargs@npm:16.2.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0000: └ Completed in 1s 390ms
➤ YN0000: ┌ Link step
➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental
➤ YN0007: │ core-js@npm:3.28.0 must be built because it never has been before or the last one failed
➤ YN0000: └ Completed in 3s 325ms
➤ YN0000: Done with warnings in 9s 440ms

C:\Users\Derk-Jan\Exercism\typescript\hello-world>

9s to completion.

The second exercise:

C:\Users\Derk-Jan\Exercism\typescript\two-fer>yarn set version stable
➤ YN0000: Retrieving https://repo.yarnpkg.com/3.4.1/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-3.4.1.cjs
➤ YN0000: Done in 0s 408ms

C:\Users\Derk-Jan\Exercism\typescript\two-fer>yarn 
➤ YN0000: ┌ Resolution step
➤ YN0032: │ fsevents@npm:2.3.2: Implicit dependencies on node-gyp are discouraged
➤ YN0061: │ w3c-hr-time@npm:1.0.2 is deprecated: Use your platform's native performance.now() and performance.timeOrigin.
➤ YN0061: │ @npmcli/move-file@npm:2.0.1 is deprecated: This functionality has been moved to @npmcli/fs
➤ YN0000: └ Completed in 4s 100ms
➤ YN0000: ┌ Fetch step
➤ YN0013: │ y18n@npm:5.0.8 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yallist@npm:3.1.1 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yallist@npm:4.0.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yargs-parser@npm:20.2.9 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yargs@npm:16.2.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0000: └ Completed in 0s 666ms
➤ YN0000: ┌ Link step
➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental
➤ YN0007: │ core-js@npm:3.28.0 must be built because it never has been before or the last one failed
➤ YN0000: └ Completed in 4s 228ms
➤ YN0000: Done with warnings in 9s 77ms

9s to completion.

In both cases, both have a local cache folder of about 40 MB.

Can we get rid of that folder? Why yes we can. The local folder allows for zero installs, but this is also available globally. Let’s first remove the cache in both folders:

yarn cache clean

Now add enableGlobalCache: true to the .yarnrc.yml and install.

C:\Users\Derk-Jan\Exercism\typescript\hello-world>yarn install
➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0013: │ y18n@npm:5.0.8 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yallist@npm:3.1.1 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yallist@npm:4.0.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yargs-parser@npm:20.2.9 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yargs@npm:16.2.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0000: └ Completed in 21s 975ms
➤ YN0000: ┌ Link step
➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental
➤ YN0000: └ Completed
➤ YN0000: Done with warnings in 22s 159ms

Oof. 9s went to 22s (which imo is fine), but this happened because we just killed the cache, which purges all files.

Let’s also delete two-fer/.yarn (manually), and delete .yarnrc.yml.

Now we start again:

yarn set version stable

Add enableGlobalCache: true,

yarn install

And the results:

$ yarn set version stable
➤ YN0000: Retrieving https://repo.yarnpkg.com/3.4.1/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-3.4.1.cjs
➤ YN0000: Done in 0s 445ms

$ yarn install
➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0000: └ Completed in 0s 307ms
➤ YN0000: ┌ Link step
➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental
➤ YN0007: │ core-js@npm:3.28.0 must be built because it never has been before or the last one failed
➤ YN0000: └ Completed in 2s 37ms
➤ YN0000: Done with warnings in 2s 494ms

Done in 2.5 seconds, with zero packages fetched.

The .yarn folder size is now ~6 MB.

I don’t know if this was your goal, but this is about as small as it’s going to get.

SleeplessByte · November 6, 2024, 5:41pm

This has been resolved by enabling PnP on the track.

iHiD · November 7, 2024, 4:29am

Nice work!