Exercism CLI not able to run PowerShell tests on Linux

Exercism Support CLI
,
1

I don’t seem to be able to run exercism test against my PowerShell exercises on Linux.

From Bash:

logavanc in 🌐 home in ~/Workspace/exercism.org/powershell/hello-world 
at Thu Sep  7 20:08:54 2023 ❯ exercism version
exercism version 3.2.0

logavanc in 🌐 home in ~/Workspace/exercism.org/powershell/hello-world 
at Thu Sep  7 20:08:57 2023 ❯ exercism test
Running tests via `Invoke-Pester`

2023/09/07 20:09:01 Failed to get error from failed subcommand: exec: "Invoke-Pester": executable file not found in $PATH

logavanc in 🌐 home in ~/Workspace/exercism.org/powershell/hello-world 
at Thu Sep  7 20:09:01 2023 ❯ pwsh -C Invoke-Pester

Starting discovery in 1 files.
Discovery found 1 tests in 271ms.
Running tests.
[+] /home/logavanc/Workspace/exercism.org/powershell/hello-world/HelloWorld.tests.ps1 654ms (119ms|321ms)
Tests completed in 680ms
Tests Passed: 1, Failed: 0, Skipped: 0 NotRun: 0

logavanc in 🌐 home in ~/Workspace/exercism.org/powershell/hello-world took 3s 
at Thu Sep  7 20:09:27 2023 ❯

And from PowerShell:

PS /home/logavanc/Workspace/exercism.org/powershell/hello-world> exercism version
exercism version 3.2.0

PS /home/logavanc/Workspace/exercism.org/powershell/hello-world> exercism test   
Running tests via `Invoke-Pester`

2023/09/07 20:11:28 Failed to get error from failed subcommand: exec: "Invoke-Pester": executable file not found in $PATH

PS /home/logavanc/Workspace/exercism.org/powershell/hello-world> Invoke-Pester

Starting discovery in 1 files.
Discovery found 1 tests in 7ms.
Running tests.
[+] /home/logavanc/Workspace/exercism.org/powershell/hello-world/HelloWorld.tests.ps1 29ms (4ms|20ms)
Tests completed in 31ms
Tests Passed: 1, Failed: 0, Skipped: 0 NotRun: 0
PS /home/logavanc/Workspace/exercism.org/powershell/hello-world>

Is there some setup step that I am missing, or might this be a bug in the Exercism CLI?

2

The Exercism CLI runs the Invoke-Pester command though the OS via the Golang’s exec.Cmd. This likely invokes a kernel execve() to run the Invoke-Pester executable. The OS searches the PATH directories to find an executable with that name.

What happens when you run Invoke-Pester directly in bash? Does that work or does that show the same error? Is Invoke-Pester not the correct command? Does it need to be invoked some other way in Linux? If so, the execism test command likely needs an update.

+cc @xavdid

3

Correct, Invoke-Pester is not something that I can run directly from Bash and isn’t something that can be found in my $PATH:

logavanc in 🌐 home in ~/Workspace/exercism.org/powershell/hello-world took 3s 
at Thu Sep  7 20:09:27 2023 ❯ Invoke-Pester
bash: Invoke-Pester: command not found...

But it is something that I can run directly in PowerSehll, which is why I was hoping that exercism test might work if I ran it from inside PowerShell, but it didn’t. =(

4

While Invoke-Pester may not be something which can be invoked from your shell, it appears that pwsh -C Invoke-Pester is something you can invoke. This may be a simple case of updating execism test to change the test command from Invoke-Pester to pwsh -C Invoke-Pester.

5

Thanks for flagging!

I copied the testing instructions from the help doc, which say:

You can run the tests from PowerShell by calling:

Invoke-Pester

This will run all .tests.ps1 files it can find in the directory.

If you are using another shell than PowerShell, you can run the tests by calling:

pwsh -Command "Invoke-Pester"

So it looks like they expect you to either:

  1. be running in powershell and have Pester installed and available on the path, or
  2. be running in bash with powershell (and presumably pester?) installed

exercism test currently assumes case 1, but we could potentially check what shell you’re in and tweak the command accordingly.

6

The latter command may work in all shells (powershell included) and be safe to just run always.

7

I am doing all of my Exercism work locally with VS Code and I have a task setup to run my tests that normally Just Works ™ because my task simply runs exercism test. Needing to have separate tasks based on the code is doable, but annoying.

I was able to add a little shim script on my local path that runs the pwsh -Command Invoke-Pester command and now the exercism test command works as expected:

logavanc in 🌐 home in ~/Workspace/exercism.org/powershell/hello-world 
at Thu Sep  7 22:13:07 2023 ❯ bat ~/.local/bin/Invoke-Pester 
───────┬──────────────────────────────────
       │ File: /home/logavanc/.local/bin/Invoke-Pester
───────┼──────────────────────────────────
   1   │ #!/usr/bin/env bash
   2   │ pwsh -Command 'Pester\Invoke-Pester'
───────┴──────────────────────────────────

logavanc in 🌐 home in ~/Workspace/exercism.org/powershell/hello-world 
at Thu Sep  7 22:13:35 2023 ❯ exercism test
Running tests via `Invoke-Pester`


Starting discovery in 1 files.
Discovery found 1 tests in 274ms.
Running tests.
[+] /home/logavanc/Workspace/exercism.org/powershell/hello-world/HelloWorld.tests.ps1 675ms (119ms|340ms)
Tests completed in 694ms
Tests Passed: 1, Failed: 0, Skipped: 0 NotRun: 0

logavanc in 🌐 home in ~/Workspace/exercism.org/powershell/hello-world took 3s 
at Thu Sep  7 22:13:51 2023 ❯

I’m not sure if there is a sexy solution to make the CLI work out of the box for everyone… maybe exposing config through environment variables for the CLI? Something like EXERCISM_POWERSHELL_PATH that if defined would cause the CLI to run the Invoke-Pester command inside the PowerShell defined in the variable?

In any case, thank you for all the help!

8

@Meatball Any thoughts? You’ve worked a lot with PowerShell lately

9

When being in the poweshell shell, you can use Invoke-Pester. If you are in another shell, you have to do pwsh -Command Invoke-Pester. I think I did document how to test, in the how to test on the powershell track doc

10

It won’t, because on Windows the command is named powershell not pwsh (but we can do Windows-specific commands).

What I meant was: would something like pwsh -Command "Invoke-Pester" be a valid way to run the tests? Would it work from a PowerShell shell itself?

11

It won’t, because on Windows the command is named powershell not pwsh (but we can do Windows-specific commands).

Running pwsh in the command prompt starts PowerShell on Windows.

What I meant was: would something like pwsh -Command "Invoke-Pester" be a valid way to run the tests? Would it work from a PowerShell shell itself?

Yes it would theoretically work, but the pwsh -Command "Invoke-Pester" is meant to be used if you are outside a PowerShell shell, But it does theoretically work, it is not the way I would recommend doing it. The “negative” thing is that you launch a new shell every time you run pwsh -Command "Invoke-Pester". It is the only way to work if you want to use a shell other than PowerShell. But if you are using Powershell so will Pester be cached if you just run Invoke-Pester, if you do pwsh -Command "Invoke-Pester" it will launch a new session and have to load in the library so rerunning tests take a lot longer since the testing library isn’t cached. On Windows, the performance loss that big, I think it is because on Windows so does it not need to be loaded, while on Linux (or specifically Ubuntu), I noticed seconds longer execution time when rerunning.

12

Note, there are different caches at different levels. If you have enough free memory, the Linux kernel does cache disk read data and if you invoke this multiple times, the disk read data may be cached. But, yes, you will incur additional start up costs. Those costs may be worth the benefits, though.

13

You can also cache the library yourself, that is what the Powershell test runner has done.

14

It doesn’t:

Microsoft Windows [Version 10.0.22621.2134]
(c) Microsoft Corporation. All rights reserved.

C:\Users\erik_>pwsh
'pwsh' is not recognized as an internal or external command,
operable program or batch file.

I think it only does that when you’ve installed powershell, the built-in powershell is named powershell

15

For what it is worth, if the expectation is that the exercism test CLI is meant to be used locally by devs, then what is a few seconds of overhead between friends? =)

Alternatively, if the exercism test command is supposed to just work for everyone out of the box (a pretty noble goal, and it seems like that is the goal based on check for, install or update of Pester by axtens · Pull Request #187 · exercism/powershell · GitHub), then maybe just brute force this?:

  • For the Windows build of the CLI, just run Invoke-Pester.
  • For the Linux build of the CLI, run pwsh -Command 'Pester\Invoke-Pester'. (That PR 187 above might need some updates too.)
16

For the Windows build of the CLI, just run Invoke-Pester.

This is not a per se Linux problem, this is a shell problem. If you just use PowerShell on Linux Invoke-Pester will work. And you can use other shells than PowerShell on Windows, it is just that it is the most common. The only difference os wised is that the pre-installed PowerShell is labeled differently than the user-installed Powershell

And Pester at least when I used it, was not cached when using it on Linux. On Windows using the command pwsh -Command 'Pester\Invoke-Pester added I think like 0,5 seconds. But on Linux for me, it added like 10 seconds. At least for me would I find that very annoying if it “normally” would take around 0,3 seconds. The ideal thing would be to have it work differently based on the shell being used but I guess otherwise the pwsh -Command 'Invoke-Pester is fine.

For the Linux build of the CLI, run pwsh -Command 'Invoke-Pester' . (That PR 187 above might need some updates too.

That pr references the CI which has nothing to do with the CLI. And since it is in a PowerShell environment it doesn’t have to open a new PowerShell environment.

The CLI can be found here: GitHub - exercism/cli: A Go based command line tool for exercism.org.

17

OK, what am I missing. I can’t run exercism test for my PowerShell solution on Windows either?

Currently on Windows 11, in PowerShell:

HomePC in ~\Workspace\exercism.org\powershell\hello-world 
at Fri Sep 15 09:00:00 2023 ❯ $PSVersionTable.PSVersion

Major  Minor  Patch  PreReleaseLabel BuildLabel
-----  -----  -----  --------------- ----------
7      3      6

I have Pester installed:

HomePC in ~\Workspace\exercism.org\powershell\hello-world
at Fri Sep 15 09:01:19 2023 ❯ Get-Command Invoke-Pester

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Function        Invoke-Pester                                      5.5.0      Pester

I can run Pester and get test results:

HomePC in ~\Workspace\exercism.org\powershell\hello-world
at Fri Sep 15 09:01:34 2023 ❯ Invoke-Pester

Starting discovery in 1 files.
Discovery found 1 tests in 126ms.
Running tests.
[+] C:\Users\logavanc\Workspace\exercism.org\powershell\hello-world\HelloWorld.tests.ps1 390ms (49ms|233ms)
Tests completed in 401ms
Tests Passed: 1, Failed: 0, Skipped: 0 NotRun: 0

But if I attempt to run the tests with exercism test, it doesn’t work:

HomePC in ~\Workspace\exercism.org\powershell\hello-world 
at Fri Sep 15 08:59:39 2023 ❯ exercism test
Running tests via `Invoke-Pester`

2023/09/15 09:00:00 Failed to get error from failed subcommand: exec: "Invoke-Pester": executable file not found in %PATH%

I’m sorry if this thread has lost its way, but so far, my PowerShell experience for local development has been a frustrating one. =/ Please let me know if this should be in its own thread or if this is something that I am just not understanding about how to get set up locally and doesn’t have to do with the Exercism CLI.

18

For now I think you can just forgo the “exercism test” step, if invoke-pester passes all the test, then just submit it.