No way to "skip" a test stating a reason

[knocte]

I'm loving canopy. That being said, I'm not a fan of "magic". And by magic I refer to these magic operators: https://lefthandedgoat.github.io/canopy/testing.html

To argue my point, let's take the example of someone wanting to ignore/skip a test for some reason. There's the &&! operator, great... but how to attach that reason to the code? In NUnit, you have the Ignore(string) attribute that accepts a way to encode the reason as a parameter.

If I were NUnit's author, I would have gone even farther and not provide the parameterless Ignore() overload. I think canopy should do this as well.

[lefthandedgoat]

What do you think the best way to represent it is? How it should be show in the console is a good start.

[knocte]

How it should be show in the console is a good start.

In the console it already shows in yellow, AFAIR, which is good. But it should state the reason of why the test is skipped.

[lefthandedgoat]

The short answer is that there isn't a reasonable way for me to add a new argument w/o hurting backwards compatibility etc, and I can explain why, and I can also give you a reasonable work around that you can use.

functions like &&! are called infix operators. They are functions just like:

let add x y = x + y

Except that they called between the arguments arg1 func arg2, instead of func arg1 arg2.

Because of this they can only take 2 arguments. Adding a 3rd argument is not possible. What we can do is charge the signature to from

string &&! func
to
string * string &&! func (a tuple of two strings and a function)

If I did this in canopy as a whole it would break everyone who had skips now. But I can help you make it work for you.

Make a new file called something like canopyExtensions.fs and add an open canopyExtensions statement to your tests, directly below your open canopy.

Add this function:

open canopy

let ( &&! ) (description, skipReason) f =
  let desc = sprintf "%s/r/n%s" description skipReason
  desc &&! f

This will let you write a test like this

"test #1", "feature not finished" &&! fun _ ->
  url "http://www.google.com/"
  //etc etc

Now any time you skip you will have to provide a reason to make the compiler happy.

Does this help?

[danuma]

This will help , especially if we are sharing the results with wide audience . Thank you Chris.

[knocte]

The short answer is that there isn't a reasonable way for me to add a new argument w/o hurting backwards compatibility

That is completely understandable. But you could always create a new operator that accepts an argument, and make the previous one obsolete?

But I can help you make it work for you.

Thanks so much for the suggested workaround.

However, after thinking a bit more about it, I would say that I wouldn't like the "new operator" solution either (the one I just suggested above), because of one reason: it's meant to make the test be skipped at "compilation time", let's say.

But, if we look again at NUnit features, they even have this feature at run-time, thanks to Assert.Ignore(). If you implemented this feature in canopy, then you wouldn't even need to deprecate the current &&! operator. That is, you would have a skip-operator at compile-time (that doesn't accept an argument) and a skip function at run-time (which forces you to send a "reason" argument). If someone needed a compile-time way to skip a test with a reason, they could just use the runtime one at the beginning of the test (making it unconditional) and be done.

[lefthandedgoat]

You want something like this?

"a normal test" &&& fun _ ->
  ignore "don't run right now"
  url "http://www.google.com/"
  click "search"

And it will mark the test as skipped?

[knocte]

Yes, so that you could also call it conditionally, for example

"a special test" &&& fun _ ->
  if (Platform.IsMac()) then
    ignore "known issue: this doesn't work in Safari"
  url "http://www.google.com/"
  click "search"

Feel free to use ignore or skip, I don't mind...

[lefthandedgoat]

You can download the latest version of canopy here: https://www.nuget.org/packages/canopy/0.9.58

just add

skip "some message" (its canopy.core.skip, there is also a canopy.runner.skip which has a different signature so you have to make sure you have your open statements correct, or fully qualify it)

inside the function body of your test and it will skip that test at run time.

[knocte]

Awesome thanks!