Mark tests to always run even when others are WIP

[gkingston]

I'd like the ability to mark certain tests to always run even if other tests are marked as WIP. I realize I can do a "once" at the start, but I'd rather not in this case.

I have a test for logging into the system that happens before any others and one for logging out at the end. When I have a test marked as WIP, those are skipped unless I also mark them as WIP. I'd just like some way to mark those two tests to always be run.

Right now I just have those tests marked as WIP as well. This is just a low priority feature request. There is a way to do a named test(&&&), wip(&&&&) and a skip(&&!), just would like a way to do an always test.

[lefthandedgoat]

Are you using wip so it goes slow and helps debug? And you want to have something like &&&&& that runs in both normal runs and wip runs, but does not run slow?

[gkingston]

Yes. It's not so much the speed, but making it so I can run just my WIP plus a few others when working with WIP. I have somewhere around 35-40 tests right now. I use WIP so it won't run all of them when I'm working on a particular (new in most cases) test. I just don't want to have to remember to go mark/unmark my login/logout tests as WIP each time when I'm switching between running just one test and running all the tests.

[lefthandedgoat]

Are all 35-40 tests all related? Like all for the same page or functionality?

I ask because its usually a good idea to break out tests into separate files if they are testing separate functionality. You would have file that was explicitly testing login and log out functionality, that bad passwords were rejected, and that log out redirected you to the login page etc.

Then for other tests that you need to be logged in to do, you just make a common login function and call it with your once. I make my login function smart enough to be aware if I am already logged in, and if I am do nothing. Then all your tests can have a consistent once and the actual &&& tests you are working on can be all turned to &&&& for debugging.

Tonight I will improve the example to be more thorough and reflect how I do stuff now which lends nicely to 'tests under dev' and 'all done tests' which may suite your development workflow.

If that doesn't help with your needs then we can come up with something that will or implement the feature you are asking for.

[gkingston]

I have my tests split in multiple files already. I am writing these tests against a long in the tooth web application that has many "modules". I'm creating a separate fs file for each module and then calling those from the Program.fs. I definitely could do a better job utilizing something like once, but since there was already some skip functionality for tests, I thought it may be nice to have some "always" functionality as well.

This is really just a feature request/wishlist item for me. No big rush and it's not stopping me from getting things done.

[lefthandedgoat]

Now available
https://www.nuget.org/packages/canopy/0.9.23

Use &&&&& and it will run under non wip mode and in wip mode, the tests run slow in wip mode as thats an always on/off thing, not a per test thing.

If you prefer some other infix operator you can alias &&&&& like this:

let ( &!&! ) desc f = desc &&&&& f

Enjoy!

[gkingston]

That was quick, thanks. I think there is an issue with it. If I have a Program.fs with calls in it to various test files and if the files only have a &&&&& in it(no &&&& tests), those tests will get skipped. If the files with tests with &&&&& also have at least one test with &&&&(wip), then both those style tests will run from that file.

I can create a demo project for this if you need me to.

[lefthandedgoat]

Ok I will look into it

[lefthandedgoat]

This works for me, and I tried different combinations

let all () =
  context "&&&&& stuff"

  "1" &&&&& fun _ ->
    printfn "1"
    1 === 1

  "2" &&&&& fun _ ->
    printfn "2"
    1 === 1

  "3" &&&&& fun _ ->
    printfn "3"

  "4" &&&&& fun _ ->
    printfn "4"

  "5" &&&&& fun _ ->
    printfn "5"

  "6" &&&&& fun _ ->
    printfn "6"

  "7" &&&&& fun _ ->
    printfn "7"

[lefthandedgoat]

Wait I think I know what it is. You have 2 files, one full of &&&&& always and another with a mix of stuff including &&&& wip right?

And only the file with wip is running?

If so , its because of this https://github.com/lefthandedgoat/canopy/blob/master/src/canopy/runner.fs#L119

Sorry I was only thinking in the context of a single file.

[gkingston]

Yes, that is correct. If one file has no wip items, it is just skipped altogether, but ones with both always and wip run both types of tests.

[lefthandedgoat]

Ok, sorry about that. I will have to get you another build tonight when I get home.

[lefthandedgoat]

This should fix it. https://www.nuget.org/packages/canopy/0.9.24