Skip tests for specific context if precondition functions failed

[mkulava]

Hi,

Is there any way to not execute tests in context if functions "once", "before" has failed for some reason?

[lefthandedgoat]

I can add a configuration flag to enable that I believe.

Let me rephrase so that I make sure I understand what you want.

If at any point during running tests for a context, if the once or before functions fails, skip all remaining tests for that context.

Sound right?

[mkulava]

Yes, that's exactly what we need. This will reduce time spent on failure investigation, since in most cases test failures are caused by incomplete execution of "before" and "once".

"Configuration flag" sounds great. Please let me know when to expect this feature. Thanks in advance!

[lefthandedgoat]

Looking at the code a failure in once should already fail the rest of the Suite. I will test that this works.

There is also a configuration flag called failFast which if you set it it to true it will stop running tests after any test fails (including the before). It will stop running all tests though, and not just tests for a specific context.

In its current state, the runner is a total mess and I need to refactor it. I will take the time to do that today hopefully and get your feature in too. It may take longer than I want and spill over a few days though.

[lefthandedgoat]

You can download this now!

https://www.nuget.org/packages/canopy/0.9.46

Set this configuration to true

skipRemainingTestsInContextOnFailure <- true

Then any failure in once or before will skip the rest of the tests for that context.

[danuma]

Im trying to add this feature to my tests, if the test fails in once block skip the rest of the tests, I am having a hard time changing the value of skipRemainingTestsInContextOnFailure to true. Pls let me know if any changes are needed to the following code.

let all _ =
context "Executing Tests"
once (fun _ ->
start firefox
url "application.com"
let currentTitle = title()

if (currentTitle <> expectedTitle) then

// added failwith and tried
skipRemainingTestsInContextOnFailure <- true
//tried the full name canopy.configuration.skipRemainingTestsInContextOnFailure <- true
)
LoginTests()
ShoppingTest()

[lefthandedgoat]

Its not compiling or its not working?

If its not working it must be because your currentTitle does equal your expected title.

[danuma]

It is compiling but not working. No , current title is google and expected is my application. I put a breakpoint at let currentTitle = title () and navigate to google.com manually to pass the condition

When the condition is true ,I see the cursor at
skipRemainingTestsInContextOnFailure <- true and upon pressing F10
it is stepping to the line number 184 in runner.fs

[lefthandedgoat]

Ok I think I understand.

This flag is just a configuration setting, so if a test fails, it will skip any remaining tests in the current context.

You need to set it to true, then if a test fails it will work.

If there is a failure in the once code it will fail the suite also, you can see that here:

https://github.com/lefthandedgoat/canopy/blob/master/src/canopy/runner.fs#L205-L208

If you want your once to fail if your title is wrong, don't use an if statement, use an assertion like:

currentTitle === expectedTitle

[danuma]

ok will try that.

[danuma]

Got the exception "equality check failed" stepped into and executed line no .111 in runner.fs and the following lines were not executed
reporter.testStart test.Id
fail ex test suite true <| safelyGetUrl()
reporter.testEnd test.Id

my code is

let all _ =
context "Executing Tests"
once (fun _ ->
start firefox
url "application.com"

let currentTitle = title()
currentTitle === expectedTitle

[lefthandedgoat]

Its generally best to not run your tests in Visual Studio because the debugger will attach to any exceptions it thinks are un-handeled.

Try running the exe from console and it should work.

[danuma]

My test environment was down today , Jenkins job executed exe , and it didnt work still , all the tests i have were executed and reported failure.

[lefthandedgoat]

I need to see your full test to help.

[danuma]

Pls let me know if I can reach you out via email.

[lefthandedgoat]

lefthandedgoat gmail

On Tue, Aug 9, 2016 at 3:37 PM, danuma [email protected] wrote:

Pls let me know if I can reach you out via email.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
#242 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAsQ6WQRY1zi4_LlsACU6c0e3sxdeYooks5qeOT5gaJpZM4HtrJp
.