Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test running takes increasingly long after a context has finished #452

Open
mkj0 opened this issue Oct 11, 2018 · 8 comments
Open

Test running takes increasingly long after a context has finished #452

mkj0 opened this issue Oct 11, 2018 · 8 comments

Comments

@mkj0
Copy link

mkj0 commented Oct 11, 2018

Hi,

I've noticed something strange, but maybe this is the way we're structuring our tests. In order to ensure a clean slate every time, each of our test contexts creates a new browser, then closes it at the end. I've noticed that as we build up the number of tests, the step of closing the browser and carrying on to the next context takes increasing amounts of time as the test run continues.
At the very beginning, the step is almost instant. It seems like with every new context that is executed, this step takes a second longer than the one before.
I've tried with the LiveHtmlReporter and the ConsolerReporter, both behave the same.
It also does the same whether I'm running debug mode in Visual Studio or straight from a command prompt.
Any advice or suggestions would be appreciated.

Cheers,

Mike

@lefthandedgoat
Copy link
Owner

In your before you start chrome and in your after quit() and the more tests you have the longer each quit takes?

@mkj0
Copy link
Author

mkj0 commented Oct 11, 2018

Almost, in our once we start chrome, and close it in the lastly. I have just left a breakpoint on the quit() calls. When I continue running after that breakpoint, that's when the pause happens.

Just in case it happened to be the ChromeDriver, I've just re-run with Firefox and the GeckoDriver. That suffers from the exact same problem.

More edits. :-) The moment the browser does close, the next context fires up very quickly. It's just the closing the browser that's taking a long time.

@lefthandedgoat
Copy link
Owner

Yes closing a browser does take a long time. If its getting slower to close each time that may be a problem I can address.

I will make a project to try to repro tomorrow and work through it.

@mkj0
Copy link
Author

mkj0 commented Oct 11, 2018

Great, thanks for having a look into it.
Out of curiosity I've just stripped everything out of our project down to a single test:

`module Test

open canopy.classic
open canopy.runner.classic

let tests () =

context "Tests"

before (fun _ ->
start chrome)

after (fun _ ->
quit ())

many 30 (fun _ ->
url "http://www.google.co.uk"
displayed "#lst-ib")`

The only difference here is that the browser is opened and closed in before and after, not once and lastly. It shows broadly the same symptoms, but it's much easier to see the time changes:
T1: 4s
T2: 8s
T3: 10s
T4: 12s
Tn: 2s longer than the last one

Also a side note, I hadn't noticed previously but this test does make it very obvious, the last test is only marked as successful after the browser has closed, rather than when I would expect, which is before after gets called. If the browser closes quickly, you'ld never notice.

Update: lol, that's 5 times I've tried reformatting the code to look like code. Maybe some VS formatting confusing things. I'm sure you can see what's going on. ;-)

@mkj0
Copy link
Author

mkj0 commented Oct 11, 2018

And the same thing is occurring on my home laptop, so it doesn't appear to be machine specific. I've attached my project, should you want to take a look.
Canopy.zip

@mkj0
Copy link
Author

mkj0 commented Oct 17, 2018

Hello,
Just an update - a colleague of mine has found that the pause appears to be a problem with the webdriver rather than canopy. Looks like I'm going to have to adapt our tests (thankfully we haven't built a huge amount) to keep the browser open between contexts.

Cheers,

Mike

@lefthandedgoat
Copy link
Owner

Mike,

Sorry I forgot to look into this. If its an issue with webdriver, make sure yours is up to date with the latest version of chrome. If it is, you may try a few previous versions in case its a recently introduced issue.

I generally just re-use the same browser throughout my tests runs because starting them is slow too.

@mkj0
Copy link
Author

mkj0 commented Oct 18, 2018

Hello again,

Two things now. I'm going to update the tests we have to avoid closing the browser every time, mainly because that probably is a better way for the project to behave, and we also found the problem. The Quit function in canopy.fs (ln 275) isn't cleaning up the list of browsers it has opened. We're not too familiar with F# so we won't be sending you a crude PR, but by removing the current browser from the browsers list, restarting a browser instance was consistently fast every time.

Hope that helps, and thanks for your time.

Mike

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants