Intermittent errors in tests

[brizental]

I have played these tests four times, one after the other and each time I got a different result.

The first time I got these errors:

× firefox on any platform - Issue-list - Pressing g goes to github issues
NoSuchElement: [POST http://localhost:4444/wd/hub/session/47c56f18-2777-4e86-bbf9-0caf33a82ecc/element / {"using":"css selector","value":".issues-listing"}] Unable to locate element: .issues-listing
For documentation on this error, please visit: http://seleniumhq.org/exceptions/no_such_element.html
Build info: version: '3.0.1', revision: '1969d75', time: '2016-10-18 09:48:19 -0700'
System info: host: 'bmachado-Lenovo-Z40-70', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0-21-generic', java.version: '1.8.0_111'
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Capabilities [{rotatable=false, raisesAccessibilityExceptions=false, marionette=true, firefoxOptions={args=[], profile=UEsDBBQACAgIACWef0oAAAAAAAAAAAAAAAAHAAAAdXNlci5qc51WTW/bMAy971cMOW3AKqTretlOXdcBA4Z1aFDsKMgSbauRJU0fcfPvR/mjSRNHbndKbJMS+fj4yOjBUeugfLconGnxiXhWQvdf6oo0TLXMAQHNCgVi8eFtyZSH91/exJ2nYAFtrHEhudTAVKj7Z4JGG8ln/DWE1rg1qUOwxNbS19uz9Nky788U6CrU6Pjx8vK52xiwAybwR0AAHkB8l86HK4yFK0C34OJhuKbBvB4pr51pgHrupA3URU2DbJLLxXL6osAKTxAOfauvlfEwnc1oLUyrlWEC79KsSsDWpv1Tg14hWgmpaXeLQdng02W0MYKpGexhE4xRnoBzxnGjvVH7cB+n72WljUbUGmgKcKvu0edz8eC9RKtgkAsOfETcSgyUcsd8nfdVUq+JsaApPAZwmqlUzFczqExlvYt6+rIWCuHkBp8Z54DljBoz90gHysEFP4nEU6Wkt4ptQdycL1e/DDInlfbTtDG+Erf6j9RYX3++JBIvMvd3P9FjwQoTw+dCMb1eHHOuT4gypeiDRzBSnLKH/ji2RysRbrQlbS0DKOkDHvA3SneKCYkGaxnI0E0j6zC5xIUsAIphYbAB8lTzibfRkhq7xuLZtAXFUwdFl0q6OEg5VVt3rCHRYoGBaIS23N6jyat1JNrUSjfZ8IBHJ8MWmaIA/xEfnOSBGicrqak1SvJtnqoaWmw7MuSTqeazQPuTSXq5ikUju1b53b286sj4Mt2cPCab8VN3DoVJRUHLE+q160NMs+34e9yIpizRDs6YtZ4g+0xLiy0VULKowrScjLBzbwf+TEd7zIcs20Y6I/dRqYLbkl4ZO/uPc7bZo0dEbu5/XpELwnZFOk7vgW1YPyGyonpYirFH8jw6jtvBQzfEspf0NE3H5/PcQ5zFYL71SrDiSIIkOufLUxran0o1C/hIYQM6zAyIvdF4ixSbjadQhq87UXtRpTwwx+uBN3NjtyE9u4mIjc1WDVn+Ii4+I4xBhJwU04p6JEKH6nMxvfr0an02th/SKMQZyAfbE9Iy7Djzy83e5PTdXN9NTpSg2Gl1FrlErekg9nrH9KOu24F+6Ot+TTm5nQ0zi3ZqJRBGXdGk/zMkqQHHJa+Br7/1gvW1//L6cUaYEDJFzGYmweTS/L/LsjJVlV4OW1xW+/d6qcv4GrUR27WQKrFPy6oO6sS023Mdb/wfaaNjI5AG98KOiIu0Szo4wGi3yqTVE4mNC+TMLrPre1X6aPsVYLdl/ANQSwcIy87bUaYDAAC2DAAAUEsBAhQAFAAICAgAJZ5/SsvO21GmAwAAtgwAAAcAAAAAAAAAAAAAAAAAAAAAAHVzZXIuanNQSwUGAAAAAAEAAQA1AAAA2wMAAAAA}, appBuildId=20170201175806, version=51.0.1, platform=LINUX, proxy={}, command_id=1, specificationLevel=0, acceptSslCerts=false, processId=8648, browserVersion=51.0.1, platformVersion=4.4.0-21-generic, name=tests/intern, XULappId={ec8030f7-c20a-464f-9b0e-13a3a9e97384}, browserName=firefox, takesScreenshot=true, idle-timeout=60, takesElementScreenshot=true, platformName=linux}]
Session ID: d7e453c8-84f5-429a-83aa-727b067b9eed
*** Element info: {Using=css selector, value=.issues-listing}
  at Test.registerSuite.Pressing g goes to github issues [as test]  <tests/functional/issue-list-non-auth.js:129:10>
× firefox on any platform - Issue-list - Pagination dropdown tests
NoSuchElement: An element could not be located on the page using the given search parameters.
  at Test.registerSuite.Pagination dropdown tests [as test]  <tests/functional/issue-list-non-auth.js:116:10>

The second time no errors.

The third time only the Pressing g goes to github issues error.

And the fourth time only the Pagination dropdown tests error.

I tried them manually and it seems like everything is working, so I don't know why they are failling sometimes.

[magsout]

@brizental yeah, I have same weird results. I don't know why, while test have passed with Travis ...

[miketaylr]

@brizental this is definitely something for Outreachy to investigate! :)

[brizental]

So, apparently these errors are not only happening in the tests from the issue-list-non-auth.js file...
I'll have a look at this and see if I can figure out what's happening :)

[brizental]

So, I wrote a small script to run tests and save the failures results, to let them run multiple times and figure out exactly which tests are failing intermittently.

I let them run overnight around 700 times and what I found out is that these are our intermittent failures:

• Issue list (non-auth) - Pagination dropdown tests
• Issue list (non-auth) - Pressing 'g' goes to github issues
• Issue list (non-auth) - Pressing 'g' inside of search input doesn't go to github issues
• Issues (auth) - Pressing 'l' opens the label editor box
• Issues (non-auth) - Pressing 'g' goes to the github issue page
• Search (auth) - Search works by Return key

If I set a big timeout for the find functions the tests stop failling and the more potent the machine they are run on, the less they fail.

@miketaylr that's it for now.

[miketaylr]

Thanks for reporting your findings @brizental -- all except the first are related to keyboard events. 🤔

The first has a lot of click()s and waiting for UI to show up/get modified. I wonder if that one could just get split up -- it's doing quite a lot of work. Maybe even just create a specific test file for the dropdowns.

[brizental]

While getting everything right for #1671 I think I figured out how to fix this, finally.

Looking at Leadfoot's documentation (https://theintern.github.io/leadfoot/module-leadfoot_Command.html#methods) I realized that most of the functions that get us in trouble return Promises, but in our code we use them as if they were synchronous functions. That's probably why we get less errors when running tests in more potent computers and why sometimes stuff works in our local machines but not in Travis.

I changed the issues-auth.js with that in mind and now everything seems to be working everywhere (https://github.com/webcompat/webcompat.com/pull/1671/files#diff-2c9e405d1862dd6a302519583a505d13). I think seeing this code example will get everything to make more sense.

The only problem that I see in this is that it has the potential for turning our test scripts into callback hell. But anyways, what do you think @miketaylr ?

[miketaylr]

I realized that most of the functions that get us in trouble return Promises, but in our code we use them as if they were synchronous functions.

Can you give some examples? The whole point of Promises is that you can treat them as if they were sync, and that they shouldn't progress down the promise chain until the previous promise resolves (or gets rejected...)

[miketaylr]

The only problem that I see in this is that it has the potential for turning our test scripts into callback hell.

callback hell doesn't really bother me, fwiw.

[brizental]

Closing this because this doesn't happen anymore.