move functional tests to BrowserStack (+ Chrome testing)

[cch5ng]

(update 02.18.17) checklist for this isssue

• 1 either clean up current intern tests which are failing (or at least know clearly which tests are currently failing as expected, locally and on travis)
• 2 update travis and intern.js configurations to run automated browser tests on BrowserStack.com (as of 07.2016 configurations were working for firefox, chrome on BrowserStack but was having some challenge configuring separate scripts to run different configurations)
• 3 make better sense of current failing tests resulting on BrowserStack (step 1 should greatly help with this)
• 4troubleshooting 2 functional authenticated search tests (one is done)
• 5 add a thank you to BrowserStack to the webcompat.com site (and README), per their request from sales rep for giving free test accounts to webcompat
  • my main contacts from 07.2016 were Ady (Adithya Chadalawada) I believe in sales/marketing role and Ashwin from support side
• 6 move the BrowserStack account from my personal email to a general webcompat email account (so I don't become a bottleneck in the future)

(updated 06.13) this issue/automated test enhancement requires

• updating travis and intern.js configurations to run automated browser tests on BrowserStack.com
• troubleshooting 2 functional authenticated search tests (one is done)
• adding a thank you to BrowserStack to the webcompat.com site (and README), per their request from sales rep for giving free test accounts to webcompat

some of the discussion of requirements is in this old PR

(old March 2016) hi (I'm one of the outreachy applicants). while I was testing css changes for a PR, I decided to check with Browser Stack whether they would offer a free testing account for this project. one of their sales reps replied yes as long as there would be a mention of Browser Stack on the project github page and website.

I was thinking there would be impacts to these files: about page/view, README.md, and CONTRIBUTING.md. I think other people who need to test the UI would be able to request an account for testing in the future. it seems handiest for IE testing (older versions).

does this sound OK? hope this is the right place to ask.

for ref: http://www.browserstack.com/
https://www.browserstack.com/pricing (scroll to free for open source)

[miketaylr]

Hi @cch5ng!

This would be cool. Currently we run our Intern tests on a Firefox instance via Travis CI (which has a built-in Firefox plugin). But more browser coverage would be awesome.

(It would also require figuring out how to get the Intern tests to run there, and how that interacts with Travis).

[magsout]

woot sounds good.

[cch5ng]

hi @miketaylr sorry for the delay. I just sent their sales rep a message to see if they are ok with adding the automated tests through TravisCI. looking at their docs, I see there is support for TravisCI from BrowserStack but I am just wondering if they might have limits in terms of throughput or browsers/versions they'd be willing to support. I'll probably just forward her response to you.

how I explained it to her is that the group runs automated browser tests for new PR's or merging PR's in github. the goal would be to add additional automated browser tests through BrowserStack to TravisCI for additional browser/version combos. hopefully that was accurate.

[cch5ng]

@miketaylr I got a BrowserStack account and started looking at the docs. a few questions (thanks in advance):

• is there a generic webcompat test email I should use for the BrowserStack account? TravisCI's .yml file would need to be updated with the BrowserStack username + auth key. I can use mine for now.
• is there some base list of environments which should be tested (OS, browser, version)?
• there was a recommendation about encrypting the BrowserStack account login info but in the Travis docs there seems to be some caveat where encrypted environment variables will result in the travis tests only being run for project participants (https://docs.travis-ci.com/user/pull-requests#sts=Security Restrictions when testing Pull Requests). is this ok?

[miketaylr]

Hey @cch5ng!

I have a generic [email protected] test account that I've used to set up some throwaway test accounts. If you shoot me an email at ([email protected]) I can give you access to that email account as a delegate (assuming you have a Google Account).

As for base environments, we can start simple, like Mac/Windows and latest browsers. Once we get it running we can tweak our support matrix.

(I suppose we also need to set up the intern tests to run on BrowserStack as well: https://theintern.github.io/intern/#hosted-selenium)

I think having encrypted env vars is OK. We already do that for the GitHub username/login (of a bot) used for the functional tests. I think if I add you to the webcompat org, you should have access to the repo's settings on Travis. I'll do that now and ping me if it gives you any trouble.

Thanks for working on this!

[cch5ng]

I opened an issue with intern (theintern/intern#622) but will look more at the intern config docs. maybe I didn't change enough.

right now I am testing this by committing changes to my forked repo (synced with travisci). should this be sufficient to test or would I have to commit changes to the main repo?

[cch5ng]

@miketaylr I'm seeing BrowserStack running tests. but most are failing. do I assume that the github login is already set in TravisCI as environment variables?

node_modules/.bin/intern-runner config=tests/intern user=testusername pw=testpassword

otherwise I can commit what I have changed and see if I need to backtrack something.

[miketaylr]

@cch5ng yeah, the login and pw are set as Travis encrypted env vars. You should have access to the main webcompat repo now, so one option is to make a branch to test with -- it should run tests from your commits on that branch.

[miketaylr]

(if you don't have push access, let me know)

Just be sure to send a PR from a branch when you're ready for review, thanks!

[cch5ng]

@miketaylr do you mind if I add a couple environment vars to Travis for browserstack authentication?

[miketaylr]

Nope, go for it.

[cch5ng]

update 04.28.16: sorry, my bad. I had copied an intern environment config using windows. when I changed it to mac, this seemed to resolve the failing browser tests.

@miketaylr when I try to run the browser tests locally, I have about half passing and I think the failing ones may be due to my setup for github access. but when I try to run the browser tests on browserstack, it seems as though most of the tests are failing. would this be due to the config of the intern proxy? should I need to change that?

[cch5ng]

the tests seem slowish (12min per 2 test environments). do you want to have browserstack configurations available as an option when pushing major changes? is the intent to move browser tests off of travis? if yes, I think the following lines in .travis.yml may be unnecessary but did not test without them

  - "mkdir -p $HOME/.selenium && cd $HOME/.selenium && wget -nc http://selenium-release.storage.googleapis.com/2.52/selenium-server-standalone-2.52.0.jar"
  - java -jar selenium-server-standalone-2.52.0.jar &> /dev/null &
  - cd -

• (update 04.28.16) this was resolved: looks like the issue is the intern.js siteRoot value. right now BrowserStack is giving a lot of cannot find element errors and it's looking for a local web application instance. is there some qa instance I should repoint to?
• (04.28.16 resolved) when browser tests run on browserstack, I'm not sure if they should be pointing to a webcompat test instance on travis or on browserstack.

[cch5ng]

revisiting checkbox 1 (resolve currently failing intern tests)

observed that it should probably just be a requirement when running local tests to always use server with fixture files

python run.py -t

then running only non auth tests locally => all tests pass consistently (2x, 3rd time, one timeout issue but could be my isp b/c see hangtime with dns lookups)

• CONTRIBUTING.md should be updated accordingly; I'm not quite sure when is a good scenario you'd want to run test w/o fixture files b/c there is at least one test that is dependent on having the fixture file(s) served (issues-non-auth.js => Issue comments load)

• there is another issue of test results (local) not being consistent across users (ex Intermittent errors in tests #1470 which I'm not observing in my recent tests)... (one common early failure was resolved by using the -t option with server

[brizental]

Closing this off so we can start fresh at #1666 :)