Enable ci-integration tests on GHA (incl fixing a few tests) #5202

cognifloyd · 2021-03-23T04:40:15Z

Move integration tests to GitHub Actions (GHA)

We also have to fix a few tests to make the transition.

List of commits:

improve CI output
improve GHA folding
fix virtualenv lock files issue again
Switch tests to check w/ CA that is present in GHA
apt download cache
use bitnami/rabbitmq service container in GHA
bust python cache
Use virtualenv from st2 virtualenv by default
Installing virtualenv in --user on github
Add workaround for failing stdin test
fix check-dependency-conflicts by updating pip-tools

- remove git clone progress reporting (reduce CI noise) - report current user

- no random ports - use separate rabbitmq.conf file on github - ci: temporarily disable all but ci-integration - ci: cp ssl_certs into rabbitmq container - reenable management plugin - manual custom.conf - add python-version for ci-integration tests - add default rabbitmq user/pass - do not hardcode RMQ user/pass in tests - override more rabbitmq test urls - fix integration test conf files on the fly - go back to guest:guest with rabbitmq - finish reverting rmq guest:guest stuff - Reconfigure RabbitMQ for ci-unit tests as well

If a virtualenv was built with symlinks instead of copies of the python binary, as happens by default on GHA integration tests, then we end up using the system virtualenv binary instead of what we installed. Currently GHA has the same version of virtualenv, but we build virtualenvs with --system-site-packages, and we need it to use the st2 virtualenv as the "system" dir instead of the GHA provided one. So, don't get the realpath of the python binary. Also, fall back to system virtualenv bin for backwards compat.

- install virtualenv in --user - pip uninstall --user - Try to recreate failure on Travis - leave TODO/FIXME around broken test

- Use close_fds=True. - add workaround for failing test

When we updated pip, we missed updating pip-tools to support the newer pip version, so there were errors (that didn't fail the build, oops) like these (fixed in pip-tools 5.4, pip-tools-PR-1216: virtualenv/bin/pip-compile req.txt Traceback (most recent call last): File "virtualenv/bin/pip-compile", line 8, in <module> sys.exit(cli()) File "/home/runner/work/st2/st2/virtualenv/lib/python3.6/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/home/runner/work/st2/st2/virtualenv/lib/python3.6/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/runner/work/st2/st2/virtualenv/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/runner/work/st2/st2/virtualenv/lib/python3.6/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/runner/work/st2/st2/virtualenv/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func return f(get_current_context(), *args, **kwargs) File "/home/runner/work/st2/st2/virtualenv/lib/python3.6/site-packages/piptools/scripts/compile.py", line 458, in cli results = resolver.resolve(max_rounds=max_rounds) File "/home/runner/work/st2/st2/virtualenv/lib/python3.6/site-packages/piptools/resolver.py", line 173, in resolve has_changed, best_matches = self._resolve_one_round() File "/home/runner/work/st2/st2/virtualenv/lib/python3.6/site-packages/piptools/resolver.py", line 278, in _resolve_one_round their_constraints.extend(self._iter_dependencies(best_match)) File "/home/runner/work/st2/st2/virtualenv/lib/python3.6/site-packages/piptools/resolver.py", line 388, in _iter_dependencies dependencies = self.repository.get_dependencies(ireq) File "/home/runner/work/st2/st2/virtualenv/lib/python3.6/site-packages/piptools/repositories/local.py", line 75, in get_dependencies return self.repository.get_dependencies(ireq) File "/home/runner/work/st2/st2/virtualenv/lib/python3.6/site-packages/piptools/repositories/pypi.py", line 232, in get_dependencies download_dir, ireq, wheel_cache File "/home/runner/work/st2/st2/virtualenv/lib/python3.6/site-packages/piptools/repositories/pypi.py", line 163, in resolve_reqs wheel_download_dir=self._wheel_download_dir, TypeError: make_requirement_preparer() got an unexpected keyword argument 'wheel_download_dir'

cognifloyd · 2021-03-23T05:03:41Z

Historical note:
I built this in st2sandbox#1 then squashed/reordered the 100+ commits into a few discreet commits in st2sandbox#2. In squashing/reordering commits, I removed most of the history of places I added various forms of debug logging.

Once this is merged, a maintainer should disable StackStorm/st2 on travis-ci.com. A future PR can clean up the repo, removing travis config, and getting rid of anything else that is travis-specific.

cognifloyd · 2021-03-23T05:20:49Z

failure in integration tests is one of the orquesta tests that have been flaky lately. Please restart GitHub Actions workflows.

We still need the symlink in ~/.local/bin after restoring the cached ~/virtualenv directory. Also, be more verbose by printing the commands.

cognifloyd · 2021-03-23T07:44:57Z

Fixes a bug with the cache. All tests pass now.

Kami · 2021-03-23T09:52:22Z

.github/workflows/ci.yaml

-          sudo service rabbitmq-server restart
-          sleep 5
-          sudo rabbitmq-plugins enable rabbitmq_management
+          sudo cp scripts/github/rabbitmq.conf /home/runner/rabbitmq_conf/custom.conf


In the future I think it would be better to move that code in scripts/ci/setup-rabbitmq.sh or similar so we can also run shell check, etc on those files.

Kami · 2021-03-23T09:53:51Z

tools/launchdev.sh

-        git clone https://github.com/StackStorm/st2tests.git
+        echo Cloning https://github.com/StackStorm/st2tests.git
+        # -q = no progress reporting (better for CI). Errors will still print.
+        git clone -q https://github.com/StackStorm/st2tests.git


I guess we could even use --depth 1 // swallow clone to speed things up, but then again that repo is not that bug.

Kami · 2021-03-23T09:54:57Z

scripts/travis/prepare-integration.sh

@@ -16,6 +16,9 @@ st2 --version
 # Clean up old screen log files
 rm -f logs/screen-*.log

+# ::group::/::endgroup:: is helpful github actions syntax to fold this section.
+echo ::group::launchdev.sh start -x


As per comment above since that script is now specific to gha, we should probably rename the directory and update the affected files :)

It's not very difficult to make this work across multiple CI providers (like I did in ansible/molecule#2976 ), but I didn't see the point since this will only be running in GHA for now.

As I don't want to disable Travis until after this PR is merged, I minimized my changes to the travis scripts. Once we disable travis, though, there's a lot of code on the chopping block.

Kami · 2021-03-23T09:59:40Z

Thanks for doing this, it's great to how all the tests now run on GHA.

As far as orquesta test failures go - yeah, it appears that some orquesta integration tests which were failing quite often on Travis are also failing quite often on GHA.

I already assumed before it's some kind of race condition and now it looks even more likely.

@m4dcoder Do you think setting up a coordination backend for integration tests would make a difference with those race conditions / occasional failures? Or it's likely some other root cause we should look at improving in tests / similar?

Here are some examples:

https://travis-ci.com/github/StackStorm/st2/builds/220669792
https://travis-ci.com/github/StackStorm/st2/builds/220020111
https://travis-ci.com/github/StackStorm/st2/builds/220015532 - here they also hand and timeout, perhaps some deadlock?

arm4b · 2021-03-23T11:49:18Z

Wohoo! Congrats @cognifloyd making that work, great achievement here that helps everyone a lot! 🤘

arm4b

✔️ integration tests for the GH Actions is the best indicator for me 👍
Great work!

Quick notes:
ci.yaml will need a bit of cleanup from the irrelevant comments and notes. Can we also remove Travis.yml?

cognifloyd · 2021-03-23T13:17:35Z

Agreed, we need to do quite a bit of cleanup to get rid of all the remnants of travis.

The scope of this PR is to do the bare minimum to get it working in GHA. I do not want to disable Travis in this PR to demonstrate that it works in both CI systems. Once merged, a Maintainer can go into the dashboard on travis-ci.com to disable running Travis on this repo.

Also, I tried to document in ci.yaml why I made various decisions so that when we do cleanup that file, we don't inadvertently break things by going back to something that didn't work. That said, there is a lot of room for improvement with the ci files. I definitely want to see better organization with some scripts moving out of the workflow definition and others just going away as we make them obsolete.

Please, let's save a lot of this cleanup for future PRs.

blag · 2021-03-23T18:51:38Z

contrib/runners/python_runner/python_runner/python_action_wrapper.py

+            # This could indicate that parent process (e.g. process which runs the tests has
+            # incorrectly opened the stdin and that one is then inherited by the process which is
+            # spawning it which will cause issues)
+            raise ValueError("Received no valid parameters data from sys.stdin")


Not a blocker, but noting this in the future: it would be a good idea to log what already has been read into stdin_data.

Suggested change

raise ValueError("Received no valid parameters data from sys.stdin")

raise ValueError(f"Received no valid parameters data from sys.stdin:\n{stdin_data}")

cognifloyd and others added 11 commits March 22, 2021 19:22

improve CI output

f1126dc

- remove git clone progress reporting (reduce CI noise) - report current user

improve GHA folding

3046d05

fix virtualenv lock files issue again

fbd09dd

Switch tests to check w/ CA that is present in GHA

204723c

apt download cache

349be7a

bust python cache

783335e

Installing virtualenv in --user on github

f87e999

- install virtualenv in --user - pip uninstall --user - Try to recreate failure on Travis - leave TODO/FIXME around broken test

Add workaround for failing stdin test

ac6fb79

- Use close_fds=True. - add workaround for failing test

cognifloyd requested a review from blag March 23, 2021 04:40

pull-request-size bot added the size/L PR that changes 100-499 lines. Requires some effort to review. label Mar 23, 2021

cognifloyd added this to the 3.5.0 milestone Mar 23, 2021

cognifloyd requested review from nmaludy and Kami March 23, 2021 04:41

Handle restored ~/virtualenv cache correctly

3193858

We still need the symlink in ~/.local/bin after restoring the cached ~/virtualenv directory. Also, be more verbose by printing the commands.

cognifloyd force-pushed the ci-integration-to-gha branch from 5732597 to 3193858 Compare March 23, 2021 07:00

Kami reviewed Mar 23, 2021

View reviewed changes

arm4b added the feature label Mar 23, 2021

arm4b approved these changes Mar 23, 2021

View reviewed changes

arm4b added the infrastructure: ci/cd label Mar 23, 2021

Merge branch 'master' into ci-integration-to-gha

28833ec

blag approved these changes Mar 23, 2021

View reviewed changes

blag merged commit 007beed into StackStorm:master Mar 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable ci-integration tests on GHA (incl fixing a few tests) #5202

Enable ci-integration tests on GHA (incl fixing a few tests) #5202

cognifloyd commented Mar 23, 2021

cognifloyd commented Mar 23, 2021

cognifloyd commented Mar 23, 2021

cognifloyd commented Mar 23, 2021

Kami Mar 23, 2021

Kami Mar 23, 2021

Kami Mar 23, 2021

cognifloyd Mar 23, 2021

Kami commented Mar 23, 2021

arm4b commented Mar 23, 2021 •

edited

Loading

arm4b left a comment

cognifloyd commented Mar 23, 2021 •

edited

Loading

blag Mar 23, 2021

	raise ValueError("Received no valid parameters data from sys.stdin")
	raise ValueError(f"Received no valid parameters data from sys.stdin:\n{stdin_data}")

Enable ci-integration tests on GHA (incl fixing a few tests) #5202

Enable ci-integration tests on GHA (incl fixing a few tests) #5202

Conversation

cognifloyd commented Mar 23, 2021

cognifloyd commented Mar 23, 2021

cognifloyd commented Mar 23, 2021

cognifloyd commented Mar 23, 2021

Kami Mar 23, 2021

Choose a reason for hiding this comment

Kami Mar 23, 2021

Choose a reason for hiding this comment

Kami Mar 23, 2021

Choose a reason for hiding this comment

cognifloyd Mar 23, 2021

Choose a reason for hiding this comment

Kami commented Mar 23, 2021

arm4b commented Mar 23, 2021 • edited Loading

arm4b left a comment

Choose a reason for hiding this comment

cognifloyd commented Mar 23, 2021 • edited Loading

blag Mar 23, 2021

Choose a reason for hiding this comment

arm4b commented Mar 23, 2021 •

edited

Loading

cognifloyd commented Mar 23, 2021 •

edited

Loading