Add compatibility tests for v5.0.x #2370

chatton · 2022-09-22T14:27:16Z

Description

While the json file is a bit noisy and has a lot of duplication, this is something we could spend time to generate with tooling if it becomes an issue. For now I think it will be fine to maintain by hand.

~~This workflow was triggered by the UI.~~

~~After bumping versions for client test~~

bump v2 and v3

You can test this by going to

Actions -> Compatibility E2E -> run workflow (choose this branch - not main) -> click Run Workflow

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

Targeted PR against correct branch (see CONTRIBUTING.md)
Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
Code follows the module structure standards.
Wrote unit and integration tests
Updated relevant documentation (docs/) or specification (x/<module>/spec/)
Added relevant godoc comments.
Added a relevant changelog entry to the Unreleased section in CHANGELOG.md
Re-reviewed Files changed in the Github PR explorer
Review Codecov Report in the comment section below once CI passes

chatton · 2022-09-22T14:27:37Z

.github/workflows/e2e-compatibility.yaml

-      type: choice
-      options:
-        - release/v5.0.x
+    inputs:


missed the inputs field which enables the UI drop down.

chatton · 2022-09-22T14:53:48Z

.github/compatibility-test-matrices/release-v5.0.x.json

+    {
+      "chain-binary": "simd",
+      "chain-a-tag": "v3.0.0",
+      "chain-b-tag": "release-v5.0.x",
+      "chain-image": "ghcr.io/cosmos/ibc-go-simd",
+      "entrypoint": "TestClientTestSuite",
+      "test": "TestClientUpdateProposal_Succeeds"
+    },
+    {
+      "chain-binary": "simd",
+      "chain-a-tag": "v2.0.0",
+      "chain-b-tag": "release-v5.0.x",
+      "chain-image": "ghcr.io/cosmos/ibc-go-simd",
+      "entrypoint": "TestClientTestSuite",
+      "test": "TestClientUpdateProposal_Succeeds"


these 2 tests are failing, is this expected?

sync'd with @charleenfei and it looks like the TrustingPeriod field was backported to 2.4 and 3.2 (not 2.0/3.0!)

charleenfei · 2022-09-23T09:51:10Z

.github/compatibility-test-matrices/release-v5.0.x.json

+    {
+      "chain-binary": "simd",
+      "chain-a-tag": "release-v5.0.x",
+      "chain-b-tag": "v3.0.0",


i think we could use 3.3.x here (the security release)

charleenfei · 2022-09-23T09:51:24Z

.github/compatibility-test-matrices/release-v5.0.x.json

+    },
+    {
+      "chain-binary": "simd",
+      "chain-a-tag": "v3.0.0",


same comment as above

charleenfei · 2022-09-23T09:52:51Z

.github/compatibility-test-matrices/release-v5.0.x.json

+    {
+      "chain-binary": "simd",
+      "chain-a-tag": "release-v5.0.x",
+      "chain-b-tag": "v2.0.0",


2.1.0 is the security release

charleenfei

just the comment on using the security releases for the v3 and v2 lines otherwise nice!

colin-axner

Excellent! ❤️

colin-axner · 2022-09-26T12:22:40Z

.github/compatibility-test-matrices/release-v5.0.x.json

@@ -1,12 +1,485 @@
 {
  "include": [
+    {
+      "chain-binary": "simd",
+      "chain-a-tag": "release-v5.0.x",


Food for thought: Base tests which should work for any version combination, maybe it'd be possible to swap out the "branch-to-test" and then only maintain files for tests which require logic for which cross versions to test

Actually I was just speaking with @crodriguezvega about this and I think we may be able to make some changes and leverage some of the built in matrix generation features of github actions to make some of these permutations re-usable (swap out the branch as you said).

I will update the versions as you suggested and take a look into seeing if this can be simplified to work a bit better with our use case!

colin-axner

We should probably change v4.0.0 -> v4.1.0 and v2.1.0 -> v2.4.0

It might be nice to actually add every minor release since there can be state machine breaking changes for each minor release, but this looks good to me for now

colin-axner · 2022-09-26T12:27:15Z

.github/compatibility-test-matrices/release-v5.0.x.json

+    {
+      "chain-binary": "simd",
+      "chain-a-tag": "release-v5.0.x",
+      "chain-b-tag": "v4.0.0",


Suggested change

"chain-b-tag": "v4.0.0",

"chain-b-tag": "v4.1.0",

colin-axner · 2022-09-26T12:28:05Z

.github/compatibility-test-matrices/release-v5.0.x.json

+    },
+    {
+      "chain-binary": "simd",
+      "chain-a-tag": "v2.1.0",


Suggested change

"chain-a-tag": "v2.1.0",

"chain-a-tag": "v2.4.0",

I think this is the latest release. Should we just use the release branch?

Edit: would prefer to use final release instead of release branch so as not to test against unreleased changes

damiannolan · 2022-09-27T12:21:08Z

shall we close this one in favour of #2396

chatton added 5 commits September 22, 2022 14:58

chore: fixing workflow inputs

b5c7185

chore: updating release v5.0.x json

329d8e0

chore: add full set of transfer tests

b86a65a

chore: adding incentivized tests

536e37d

chore: adding tests for client update

d5c758b

chatton commented Sep 22, 2022

View reviewed changes

chatton marked this pull request as ready for review September 22, 2022 14:53

chatton requested review from colin-axner, AdityaSripal, crodriguezvega, seantking, charleenfei and damiannolan as code owners September 22, 2022 14:53

chatton commented Sep 22, 2022

View reviewed changes

chatton and others added 3 commits September 22, 2022 16:58

chore: change to 3.2 and 2.4

2759086

Merge branch 'main' into cian/add-additional-tests-for-v5

f123c50

corrected tag

1a4cc7d

charleenfei reviewed Sep 23, 2022

View reviewed changes

charleenfei approved these changes Sep 23, 2022

View reviewed changes

chatton and others added 2 commits September 23, 2022 11:21

chore: bump 3.0 to 3.3 and 2.0 to 2.1

c6508d9

Merge branch 'main' into cian/add-additional-tests-for-v5

b52332b

colin-axner approved these changes Sep 26, 2022

View reviewed changes

colin-axner reviewed Sep 26, 2022

View reviewed changes

chatton mentioned this pull request Sep 26, 2022

Add compatibility tests for v5.0.x #2 #2396

Merged

9 tasks

chatton closed this Sep 27, 2022

chatton deleted the cian/add-additional-tests-for-v5 branch September 27, 2022 12:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add compatibility tests for v5.0.x #2370

Add compatibility tests for v5.0.x #2370

chatton commented Sep 22, 2022 •

edited

Loading

chatton Sep 22, 2022

chatton Sep 22, 2022

chatton Sep 23, 2022

charleenfei Sep 23, 2022 •

edited

Loading

charleenfei Sep 23, 2022

charleenfei Sep 23, 2022

charleenfei left a comment

colin-axner left a comment

colin-axner Sep 26, 2022

chatton Sep 26, 2022

colin-axner left a comment

colin-axner Sep 26, 2022

colin-axner Sep 26, 2022 •

edited

Loading

damiannolan commented Sep 27, 2022

Add compatibility tests for v5.0.x #2370

Add compatibility tests for v5.0.x #2370

Conversation

chatton commented Sep 22, 2022 • edited Loading

Description

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

charleenfei Sep 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

charleenfei left a comment

Choose a reason for hiding this comment

colin-axner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

colin-axner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

colin-axner Sep 26, 2022 • edited Loading

Choose a reason for hiding this comment

damiannolan commented Sep 27, 2022

chatton commented Sep 22, 2022 •

edited

Loading

charleenfei Sep 23, 2022 •

edited

Loading

colin-axner Sep 26, 2022 •

edited

Loading