Make integration test suites reusable by apps #6711

aaronc · 2020-07-13T21:39:38Z

This is a relatively small refactor of the work done so far on #6423 to make in-process integration tests easily reusable by apps.

For the existing test suites (bank, crisis and distribution) the following changes were made:

move IntegrationTestSuite into client/testutil/suite.go files
add a NewIntegrationTestSuite constructor to the suites which takes a single Config parameter

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

Targeted PR against correct branch (see CONTRIBUTING.md)
Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
Code follows the module structure standards.
Wrote unit and integration tests
Updated relevant documentation (docs/) or specification (x/<module>/spec/)
Added relevant godoc comments.
Added a relevant changelog entry to the Unreleased section in CHANGELOG.md
Re-reviewed Files changed in the Github PR explorer
Review Codecov Report in the comment section below once CI passes

codecov · 2020-07-13T21:43:00Z

Codecov Report

Merging #6711 (dcccac7) into master (1a15412) will increase coverage by 0.81%.
The diff coverage is 98.95%.

@@            Coverage Diff             @@
##           master    #6711      +/-   ##
==========================================
+ Coverage   58.89%   59.70%   +0.81%     
==========================================
  Files         585      595      +10     
  Lines       32801    37315    +4514     
==========================================
+ Hits        19318    22279    +2961     
- Misses      11199    13057    +1858     
+ Partials     2284     1979     -305

Impacted Files	Coverage Δ
x/bank/client/testutil/suite.go	`97.71% <95.00%> (ø)`
types/decimal.go	`70.49% <100.00%> (+0.19%)`	⬆️
x/auth/client/testutil/suite.go	`95.41% <100.00%> (ø)`
x/auth/vesting/client/testutil/suite.go	`100.00% <100.00%> (ø)`
x/authz/client/testutil/query.go	`100.00% <100.00%> (ø)`
x/authz/client/testutil/tx.go	`97.63% <100.00%> (ø)`
x/crisis/client/testsuite/suite.go	`100.00% <100.00%> (ø)`
x/distribution/client/testutil/suite.go	`100.00% <100.00%> (ø)`
x/evidence/client/testutil/suite.go	`100.00% <100.00%> (ø)`
x/feegrant/client/testutil/suite.go	`99.13% <100.00%> (ø)`
... and 62 more

alexanderbez

I just don't personally see the benefit to this approach. I don't want to block on my review, however. SDK developers at large should appropriately review this and merge if there is alignment.

iramiller · 2020-07-17T16:48:10Z

For my team we are very interested in the efforts to make the test suites accessible for integration with the tests of our own modules. There is a lot of benefit from our perspective to ensuring that the application we have built on the SDK continues to support base behaviors correctly as new modules are added. This is especially important in cases such as a module that create extensions on BaseAccount.

My view is that all of the features contained in the resulting app need to be fully exercised regardless of if they were inherited or directly implemented in order to achieve a comprehensive integration test.

fedekunze

good from my end too. Pending lint and conflict fixes

fedekunze · 2020-07-17T16:49:48Z

x/bank/client/testutil/suite.go

@@ -275,7 +272,7 @@ func (s *IntegrationTestSuite) TestNewSendTxCmd() {
 			ctx := context.Background()
 			ctx = context.WithValue(ctx, client.ClientContextKey, &clientCtx)


ineffassign

fedekunze · 2020-07-17T16:49:58Z

x/distribution/client/testutil/suite.go

@@ -543,7 +543,7 @@ func (s *IntegrationTestSuite) TestNewWithdrawRewardsCmd() {
 			ctx := context.Background()
 			ctx = context.WithValue(ctx, client.ClientContextKey, &clientCtx)


ineffassign

…ronc/6423-refactor

alexanderbez · 2020-07-17T18:34:05Z

For my team we are very interested in the efforts to make the test suites accessible for integration with the tests of our own modules. There is a lot of benefit from our perspective to ensuring that the application we have built on the SDK continues to support base behaviors correctly as new modules are added. This is especially important in cases such as a module that create extensions on BaseAccount.

My view is that all of the features contained in the resulting app need to be fully exercised regardless of if they were inherited or directly implemented in order to achieve a comprehensive integration test.

How does what is contained in this PR aid in this regard? Is there a PR or code you can point me to that references this? Module CLI integration tests are self-contained and isolated. How does accessing, say x/bank integration test suite, help you test your module?

I've stated above I'm very reluctant to merging this PR.

aaronc · 2020-07-17T19:01:21Z

For my team we are very interested in the efforts to make the test suites accessible for integration with the tests of our own modules. There is a lot of benefit from our perspective to ensuring that the application we have built on the SDK continues to support base behaviors correctly as new modules are added. This is especially important in cases such as a module that create extensions on BaseAccount.
My view is that all of the features contained in the resulting app need to be fully exercised regardless of if they were inherited or directly implemented in order to achieve a comprehensive integration test.

How does what is contained in this PR aid in this regard? Is there a PR or code you can point me to that references this? Module CLI integration tests are self-contained and isolated. How does accessing, say x/bank integration test suite, help you test your module?

I've stated above I'm very reluctant to merging this PR.

Module CLI integration tests are not self-contained and isolated, they are running against simapp. Simapp is not what zone developers are deploying. I want to be able to run these integration test suites against the wired together app that I am building. I believe that is @iramiller's use case as well.

I am actually in favor of still running these suites as unit tests scoped to their modules - I might update this PR to keep those in place.

But I also want to be able to wire up these suites in Regen's code base to run against Regen's app.

iramiller · 2020-07-17T19:11:40Z

I want to be able to run these integration test suites against the wired together app that I am building. I believe that is @iramiller's use case as well.

Indeed. I am interested in any changes that could be leveraged for exercising the built in SDK unit/integration type tests against my team's app. For reference when I tested the cli_tests from gaia (which required a great deal of tweaking to run in our environment) I encountered several errors where we had made changes that broke expected behavior. As a specific example we have patched our command flags to set some default/provide some structure that is appropriate for our use. During the run of the cli_tests it became clear that we had inadvertently wired a piece incorrectly. While our app behaved as expected an end user trying to follow SDK examples and use some of the extra flags would have found them to not work as expected.

This experience has made me very aware that we have a significant number of crucial pieces in our system that are inherited from the SDK and that they are not entirely isolated and safe from accidental breakage. From my perspective this is very solvable with the simulations, unit, and related integration tests that exist today in the SDK so my hope is that we will see changes that make these more accessible to SDK consumers for comprehensive testing of their own apps.

alexanderbez · 2020-07-17T19:12:14Z

Module CLI integration tests are not self-contained and isolated, they are running against simapp.

Ok, then maybe we have different definitions of self-contained. Just because a suite uses simapp, doesn't mean its not self-contained and isolated. e.g. x/bank CLI integration tests are running x/bank CLI commands and x/bank only -- it doesn't rely on anything else and runs in isolation from other modules integration test suites.

alexanderbez · 2020-07-17T19:20:52Z

I'm still not quite following @iramiller. How Gaia tests the CLI is radically broken, outdated, poorly designed, and most importantly, different than how we perform integration testing now (take a look at x/bank for example).

iramiller · 2020-07-17T19:28:21Z

My point was not that the specific implementation used in gaia cli_test is the ideal... rather it is the ability to exercise tests built into the SDK for testing SDK functionality even when incorporated into an end user application that is important. Any group that builds with the SDK is taking a risk if they release software that has not verified the base features are still present and working correctly.

github-actions · 2020-09-05T00:16:47Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

sahith-narahari · 2020-09-06T19:23:38Z

@aaronc is this still in path for stargate

cc @clevinson

amaury1093 · 2021-04-14T11:25:51Z

I think the proposed changes won't break that - we will need to run/select tests through command line

I'd love that. Could you write the command to run a single test in this PR? If we have that, I'm approving this PR.

robert-zaremba · 2021-04-14T14:47:20Z

I'd love that. Could you write the command to run a single test in this PR? If we have that, I'm approving this PR.

@AmauryM , here is an example with a full log:

cosmos/cosmos-sdk/x/auth/client/testutil @b18f3310* ❯ go test -tags norace -testify.m TestCLIValidateSignatures -v                               20s 16:45:43
=== RUN   TestIntegrationTestSuite
    suite.go:49: setting up integration test suite
    network.go:171: acquiring test network lock
    network.go:176: created temporary directory: /tmp/TestIntegrationTestSuite1178717921/001/chain-byZajI145057107
    network.go:185: preparing test network...
    network.go:373: starting test network...
    network.go:378: started test network
=== RUN   TestIntegrationTestSuite/TestCLIValidateSignatures
=== CONT  TestIntegrationTestSuite
    suite.go:72: tearing down integration test suite
    network.go:465: cleaning up test network...
    network.go:488: finished cleaning up test network
    network.go:462: released test network lock
--- PASS: TestIntegrationTestSuite (17.26s)
    --- PASS: TestIntegrationTestSuite/TestCLIValidateSignatures (0.09s)
PASS
ok      github.com/cosmos/cosmos-sdk/x/auth/client/testutil     17.315s

You can remove -v to surpass the suite logs.

robert-zaremba · 2021-04-14T14:49:38Z

BTW, @alessio - it's possible to use only the go test standard flags (instead of -testify.*, to filter the tests eg:

go test -tags norace -test.run TestIntegrationTestSuite/TestCLIValidateSignatures  -v

amaury1093 · 2021-04-14T15:11:23Z

@robert-zaremba thanks, that works 👍 . I'm good with this. @atheeshp could you fix the conflicts?

aaronc · 2021-04-14T15:16:12Z

I didn't read through all changes. Could you roughly summarize, why do we need to move tests into subpackges?

Just to reiterate, the main goal of this refactoring is so that other apps can run these integration test suites against their app instead of simapp.

…3-refactor

amaury1093

lgtm

atheeshp · 2021-04-22T08:47:20Z

Any further actions on this PR? @clevinson , @aaronc

aaronc · 2021-04-23T18:49:43Z

I'm going to move forward with merging this. It improves the testability of the SDK by apps and we will start leveraging this ASAP for additional QA in gaia, regen and other apps.

@shahankhatch, someone on our team can walk you through using this in gaia

aaronc added 2 commits July 13, 2020 17:36

WIP on refactoring new integration tests.

818400a

godocs

3f0ad0a

godocs

ed3a63c

aaronc marked this pull request as ready for review July 13, 2020 21:44

aaronc requested review from alessio, alexanderbez, clevinson, fedekunze and jgimeno as code owners July 13, 2020 21:44

aaronc mentioned this pull request Jul 13, 2020

In Process Integration Tests #6423

Closed

19 tasks

alexanderbez reviewed Jul 13, 2020

View reviewed changes

fedekunze previously approved these changes Jul 17, 2020

View reviewed changes

aaronc added 2 commits July 17, 2020 12:55

Merge branch 'master' of https://github.com/cosmos/cosmos-sdk into aa…

6439e0a

…ronc/6423-refactor

Fix ineff assign

049d857

Merge branch 'master' into aaronc/6423-refactor

fc68fb7

sahith-narahari marked this pull request as draft July 21, 2020 15:09

github-actions bot added the stale label Sep 5, 2020

github-actions bot removed the stale label Sep 7, 2020

aaronc mentioned this pull request Oct 21, 2020

Reuse integration tests from Cosmos SDK regen-network/regen-ledger#114

Closed

clevinson assigned amaury1093 Oct 21, 2020

aaronc mentioned this pull request Apr 14, 2021

refactor: Split regen ledger into multiple go modules regen-network/regen-ledger#322

Merged

atheeshp added 3 commits April 15, 2021 12:23

Merge branch 'master' of github.com:cosmos/cosmos-sdk into aaronc/642…

68de88c

…3-refactor

update feegrant cfg

713a3fe

fix lint

ed174c1

atheeshp requested review from alessio, robert-zaremba, amaury1093, fedekunze and alexanderbez April 15, 2021 07:02

amaury1093 approved these changes Apr 15, 2021

View reviewed changes

Merge branch 'master' into aaronc/6423-refactor

2920179

amaury1093 requested a review from cyberbono3 April 16, 2021 13:44

atheeshp added 3 commits April 19, 2021 13:30

Merge branch 'master' into aaronc/6423-refactor

8a8fb16

Merge branch 'master' into aaronc/6423-refactor

82d90c2

Merge branch 'master' into aaronc/6423-refactor

feaf4fd

atheeshp added 2 commits April 22, 2021 14:17

Merge branch 'master' into aaronc/6423-refactor

0f1249d

Merge branch 'master' into aaronc/6423-refactor

dcccac7

aaronc merged commit 3c65c3d into master Apr 23, 2021

aaronc deleted the aaronc/6423-refactor branch April 23, 2021 18:49

aaronc removed the backlog label Apr 23, 2021

shahankhatch mentioned this pull request May 4, 2021

Identify and develop gaia-specific tests cosmos/gaia#846

Closed

6 tasks

aleem1314 mentioned this pull request May 19, 2021

x/upgrade gRPC methods for VersionMap #9073

Merged

9 tasks

amaury1093 mentioned this pull request Jul 28, 2022

chore!: Refactor x/bank CLI Tests #12706

Merged

19 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make integration test suites reusable by apps #6711

Make integration test suites reusable by apps #6711

aaronc commented Jul 13, 2020 •

edited

Loading

codecov bot commented Jul 13, 2020 •

edited

Loading

alexanderbez left a comment

iramiller commented Jul 17, 2020

fedekunze left a comment

fedekunze Jul 17, 2020

fedekunze Jul 17, 2020

alexanderbez commented Jul 17, 2020 •

edited

Loading

aaronc commented Jul 17, 2020 •

edited

Loading

iramiller commented Jul 17, 2020

alexanderbez commented Jul 17, 2020

alexanderbez commented Jul 17, 2020

iramiller commented Jul 17, 2020

github-actions bot commented Sep 5, 2020

sahith-narahari commented Sep 6, 2020

amaury1093 commented Apr 14, 2021

robert-zaremba commented Apr 14, 2021

robert-zaremba commented Apr 14, 2021

amaury1093 commented Apr 14, 2021

aaronc commented Apr 14, 2021

amaury1093 left a comment

atheeshp commented Apr 22, 2021

aaronc commented Apr 23, 2021

		@@ -275,7 +272,7 @@ func (s *IntegrationTestSuite) TestNewSendTxCmd() {
		ctx := context.Background()
		ctx = context.WithValue(ctx, client.ClientContextKey, &clientCtx)

		@@ -543,7 +543,7 @@ func (s *IntegrationTestSuite) TestNewWithdrawRewardsCmd() {
		ctx := context.Background()
		ctx = context.WithValue(ctx, client.ClientContextKey, &clientCtx)

Make integration test suites reusable by apps #6711

Make integration test suites reusable by apps #6711

Conversation

aaronc commented Jul 13, 2020 • edited Loading

codecov bot commented Jul 13, 2020 • edited Loading

Codecov Report

alexanderbez left a comment

Choose a reason for hiding this comment

iramiller commented Jul 17, 2020

fedekunze left a comment

Choose a reason for hiding this comment

fedekunze Jul 17, 2020

Choose a reason for hiding this comment

fedekunze Jul 17, 2020

Choose a reason for hiding this comment

alexanderbez commented Jul 17, 2020 • edited Loading

aaronc commented Jul 17, 2020 • edited Loading

iramiller commented Jul 17, 2020

alexanderbez commented Jul 17, 2020

alexanderbez commented Jul 17, 2020

iramiller commented Jul 17, 2020

github-actions bot commented Sep 5, 2020

sahith-narahari commented Sep 6, 2020

amaury1093 commented Apr 14, 2021

robert-zaremba commented Apr 14, 2021

robert-zaremba commented Apr 14, 2021

amaury1093 commented Apr 14, 2021

aaronc commented Apr 14, 2021

amaury1093 left a comment

Choose a reason for hiding this comment

atheeshp commented Apr 22, 2021

aaronc commented Apr 23, 2021

aaronc commented Jul 13, 2020 •

edited

Loading

codecov bot commented Jul 13, 2020 •

edited

Loading

alexanderbez commented Jul 17, 2020 •

edited

Loading

aaronc commented Jul 17, 2020 •

edited

Loading