Add SDK testing framework

[faboweb]

Summary

To create solid applications with the SDK. Developers will need a testing framework. This testing framework will need the following abilities:

• create a node and/or Gaia Lite server per test
• create a deterministic state per test including the keys/addresses

Credits to @NodeGuy for pushing this

Problem Definition

Proposal

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[fedekunze]

@faboweb should we add this functionality to #1553 ?

[faboweb]

I am not sure. This should be used for more then just the LCD.

[jaekwon]

• create a node and/or Gaia Lite server per test
• create a deterministic state per test including the keys/addresses

Do these items need to be together? I think more context would be helpful, and following discussion/review for a spec for how this would be implemented.

[faboweb]

Pulling in @NodeGuy

[jaekwon]

I'm just thinking out loud here, this may be orthogonal to the discussion at hand. I'm just trying to see what an ideal test initialization system looks like that would satisfy "create a deterministic state per test including the keys/addresses", I think much depends on the interfaces/conventions we have at the model (store) layer. So for example, if we supported a convention like:

store := tsuite.NewIAVLStore()
acc := types.NewAccount(...)

// POJO convention that also handles index entries:
acc.Save(store) // panics if already exists
acc.Update(store) // panics if doesn't already exist
acc.SaveOrUpdate(store) // never panics
acc.Delete(store) // panics if it doesn't exist
acc.DeleteOrNoop(store) // never panics

// You can create complex structures that implement the same...
type AccountSet struct {...}
func (accs AccountSet) Save(store) {
    for _, acc := range accs.accounts {
        acc.Save(store)
    }
}
func (accs AccountSet) Update(store) {
    for _, acc := range accs.accounts {
        acc.Update(store)
    }
}
...

then we could do:

func setupTestAccounts(store) {
    testAccs := generateTestAccounts()
    testAccs.Save(store)
}

So I guess what I'm trying to emphasize is that the objective isn't to create a testing "framework" per se, but figuring out ways to make testing easier... depending on concrete points of difficulty in testing, we can figure out the best way to make that (and future tests) easier, and often the changes that need to be made are best made by refactoring the model/logic layer, which sits under the "testing framework" layer.

[rigelrozanski]

As I understand it - the intent of the framework is to create a way for developers to quickly/painlessly setup a scenario with which to run their test scenario on top of. YES great idea.

As a part of the "setup" phase of a test we would also want to expose all the keepers/stores of the application to be able to execute/retrieve results from certain commands.

---

More on what @jaekwon comment.

So I guess what I'm trying to emphasize is that the objective isn't to create a testing "framework" per se, but figuring out ways to make testing easier...

Yeah maybe this shouldn't be dubbed a framework - but more of a suite of tooling for devs to draw from to quickly create their test situation.

Creating some conventions as a part of this suite, such as the account interface you've proposed seems like a reasonable way to streamline test creation/modularity. I like it 👍

---

Overall I'd like to see this suite created it such a way that allow easy setup for both high level and low level testing - high level testing being more testing at a handler level where the test is really only sending transactions and querying for the state. Low-level testing being where are calling individual (maybe unexposed) functions within a module and parsing out details of the state from special keeper get functions.

If we do it right this will probably significantly reduce bugs in the long term just by merit that people will feel comfortable writing more testing.

[jackzampolin]

I think conversation is very focused on the go code right now and isn't thinking about clients. I think what @faboweb is asking for is more a way to have a test network that the developer has much more granular control over.

I think our best bet is something similar to the 10 blocks test in CI where we spin up a network of nodes using make localnet-start, but we also attach an LCD. All we would need to do here then is document how to access the keys for all the accounts in the generated genesis. We could also add more state there (i.e. additional tokens, etc..). This would be a stable platform to simulate a live network which we could continue to support.

Thoughts? cc @greg-szabo

[NodeGuy]

I'm the instigator so I can add more clarity to the request.

So I guess what I'm trying to emphasize is that the objective isn't to create a testing "framework" per se, but figuring out ways to make testing easier... depending on concrete points of difficulty in testing, we can figure out the best way to make that (and future tests) easier, and often the changes that need to be made are best made by refactoring the model/logic layer, which sits under the "testing framework" layer.

Exactly.

I think conversation is very focused on the go code right now and isn't thinking about clients. I think what @faboweb is asking for is more a way to have a test network that the developer has much more granular control over.

That sounds super cool but is not part of this request.

Purpose

Let's start with the purpose. Good tests have the following properties (excerpted from F.I.R.S.T):

• Fast: Tests must be fast. If you hesitate to run the tests after a simple one-liner change, your tests are far too slow. Make the tests so fast you don't have to consider them. A test that takes a second or more is not a fast test, but an impossibly slow test.
• Isolated: Tests isolate failures. ... A good unit test has a laser-tight focus on a single effect or decision in the system under test. ... Good tests interferes with no other tests in any way. They impose their initial state without aid from other tests. They clean up after themselves.
• Repeatable: Tests must be able to be run repeatedly without intervention. They must not depend upon any assumed initial state, they must not leave any residue behind that would prevent them from being re-run. This is particularly important when one considers resources outside of the program's memory, like databases and files and shared memory segments. (Ed: This implies determinism—tests should never create random data (unless they're fuzz tests) and should yield the same results every time.)

Solution

The solution is to refactor the existing SDK API so that this is easy to do. No new testing framework is necessary.

Let's take an example. Suppose I want to write a test for a function in the light client that takes an account address and returns a list of validators to which that account has delegated. There does exist a test for that today. It's >150 lines of code, creates complex non-deterministic state, exercises REST calls in the light client, and tests 23 assertions in a single function. How can we do this better? Let's start from scratch.

The test needs to create a new instance of a light client. That implies a function in the SDK, let's call it createLightClient. What does createLightClient need? It needs a full node to talk to. OK, let's create a node first, using a function called createNode:

node := createNode()
lightClient := createLightClient(node)

What is the minimum that createNode needs us to provide? Well, I'm not super familiar with this territory but I'll hazard a guess that it needs, at a minimum, a genesis document that allocates a balance to a single validator.

How to generate the validator address? Now we can't just call pvm.LoadOrGenFilePV, which is what the current tests do, because that randomly generates a private key. Instead we need to generate it deterministically, something more like this:

seed := bip39.MnemonicToSeed("barrel original fuel morning among eternal " +
	"filter ball stove pluck matrix mechanic")
master, ch := ComputeMastersFromSeed(seed)
priv, _ := DerivePrivateKeyForPath(master, ch, "44'/0'/0'/0/0")

Back to our psuedo-code:

seed := "blah blah blah"
validator := createKeyFromSeed(seed, 1)
validatorBalance := 42
genesisDocument := createGenesisDocument(validator, validatorBalance)
node := createNode(genesisDocument)
lightClient := createLightClient(node)

OK, now we're getting somewhere. But to test a delegation relationship query we need first to create a delegation relationship:

delegator := createKeyFromSeed(seed, 2)

// Create the delegator account balance.
sendTokens(validator, delegator, 10)

// Delegate 5 tokens to the validator.
delegate(delegator, validator, 5)

Now we have the initial state we need. Note that we did this using the Go API, not by sending REST calls to the light client (which is what the current tests do), because that would add additional surface to the system under test, decreasing our needed isolation described in the above principles.

Ideally all of this would happen without writing anything to disk so that a) the tests would be faster and b) there's less risk of files being written in the wrong place (happened to me) and not being cleaned up properly (happened to me). The APIs would either provide optional parameters for writing to disk instead of memory (exactly like how a database can be "in memory"—notice how popular that option is in tests—instead of writing to disk) or would delegate that task to the calling code. If you do this properly then it becomes trivial to spin up quickly 1,000 nodes, all in memory, and to conduct interesting complex tests against them.

Summary

• No new testing framework is needed.
• We need either to provide examples of how to write tests against the existing API using best practices (e.g., F.I.R.S.T) or to refactor and document the API so that it's easy to do naturally.

[jaekwon]

Nice ideas in there... I think we can get there by migrating our codebase towards it over time (we may not have all of these ideas implemented by launch). I think maybe a good starting point may be to start a new repo (e.g. cosmos-sdk-testing-spec) w/ these ideas as code or pseudocode where we can discuss specific pros/cons of each testing pattern/strategy/tool idea, so that small pieces can begin to implement towards the new standard/convention, and as we learn more we'll be more confident in how best to refactor existing code toward it as well.

I think it should be a new repo (that we can later throw away) because the merge frequency of this SDK will be slower than that of the testing-spec repo.

[NodeGuy]

Thanks, that sounds interesting. It would also be helpful to create a template test now, before any API changes are made, so that we can start using it to write simple tests.

[jackzampolin]

I think this is how the current LCD tests work. Going to close this issue.