Benchmark current invariants, probably remove some from every block

[cwgoes]

Description

As A Gaia developer
I Want to benchmark current invariants run in gaiad
So That I can better measure gaiad invariant performance

Acceptance Criteria

Invariants can be benchmarked via a makefile target (just like all the other simulations) or directly via go test with custom parameters.

Technical Details

Many are O(n) or worse with n = # accounts or n = # validators, etc.
Likely we will need to disable those by default and rely on the invariant-broken tx.

[cwgoes]

cc @rigelrozanski

[hendrikhofstadt]

Especially the balance invariant was pretty slow when I last checked.

Could we also remove the invariant checks when we are fast syncing? Most of the time during a sync is currently spent for the invariant checks which are not necessary since we already have valid commits and invariants are only relevant for a running chain (or the latest blocks of one).

[rigelrozanski]

In full agreement - let's try to get the invar-tx stuff in here for next release

[rigelrozanski]

@cwgoes do you think this issues is still relevant now that we've removed usage of all blockly checks by default? The optional feature allows a node to run blockly checks for all invariants that we accepts within the crisis module

[cwgoes]

I think it is much less critical for the immediate future, we probably don't need any code changes now.

It still would be prudent for someone to keep tabs (by running a full node) on the time it takes to run different invariants, so that we'll know ahead of time if some are becoming too slow (even given the Atom deposit for reporting an invariant violation).

[alexanderbez]

I'd be happy to run and gather the benchmarks into a document/markdown file. Do you care if I take this @rigelrozanski?

[rigelrozanski]

@alexanderbez please!

[jaekwon]

Yeah so we need good tooling & need to set up internally a daily report on how long it takes to run this invariant locally given the current state of the chain... and we need to take care that the crisis check doesn't end up halting the chain even though there is no invariant issue, due to OOM issues. This probably means we need some kind of memory usage monitoring during the invariant checks as well.

[rigelrozanski]

Oh I like the idea of measuring the invariant run time. Currently we have it set so that nodes don't run invariants by default, but they can run it every block if they like. What we could do is modify this further such that they instead could alternatively run invariants once every X timeduration... and log the results.... I suppose this ultimately makes more sense then running it once per block anyways (because if you wanted to run it once per block you could just set this duration to 1 second)

[alexanderbez]

Yeah so we need good tooling & need to set up internally a daily report on how long it takes to run this invariant locally given the current state of the chain...

The "daily report" can just run the benchmarks -- not sure what tooling we need?

What we could do is modify this further such that they instead could alternatively run invariants once every X timeduration...

Wouldn't it make more sense to do it every n blocks instead? That way it'll be a bit more deterministic.