forked from cosmos/cosmos-sdk
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: ADR-067 Simulator v2 (cosmos#16400)
- Loading branch information
1 parent
c82f649
commit 0edd9a3
Showing
4 changed files
with
200 additions
and
5 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6,7 +6,7 @@ | |
|
|
||
| ## Status | ||
|
|
||
| Proposed | ||
| ACCEPTED | ||
|
|
||
| ## Context | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6,7 +6,7 @@ | |
|
|
||
| ## Status | ||
|
|
||
| Proposed | ||
| ACCEPTED | ||
|
|
||
| ## Abstract | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,194 @@ | ||
| # ADR 067: Simulator v2 | ||
|
|
||
| ## Changelog | ||
|
|
||
| * June 01, 2023: Initial Draft (@alexanderbez) | ||
|
|
||
| ## Status | ||
|
|
||
| DRAFT | ||
|
|
||
| ## Abstract | ||
|
|
||
| The Cosmos SDK simulator is a tool that allows developers to test the entirety | ||
| of their application's state machine through the use of pseudo-randomized "operations", | ||
| which represent transactions. The simulator also provides primitives that ensures | ||
| there are no non-determinism issues and that the application's state machine can | ||
| be successfully exported and imported using randomized state. | ||
|
|
||
| The simulator has played an absolutely critical role in the development and testing | ||
| of the Cosmos Hub and all the releases of the Cosmos SDK after the launch of the | ||
| Cosmos Hub. Since the Hub, the simulator has relatively not changed much, so it's | ||
| overdue for a revamp. | ||
|
|
||
| ## Context | ||
|
|
||
| The current simulator, `x/simulation`, acts as a semi-fuzz testing suite that takes | ||
| in an integer that represents a seed into a PRNG. The PRNG is used to generate a | ||
| sequence of "operations" that are meant to reflect transactions that an application's | ||
| state machine can process. Through the use of the PRNG, all aspects of block production | ||
| and consumption are randomized. This includes a block's proposer, the validators | ||
| who both sign and miss the block, along with the transaction operations themselves. | ||
|
|
||
| Each Cosmos SDK module defines a set of simulation operations that _attempt_ to | ||
| produce valid transactions, e.g. `x/distribution/simulation/operations.go`. These | ||
| operations can sometimes fail depending on the accumulated state of the application | ||
| within that simulation run. The simulator will continue to generate operations | ||
| until it has reached a certain number of operations or until it has reached a | ||
| fatal state, reporting results. This gives the ability for application developers | ||
| to reliably execute full range application simulation and fuzz testing against | ||
| their application. | ||
|
|
||
| However, there are a few major drawbacks. Namely, with the advent of ABCI++, specifically | ||
| `FinalizeBlock`, the internal workings of the simulator no longer comply with how | ||
| an application would actually perform. Specifically, operations are executed | ||
| _after_ `FinalizeBlock`, whereas they should be executed _within_ `FinalizeBlock`. | ||
|
|
||
| Additionally, the simulator is not very extensible. Developers should be able to | ||
| easily define and extend the following: | ||
|
|
||
| * Consistency or validity predicates (what are known as invariants today) | ||
| * Property tests of state before and after a block is simulated | ||
|
|
||
| In addition, we also want to achieve the following: | ||
|
|
||
| * Consolidated weight management, i.e. define weights within the simulator itself | ||
| via a config and not defined in each module | ||
| * Observability of the simulator's execution, i.e. have easy to understand output/logs | ||
| with the ability to pipe those logs into some external sink | ||
| * Smart replay, i.e. the ability to not only rerun a simulation from a seed, but | ||
| also the ability to replay from an arbitrary breakpoint | ||
| * Run a simulation based off of real network state | ||
|
|
||
| ## Decision | ||
|
|
||
| Instead of refactoring the existing simulator, `x/simulation`, we propose to create | ||
| a new package in the root of the Cosmos SDK, `simulator`, that will be the new | ||
| simulation framework. The simulator will more accurately reflect the complete | ||
| lifecycle of an ABCI application. | ||
|
|
||
| Specifically, we propose a similar implementation and use of a `simulator.Manager`, | ||
| that exists today, that is responsible for managing the execution of a simulation. | ||
| The manager will wrap an ABCI application and will be responsible for the following: | ||
|
|
||
| * Populating the application's mempool with a set of pseudo-random transactions | ||
| before each block, some of which may contain invalid messages. | ||
| * Selecting transactions and a random proposer to execute `PrepareProposal`. | ||
| * Executing `ProcessProposal`, `FinalizeBlock` and `Commit`. | ||
| * Executing a set of validity predicates before and after each block. | ||
| * Maintaining a CPU and RAM profile of the simulation execution. | ||
| * Allowing a simulation to stop and resume from a given block height. | ||
| * Simulation liveness of each validator per-block. | ||
|
|
||
| From an application developer's perspective, they will only need to provide the | ||
| modules to be used in the simulator and the manager will take care of the rest. | ||
| In addition, they will not need to write their own simulation test(s), e.g. | ||
| non-determinism, multi-seed, etc..., as the manager will provide these as well. | ||
|
|
||
| ```go | ||
| type Manager struct { | ||
| app sdk.Application | ||
| mempool sdk.Mempool | ||
| rng rand.Rand | ||
| // ... | ||
| } | ||
| ``` | ||
|
|
||
| ### Configuration | ||
|
|
||
| The simulator's testing input will be driven by a configuration file, as opposed | ||
| to CLI arguments. This will allow for more extensibility and ease of use along with | ||
| the ability to have shared configuration files across multiple simulations. | ||
|
|
||
| ### Execution | ||
|
|
||
| As alluded to previously, after the execution of each block, the manager will | ||
| generate a series of pseudo-random transactions and attempt to insert them into | ||
| the mempool via `BaseApp#CheckTx`. During the ABCI lifecycle of a block, this | ||
| mempool will be used to seed the transactions into a block proposal as it would | ||
| in a real network. This allows us to not only test the state machine, but also | ||
| test the ABCI lifecycle of a block. | ||
|
|
||
| Statistics, such as total blocks and total failed proposals, will be collected, | ||
| logged and written to output after the full or partial execution of a simulation. | ||
| The output destination of these statistics will be configurable. | ||
|
|
||
| ```go | ||
| func (s *Simulator) SimulateBlock() { | ||
| rProposer := s.SelectRandomProposer() | ||
| rTxs := s.SelectTxs() | ||
|
|
||
| prepareResp, err := s.app.PrepareProposal(&abci.RequestPrepareProposal{Txs: rTxs}) | ||
| // handle error | ||
|
|
||
| processResp, err := s.app.ProcessProposal(&abci.RequestProcessProposal{ | ||
| Txs: prepareResp.Txs, | ||
| // ... | ||
| }) | ||
| // handle error | ||
|
|
||
| // execute liveness matrix... | ||
|
|
||
| _, err = s.app.FinalizeBlock(...) | ||
| // handle error | ||
|
|
||
| _, err = s.app.Commit(...) | ||
| // handle error | ||
| } | ||
| ``` | ||
|
|
||
| Note, some application do not define or need their own app-side mempool, so we | ||
| propose that `SelectTxs` mimic CometBFT and just return FIFO-ordered transactions | ||
| from an ad-hoc simulator mempool. In the case where an application does define | ||
| its own mempool, it will simply ignore what is provided in `RequestPrepareProposal`. | ||
|
|
||
| ### Profiling | ||
|
|
||
| The manager will be responsible for collecting CPU and RAM profiles of the simulation | ||
| execution. We propose to use [Pyroscope](https://pyroscope.io/docs/golang/) to | ||
| capture profiles and export them to a local file and via an HTTP endpoint. This | ||
| can be disabled or enabled by configuration. | ||
|
|
||
| ### Breakpoints | ||
|
|
||
| Via configuration, a caller can express a height-based breakpoint that will allow | ||
| the simulation to stop and resume from a given height. This will allow for debugging | ||
| of CPU, RAM, and state. | ||
|
|
||
| ### Validity Predicates | ||
|
|
||
| We propose to provide the ability for an application to provide the simulator a | ||
| set of validity predicates, i.e. invariant checkers, that will be executed before | ||
| and after each block. This will allow for the application to assert that certain | ||
| state invariants are held before and after each block. Note, as a consequence of | ||
| this, we propose to remove the existing notion of invariants from module production | ||
| execution paths and deprecate their usage all together. | ||
|
|
||
| ```go | ||
| type Manager struct { | ||
| // ... | ||
| preBlockValidator func(sdk.Context) error | ||
| postBlockValidator func(sdk.Context) error | ||
| } | ||
| ``` | ||
|
|
||
| ## Consequences | ||
|
|
||
| ### Backwards Compatibility | ||
|
|
||
| The new simulator package will not naturally not be backwards compatible with the | ||
| existing `x/simulation` module. However, modules will still be responsible for | ||
| providing pseudo-random transactions to the simulator. | ||
|
|
||
| ### Positive | ||
|
|
||
| * Providing more intuitive and cleaner APIs for application developers | ||
| * More closely resembling the true lifecycle of an ABCI application | ||
|
|
||
| ### Negative | ||
|
|
||
| * Breaking current Cosmos SDK module APIs for transaction generation | ||
|
|
||
| ## References | ||
|
|
||
| * [Osmosis Simulation ADR](https://github.com/osmosis-labs/osmosis/blob/main/simulation/ADR.md) |