Skip to content

Commit

Permalink
chore(store): upgrade iavl to v0.19.0 (cosmos#12626)
Browse files Browse the repository at this point in the history
## Description

Closes: #XXXX

Upgrading IAVL to v0.19. This version includes the fast index optimization and errors propagated up top.

Currently, SDK panics on any db error, leaving the proper error handling and refactoring to future work.

Similar change in Osmosis fork, ref: osmosis-labs#108

### Author Checklist

*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*

I have...

- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [x] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/building-modules)
- [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing)
- [x] added a changelog entry to `CHANGELOG.md`
- [x] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [x] updated the relevant documentation or specification
- [x] reviewed "Files changed" and left comments if necessary
- [x] confirmed all CI checks have passed

### Reviewers Checklist

*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*

I have...

- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed 
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
  • Loading branch information
p0mvn authored Jul 20, 2022
1 parent 987959c commit 6f63115
Show file tree
Hide file tree
Showing 16 changed files with 138 additions and 68 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ Ref: https://keepachangelog.com/en/1.0.0/
* [#12187](https://github.com/cosmos/cosmos-sdk/pull/12187) Add batch operation for x/nft module.
* [#12453](https://github.com/cosmos/cosmos-sdk/pull/12453) Add `NewInMemoryWithKeyring` function which allows the creation of in memory `keystore` instances with a specified set of existing items.
* [#11390](https://github.com/cosmos/cosmos-sdk/pull/11390) `LatestBlockResponse` & `BlockByHeightResponse` types' `Block` filed has been deprecated and they now contains new field `sdk_block` with `proposer_address` as `string`
* [#12626](https://github.com/cosmos/cosmos-sdk/pull/12626) Upgrade IAVL to v0.19.0 with fast index and error propagation. NOTE: first start will take a while to propagate into new model.

### State Machine Breaking

Expand Down
17 changes: 17 additions & 0 deletions UPGRADING.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,20 @@ Additionally, new packages have been introduced in order to further split the co
#### PostHandler

`postHandler` is like an `antehandler`, but is run _after_ the `runMsgs` execution. It is in the same store branch that `runMsgs`, meaning that both `runMsgs` and `postHandler`. This allows to run a custom logic after the execution of the messages.

### IAVL

v0.19.0 IAVL introduces a new "fast" index. This index represents the latest state of the
IAVL laid out in a format that preserves data locality by key. As a result, it allows for faster queries and iterations
since data can now be read in lexicographical order that is frequent for Cosmos-SDK chains.

The first time the chain is started after the upgrade, the aforementioned index is created. The creation process
might take time and depends on the size of the latest state of the chain. For example, Osmosis takes around 15 minutes to rebuild the index.

While the index is being created, node operators can observe the following in the logs:
"Upgrading IAVL storage for faster queries + execution on the live state. This may take a while". The store
key is appended to the message. The message is printed for every module that has a non-transient store.
As a result, it gives a good indication of the progress of the upgrade.

There is also downgrade and re-upgrade protection. If a node operator chooses to downgrade to IAVL pre-fast index, and then upgrade again, the index is rebuilt from scratch. This implementation detail should not be relevant in most cases. It was added as a safeguard against operator
mistakes.
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ require (
github.com/cosmos/cosmos-proto v1.0.0-alpha7
github.com/cosmos/cosmos-sdk/db v1.0.0-beta.1
github.com/cosmos/go-bip39 v1.0.0
github.com/cosmos/iavl v0.18.0
github.com/cosmos/iavl v0.19.0
github.com/cosmos/ledger-cosmos-go v0.11.1
github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f // indirect
github.com/gogo/gateway v1.1.0
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -283,8 +283,8 @@ github.com/cosmos/go-bip39 v1.0.0 h1:pcomnQdrdH22njcAatO0yWojsUnCO3y2tNoV1cb6hHY
github.com/cosmos/go-bip39 v1.0.0/go.mod h1:RNJv0H/pOIVgxw6KS7QeX2a0Uo0aKUlfhZ4xuwvCdJw=
github.com/cosmos/gorocksdb v1.2.0 h1:d0l3jJG8M4hBouIZq0mDUHZ+zjOx044J3nGRskwTb4Y=
github.com/cosmos/gorocksdb v1.2.0/go.mod h1:aaKvKItm514hKfNJpUJXnnOWeBnk2GL4+Qw9NHizILw=
github.com/cosmos/iavl v0.18.0 h1:02ur4vnalMR2GuWCFNkuseUcl/BCVmg9tOeHOGiZOkE=
github.com/cosmos/iavl v0.18.0/go.mod h1:L0VZHfq0tqMNJvXlslGExaaiZM7eSm+90Vh9QUbp6j4=
github.com/cosmos/iavl v0.19.0 h1:sgyrjqOkycXiN7Tuupuo4QAldKFg7Sipyfeg/IL7cps=
github.com/cosmos/iavl v0.19.0/go.mod h1:l5h9pAB3m5fihB3pXVgwYqdY8aBsMagqz7T0MUjxZeA=
github.com/cosmos/ledger-cosmos-go v0.11.1 h1:9JIYsGnXP613pb2vPjFeMMjBI5lEDsEaF6oYorTy6J4=
github.com/cosmos/ledger-cosmos-go v0.11.1/go.mod h1:J8//BsAGTo3OC/vDLjMRFLW6q0WAaXvHnVc7ZmE8iUY=
github.com/cosmos/ledger-go v0.9.2 h1:Nnao/dLwaVTk1Q5U9THldpUMMXU94BOTWPddSmVB6pI=
Expand Down
85 changes: 58 additions & 27 deletions store/iavl/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
ics23 "github.com/confio/ics23/go"
"github.com/cosmos/iavl"
abci "github.com/tendermint/tendermint/abci/types"
"github.com/tendermint/tendermint/libs/log"
tmcrypto "github.com/tendermint/tendermint/proto/tendermint/crypto"
dbm "github.com/tendermint/tm-db"

Expand Down Expand Up @@ -42,20 +43,35 @@ type Store struct {
// LoadStore returns an IAVL Store as a CommitKVStore. Internally, it will load the
// store's version (id) from the provided DB. An error is returned if the version
// fails to load, or if called with a positive version on an empty tree.
func LoadStore(db dbm.DB, id types.CommitID, lazyLoading bool, cacheSize int) (types.CommitKVStore, error) {
return LoadStoreWithInitialVersion(db, id, lazyLoading, 0, cacheSize)
func LoadStore(db dbm.DB, logger log.Logger, key types.StoreKey, id types.CommitID, lazyLoading bool, cacheSize int) (types.CommitKVStore, error) {
return LoadStoreWithInitialVersion(db, logger, key, id, lazyLoading, 0, cacheSize)
}

// LoadStoreWithInitialVersion returns an IAVL Store as a CommitKVStore setting its initialVersion
// to the one given. Internally, it will load the store's version (id) from the
// provided DB. An error is returned if the version fails to load, or if called with a positive
// version on an empty tree.
func LoadStoreWithInitialVersion(db dbm.DB, id types.CommitID, lazyLoading bool, initialVersion uint64, cacheSize int) (types.CommitKVStore, error) {
func LoadStoreWithInitialVersion(db dbm.DB, logger log.Logger, key types.StoreKey, id types.CommitID, lazyLoading bool, initialVersion uint64, cacheSize int) (types.CommitKVStore, error) {
tree, err := iavl.NewMutableTreeWithOpts(db, cacheSize, &iavl.Options{InitialVersion: initialVersion})
if err != nil {
return nil, err
}

isUpgradeable, err := tree.IsUpgradeable()
if err != nil {
return nil, err
}

if isUpgradeable && logger != nil {
logger.Info(
"Upgrading IAVL storage for faster queries + execution on live state. This may take a while",
"store_key", key.String(),
"version", initialVersion,
"commit", fmt.Sprintf("%X", id),
"is_lazy", lazyLoading,
)
}

if lazyLoading {
_, err = tree.LazyLoadVersion(id.Version)
} else {
Expand All @@ -66,6 +82,10 @@ func LoadStoreWithInitialVersion(db dbm.DB, id types.CommitID, lazyLoading bool,
return nil, err
}

if logger != nil {
logger.Debug("Finished loading IAVL tree")
}

return &Store{
tree: tree,
}, nil
Expand Down Expand Up @@ -121,9 +141,14 @@ func (st *Store) Commit() types.CommitID {

// LastCommitID implements Committer.
func (st *Store) LastCommitID() types.CommitID {
hash, err := st.tree.Hash()
if err != nil {
panic(err)
}

return types.CommitID{
Version: st.tree.Version(),
Hash: st.tree.Hash(),
Hash: hash,
}
}

Expand Down Expand Up @@ -179,14 +204,21 @@ func (st *Store) Set(key, value []byte) {
// Implements types.KVStore.
func (st *Store) Get(key []byte) []byte {
defer telemetry.MeasureSince(time.Now(), "store", "iavl", "get")
_, value := st.tree.Get(key)
value, err := st.tree.Get(key)
if err != nil {
panic(err)
}
return value
}

// Implements types.KVStore.
func (st *Store) Has(key []byte) (exists bool) {
defer telemetry.MeasureSince(time.Now(), "store", "iavl", "has")
return st.tree.Has(key)
has, err := st.tree.Has(key)
if err != nil {
panic(err)
}
return has
}

// Implements types.KVStore.
Expand All @@ -204,30 +236,20 @@ func (st *Store) DeleteVersions(versions ...int64) error {

// Implements types.KVStore.
func (st *Store) Iterator(start, end []byte) types.Iterator {
var iTree *iavl.ImmutableTree

switch tree := st.tree.(type) {
case *immutableTree:
iTree = tree.ImmutableTree
case *iavl.MutableTree:
iTree = tree.ImmutableTree
iterator, err := st.tree.Iterator(start, end, true)
if err != nil {
panic(err)
}

return newIAVLIterator(iTree, start, end, true)
return iterator
}

// Implements types.KVStore.
func (st *Store) ReverseIterator(start, end []byte) types.Iterator {
var iTree *iavl.ImmutableTree

switch tree := st.tree.(type) {
case *immutableTree:
iTree = tree.ImmutableTree
case *iavl.MutableTree:
iTree = tree.ImmutableTree
iterator, err := st.tree.Iterator(start, end, false)
if err != nil {
panic(err)
}

return newIAVLIterator(iTree, start, end, false)
return iterator
}

// SetInitialVersion sets the initial version of the IAVL tree. It is used when
Expand Down Expand Up @@ -302,7 +324,12 @@ func (st *Store) Query(req abci.RequestQuery) (res abci.ResponseQuery) {
break
}

_, res.Value = tree.GetVersioned(key, res.Height)
value, err := tree.GetVersioned(key, res.Height)
if err != nil {
panic(err)
}
res.Value = value

if !req.Prove {
break
}
Expand Down Expand Up @@ -382,7 +409,7 @@ func getProofFromTree(tree *iavl.MutableTree, key []byte, exists bool) *tmcrypto

// iavlIterator implements types.Iterator.
type iavlIterator struct {
*iavl.Iterator
dbm.Iterator
}

var _ types.Iterator = (*iavlIterator)(nil)
Expand All @@ -391,8 +418,12 @@ var _ types.Iterator = (*iavlIterator)(nil)
// CONTRACT: Caller must release the iavlIterator, as each one creates a new
// goroutine.
func newIAVLIterator(tree *iavl.ImmutableTree, start, end []byte, ascending bool) *iavlIterator {
iterator, err := tree.Iterator(start, end, ascending)
if err != nil {
panic(err)
}
iter := &iavlIterator{
Iterator: tree.Iterator(start, end, ascending),
Iterator: iterator,
}
return iter
}
22 changes: 15 additions & 7 deletions store/iavl/store_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (
"github.com/cosmos/iavl"
"github.com/stretchr/testify/require"
abci "github.com/tendermint/tendermint/abci/types"
"github.com/tendermint/tendermint/libs/log"
dbm "github.com/tendermint/tm-db"

"github.com/cosmos/cosmos-sdk/store/types"
Expand Down Expand Up @@ -58,21 +59,27 @@ func TestLoadStore(t *testing.T) {
store := UnsafeNewStore(tree)

// Create non-pruned height H
require.True(t, tree.Set([]byte("hello"), []byte("hallo")))
updated, err := tree.Set([]byte("hello"), []byte("hallo"))
require.NoError(t, err)
require.True(t, updated)
hash, verH, err := tree.SaveVersion()
cIDH := types.CommitID{Version: verH, Hash: hash}
require.Nil(t, err)

// Create pruned height Hp
require.True(t, tree.Set([]byte("hello"), []byte("hola")))
updated, err = tree.Set([]byte("hello"), []byte("hola"))
require.NoError(t, err)
require.True(t, updated)
hash, verHp, err := tree.SaveVersion()
cIDHp := types.CommitID{Version: verHp, Hash: hash}
require.Nil(t, err)

// TODO: Prune this height

// Create current height Hc
require.True(t, tree.Set([]byte("hello"), []byte("ciao")))
updated, err = tree.Set([]byte("hello"), []byte("ciao"))
require.NoError(t, err)
require.True(t, updated)
hash, verHc, err := tree.SaveVersion()
cIDHc := types.CommitID{Version: verHc, Hash: hash}
require.Nil(t, err)
Expand All @@ -93,17 +100,17 @@ func TestLoadStore(t *testing.T) {
require.Equal(t, string(hcStore.Get([]byte("hello"))), "ciao")

// Querying a new store at some previous non-pruned height H
newHStore, err := LoadStore(db, cIDH, false, DefaultIAVLCacheSize)
newHStore, err := LoadStore(db, log.NewNopLogger(), types.NewKVStoreKey("test"), cIDH, false, DefaultIAVLCacheSize)
require.NoError(t, err)
require.Equal(t, string(newHStore.Get([]byte("hello"))), "hallo")

// Querying a new store at some previous pruned height Hp
newHpStore, err := LoadStore(db, cIDHp, false, DefaultIAVLCacheSize)
newHpStore, err := LoadStore(db, log.NewNopLogger(), types.NewKVStoreKey("test"), cIDHp, false, DefaultIAVLCacheSize)
require.NoError(t, err)
require.Equal(t, string(newHpStore.Get([]byte("hello"))), "hola")

// Querying a new store at current height H
newHcStore, err := LoadStore(db, cIDHc, false, DefaultIAVLCacheSize)
newHcStore, err := LoadStore(db, log.NewNopLogger(), types.NewKVStoreKey("test"), cIDHc, false, DefaultIAVLCacheSize)
require.NoError(t, err)
require.Equal(t, string(newHcStore.Get([]byte("hello"))), "ciao")
}
Expand All @@ -113,7 +120,8 @@ func TestGetImmutable(t *testing.T) {
tree, cID := newAlohaTree(t, db)
store := UnsafeNewStore(tree)

require.True(t, tree.Set([]byte("hello"), []byte("adios")))
updated, err := tree.Set([]byte("hello"), []byte("adios"))
require.True(t, updated)
hash, ver, err := tree.SaveVersion()
cID = types.CommitID{Version: ver, Hash: hash}
require.Nil(t, err)
Expand Down
22 changes: 12 additions & 10 deletions store/iavl/tree.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package iavl
import (
"fmt"

"github.com/cosmos/cosmos-sdk/store/types"
"github.com/cosmos/iavl"
)

Expand All @@ -17,20 +18,21 @@ type (
// implemented by an iavl.MutableTree. For an immutable IAVL tree, a wrapper
// must be made.
Tree interface {
Has(key []byte) bool
Get(key []byte) (index int64, value []byte)
Set(key, value []byte) bool
Remove(key []byte) ([]byte, bool)
Has(key []byte) (bool, error)
Get(key []byte) ([]byte, error)
Set(key, value []byte) (bool, error)
Remove(key []byte) ([]byte, bool, error)
SaveVersion() ([]byte, int64, error)
DeleteVersion(version int64) error
DeleteVersions(versions ...int64) error
Version() int64
Hash() []byte
Hash() ([]byte, error)
VersionExists(version int64) bool
GetVersioned(key []byte, version int64) (int64, []byte)
GetVersioned(key []byte, version int64) ([]byte, error)
GetVersionedWithProof(key []byte, version int64) ([]byte, *iavl.RangeProof, error)
GetImmutable(version int64) (*iavl.ImmutableTree, error)
SetInitialVersion(version uint64)
Iterator(start, end []byte, ascending bool) (types.Iterator, error)
AvailableVersions() []int
}

Expand All @@ -42,11 +44,11 @@ type (
}
)

func (it *immutableTree) Set(_, _ []byte) bool {
func (it *immutableTree) Set(_, _ []byte) (bool, error) {
panic("cannot call 'Set' on an immutable IAVL tree")
}

func (it *immutableTree) Remove(_ []byte) ([]byte, bool) {
func (it *immutableTree) Remove(_ []byte) ([]byte, bool, error) {
panic("cannot call 'Remove' on an immutable IAVL tree")
}

Expand All @@ -70,9 +72,9 @@ func (it *immutableTree) VersionExists(version int64) bool {
return it.Version() == version
}

func (it *immutableTree) GetVersioned(key []byte, version int64) (int64, []byte) {
func (it *immutableTree) GetVersioned(key []byte, version int64) ([]byte, error) {
if it.Version() != version {
return -1, nil
return nil, fmt.Errorf("version mismatch on immutable IAVL tree; got: %d, expected: %d", version, it.Version())
}

return it.Get(key)
Expand Down
4 changes: 0 additions & 4 deletions store/iavl/tree_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,6 @@ func TestImmutableTreePanics(t *testing.T) {
require.Panics(t, func() { it.Remove([]byte{}) })
require.Panics(t, func() { it.SaveVersion() }) // nolint:errcheck
require.Panics(t, func() { it.DeleteVersion(int64(1)) }) // nolint:errcheck
v, _ := it.GetVersioned([]byte{0x01}, 1)
require.Equal(t, int64(-1), v)
v, _ = it.GetVersioned([]byte{0x01}, 0)
require.Equal(t, int64(0), v)

val, proof, err := it.GetVersionedWithProof(nil, 1)
require.Error(t, err)
Expand Down
2 changes: 1 addition & 1 deletion store/rootmulti/proof_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import (
func TestVerifyIAVLStoreQueryProof(t *testing.T) {
// Create main tree for testing.
db := dbm.NewMemDB()
iStore, err := iavl.LoadStore(db, types.CommitID{}, false, iavl.DefaultIAVLCacheSize)
iStore, err := iavl.LoadStore(db, log.NewNopLogger(), types.NewKVStoreKey("test"), types.CommitID{}, false, iavl.DefaultIAVLCacheSize)
store := iStore.(*iavl.Store)
require.Nil(t, err)
store.Set([]byte("MYKEY"), []byte("MYVALUE"))
Expand Down
4 changes: 2 additions & 2 deletions store/rootmulti/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -865,9 +865,9 @@ func (rs *Store) loadCommitStoreFromParams(key types.StoreKey, id types.CommitID
var err error

if params.initialVersion == 0 {
store, err = iavl.LoadStore(db, id, rs.lazyLoading, rs.iavlCacheSize)
store, err = iavl.LoadStore(db, rs.logger, key, id, rs.lazyLoading, rs.iavlCacheSize)
} else {
store, err = iavl.LoadStoreWithInitialVersion(db, id, rs.lazyLoading, params.initialVersion, rs.iavlCacheSize)
store, err = iavl.LoadStoreWithInitialVersion(db, rs.logger, key, id, rs.lazyLoading, params.initialVersion, rs.iavlCacheSize)
}

if err != nil {
Expand Down
2 changes: 1 addition & 1 deletion store/tools/ics23/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ go 1.18
require (
github.com/confio/ics23/go v0.7.0
github.com/cosmos/cosmos-sdk v0.46.0-rc3
github.com/cosmos/iavl v0.18.0
github.com/cosmos/iavl v0.19.0
github.com/lazyledger/smt v0.2.1-0.20210709230900-03ea40719554
github.com/tendermint/tendermint v0.34.20-rc1
github.com/tendermint/tm-db v0.6.7
Expand Down
Loading

0 comments on commit 6f63115

Please sign in to comment.