Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: state migration from IAVL to SMT (ADR-040) #10962

Merged
merged 41 commits into from
Mar 20, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
4cc992f
feat: add migration for adr40
Jan 18, 2022
df0dc4f
chore: removed sort from migration func
Jan 19, 2022
fca170d
Merge branch 'master' into sai/migration_for_adr40
gsk967 Jan 21, 2022
79583ec
chore: address the pr comments
Jan 27, 2022
bcfeff1
Merge branch 'sai/migration_for_adr40' of github.com:vulcanize/cosmos…
Jan 27, 2022
d511fa3
Merge branch 'master' into sai/migration_for_adr40
gsk967 Jan 27, 2022
00b99a7
Merge branch 'master' into sai/migration_for_adr40
gsk967 Jan 27, 2022
0eaec31
Merge branch 'master' into sai/migration_for_adr40
gsk967 Jan 28, 2022
9f496e9
test: add test for migrationv2
Jan 29, 2022
39b303b
Merge branch 'master' into sai/migration_for_adr40
gsk967 Jan 29, 2022
8b16dad
Merge branch 'master' into sai/migration_for_adr40
gsk967 Jan 31, 2022
b305280
chore: change the `MigrationV2`
Jan 31, 2022
ebc1d3b
Merge branch 'master' into sai/migration_for_adr40
gsk967 Jan 31, 2022
64e1cb4
Merge branch 'master' into sai/migration_for_adr40
gsk967 Feb 1, 2022
39d51cf
chore: refactored the migrationV2
Feb 1, 2022
242939e
Merge branch 'master' into sai/migration_for_adr40
gsk967 Feb 1, 2022
a9fa647
Merge branch 'master' into sai/migration_for_adr40
gsk967 Feb 2, 2022
ada2859
Merge branch 'master' into sai/migration_for_adr40
gsk967 Feb 3, 2022
38527f7
chore: address the pr review comments
Feb 15, 2022
1e884cf
Merge branch 'master' into sai/migration_for_adr40
gsk967 Feb 15, 2022
1b79269
Merge branch 'master' into sai/migration_for_adr40
gsk967 Feb 17, 2022
00b6360
Merge branch 'master' into sai/migration_for_adr40
gsk967 Feb 22, 2022
dc57212
Merge branch 'master' into sai/migration_for_adr40
gsk967 Feb 25, 2022
4695365
fix: fix the build issue
Feb 25, 2022
3a5bdf0
Merge branch 'master' into sai/migration_for_adr40
gsk967 Feb 25, 2022
c103f16
Merge branch 'master' into sai/migration_for_adr40
gsk967 Feb 28, 2022
4a9feb5
Merge branch 'master' into sai/migration_for_adr40
gsk967 Mar 4, 2022
677639e
chore: address the pr comments
Mar 4, 2022
074da95
Merge branch 'master' into sai/migration_for_adr40
gsk967 Mar 4, 2022
a05f72e
Merge branch 'master' into sai/migration_for_adr40
gsk967 Mar 6, 2022
6eb51d1
Merge branch 'master' into sai/migration_for_adr40
gsk967 Mar 9, 2022
c16dc89
Merge branch 'master' into sai/migration_for_adr40
gsk967 Mar 10, 2022
119b77a
Merge branch 'master' into sai/migration_for_adr40
gsk967 Mar 11, 2022
a97ed5a
address the pr comments
Mar 11, 2022
179743c
Merge branch 'master' into sai/migration_for_adr40
gsk967 Mar 15, 2022
bb61b5d
address the pr comments
Mar 15, 2022
f381e09
Merge branch 'master' into sai/migration_for_adr40
gsk967 Mar 16, 2022
7eec931
Merge branch 'master' into sai/migration_for_adr40
gsk967 Mar 17, 2022
1af9b3b
chore: update changelog
Mar 17, 2022
ec331a0
Merge branch 'master' into sai/migration_for_adr40
gsk967 Mar 18, 2022
e21a6f8
Merge branch 'master' into sai/migration_for_adr40
gsk967 Mar 20, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions store/rootmulti/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,11 @@ func (rs *Store) GetPruning() types.PruningOptions {
return rs.pruningOpts
}

// GetStores will return the stores
func (rs *Store) GetStores() map[types.StoreKey]types.CommitKVStore {
return rs.stores
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seams we only need store keys. Let's change this function to:

GetStoreKeys() []types.StoreKey

This way we won't expose the internal map.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@robert-zaremba is there any concern with exposing the StoreKey here? or as long as it's not exposed to modules from Context, is it fine?


// SetPruning sets the pruning strategy on the root store and all the sub-stores.
// Note, calling SetPruning on the root store prior to LoadVersion or
// LoadLatestVersion performs a no-op as the stores aren't mounted yet.
Expand Down
70 changes: 70 additions & 0 deletions store/v2/multi/migration.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
package root
gsk967 marked this conversation as resolved.
Show resolved Hide resolved

import (
prefixdb "github.com/cosmos/cosmos-sdk/db/prefix"
gsk967 marked this conversation as resolved.
Show resolved Hide resolved
"github.com/cosmos/cosmos-sdk/store/iavl"
"github.com/cosmos/cosmos-sdk/store/mem"
v1Store "github.com/cosmos/cosmos-sdk/store/rootmulti"
"github.com/cosmos/cosmos-sdk/store/transient"
"github.com/cosmos/cosmos-sdk/store/types"
sdkerrors "github.com/cosmos/cosmos-sdk/types/errors"
"sort"
"strings"
)

// MigrationFromIAVLStoreToSMTStore will migrate the complete state from iavl to smt
func MigrationFromIAVLStoreToSMTStore(rs *v1Store.Store, rootStore *Store) error {
gsk967 marked this conversation as resolved.
Show resolved Hide resolved
// Collect stores to snapshot (only IAVL stores are supported)
type namedStore struct {
*iavl.Store
name string
}
var stores []namedStore
for key := range rs.GetStores() {
switch store := rs.GetCommitKVStore(key).(type) {
case *iavl.Store:
stores = append(stores, namedStore{name: key.Name(), Store: store})
case *transient.Store, *mem.Store:
continue
default:
continue
}
}

sort.Slice(stores, func(i, j int) bool {
gsk967 marked this conversation as resolved.
Show resolved Hide resolved
return strings.Compare(stores[i].name, stores[j].name) == -1
})

// make new smt store schema
if len(rootStore.schema) != 0 {
gsk967 marked this conversation as resolved.
Show resolved Hide resolved
// schema is already exists
gsk967 marked this conversation as resolved.
Show resolved Hide resolved
return sdkerrors.Wrapf(sdkerrors.ErrLogic, "smt store already have schema")
}

// set the schema to smt store
schemaWriter := prefixdb.NewPrefixWriter(rootStore.stateTxn, schemaPrefix)
for _, store := range stores {
rootStore.schema[store.name] = types.StoreTypePersistent
err := schemaWriter.Set([]byte(store.name), []byte{byte(types.StoreTypePersistent)})
if err != nil {
return sdkerrors.Wrap(err, "error at set the store schema key values")
}
}

// iterate through all iavl stores
for _, store := range stores {
subStore, err := rootStore.getSubstore(store.name)
Copy link
Collaborator

@robert-zaremba robert-zaremba Mar 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't we make sure that we use the same store key when doing migration?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if err != nil {
return err
}
// iterate all iavl tree node key/values
iterator := store.Iterator(nil, nil)
for ; iterator.Valid(); iterator.Next() {
// set the iavl key,values into smt node
subStore.Set(iterator.Key(), iterator.Value())
}
}
// commit the all key/values from iavl to smt tree (SMT Store)
rootStore.Commit()
return nil
}