Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x{stake,slash,gov,distrib} In-place Store Migrations #8504

Merged
merged 60 commits into from
Feb 25, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
10d72d7
Add 1st version of migrate
amaury1093 Feb 1, 2021
8036fca
Merge branch 'master' of ssh://github.com/cosmos/cosmos-sdk into am-8…
amaury1093 Feb 2, 2021
a9a406e
Put migration logic into Configurator
amaury1093 Feb 2, 2021
9ed3987
add test to bank store migration
amaury1093 Feb 2, 2021
a095d3a
add test for configurator
amaury1093 Feb 2, 2021
d550bff
Merge branch 'master' into am-8345-migration
amaury1093 Feb 2, 2021
b7141f9
Error if no migration found
amaury1093 Feb 3, 2021
2d554ce
Remove RunMigrations from Configurator interface
amaury1093 Feb 3, 2021
3df3f44
Update spec
amaury1093 Feb 3, 2021
ba6bd44
Rename folders
amaury1093 Feb 3, 2021
197b7ce
Merge branch 'master' into am-8345-migration
amaury1093 Feb 3, 2021
424932c
copy-paste from keys.go
amaury1093 Feb 3, 2021
254a71f
Merge branch 'am-8345-migration' of ssh://github.com/cosmos/cosmos-sd…
amaury1093 Feb 3, 2021
b6c0714
Fix nil map
amaury1093 Feb 3, 2021
9ab7f13
rename function
amaury1093 Feb 3, 2021
fc076f5
Update simapp/app.go
amaury1093 Feb 5, 2021
ae7b7dc
Update simapp/app_test.go
amaury1093 Feb 5, 2021
2c26734
Adderss reviews
amaury1093 Feb 8, 2021
eb15047
Merge branch 'am-8345-migration' of ssh://github.com/cosmos/cosmos-sd…
amaury1093 Feb 8, 2021
291a9e6
Fix tests
amaury1093 Feb 8, 2021
41d29ed
Update testutil/context.go
amaury1093 Feb 8, 2021
33494dc
Update docs for ConsensusVersion
amaury1093 Feb 8, 2021
e3a2412
Rename to forVersion
amaury1093 Feb 8, 2021
6db31d4
Merge branch 'am-8345-migration' of ssh://github.com/cosmos/cosmos-sd…
amaury1093 Feb 8, 2021
55f547e
Fix tests
amaury1093 Feb 8, 2021
8f3b9d4
Check error early
amaury1093 Feb 8, 2021
29b85b7
Return 1 for intiial version
amaury1093 Feb 8, 2021
41e0339
Use MigrationKeeper
amaury1093 Feb 10, 2021
d922003
Fix test
amaury1093 Feb 10, 2021
f422b93
Revert adding marshaler to Configurator
amaury1093 Feb 10, 2021
8cf88fe
Godoc updates
amaury1093 Feb 10, 2021
5add13a
Update docs
amaury1093 Feb 10, 2021
74b761a
Add distrib legacy folder
amaury1093 Feb 3, 2021
2816a20
Add tests for distrib migration
amaury1093 Feb 3, 2021
f18f253
Add gov migrations
amaury1093 Feb 3, 2021
9e5999e
Copy paste whole keys file
amaury1093 Feb 3, 2021
b15f03d
Add gov migrations
amaury1093 Feb 3, 2021
236e46c
Add staking
amaury1093 Feb 3, 2021
a52441e
Fix staking tests
amaury1093 Feb 3, 2021
b403770
Update spec and module.go
amaury1093 Feb 3, 2021
9546583
Update to latest changes
amaury1093 Feb 8, 2021
33d4e70
Merge branch 'master' of ssh://github.com/cosmos/cosmos-sdk into am-8…
amaury1093 Feb 10, 2021
9e5d611
Update migration scripts
amaury1093 Feb 10, 2021
518b878
capability to 1
amaury1093 Feb 11, 2021
9b625b0
Fix tests
amaury1093 Feb 11, 2021
2392629
Merge branch 'master' into am-8345-modules
amaury1093 Feb 11, 2021
c734484
Add package godoc
amaury1093 Feb 11, 2021
9c11560
Merge branch 'master' into am-8345-modules
amaury1093 Feb 16, 2021
b0a665e
Merge branch 'master' of ssh://github.com/cosmos/cosmos-sdk into am-8…
amaury1093 Feb 18, 2021
0e24bad
Merge branch 'master' into am-8345-modules
amaury1093 Feb 22, 2021
4942857
Remove whitespace
amaury1093 Feb 23, 2021
30d9045
Remove global
amaury1093 Feb 23, 2021
828a41b
Use Migrator
amaury1093 Feb 23, 2021
f3ca3e5
Remove 042 keys files
amaury1093 Feb 23, 2021
0e17848
Fix build
amaury1093 Feb 23, 2021
95f524f
Merge branch 'am-8345-modules' of ssh://github.com/cosmos/cosmos-sdk …
amaury1093 Feb 23, 2021
5456591
Unlambda
amaury1093 Feb 23, 2021
b5ff927
Merge branch 'master' into am-8345-modules
amaury1093 Feb 23, 2021
036bd30
Rename to Migrate1to2
amaury1093 Feb 25, 2021
21e8a15
Merge branch 'master' into am-8345-modules
amaury1093 Feb 25, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 65 additions & 6 deletions simapp/app_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,27 @@ import (
tmproto "github.com/tendermint/tendermint/proto/tendermint/types"
dbm "github.com/tendermint/tm-db"

"github.com/cosmos/cosmos-sdk/baseapp"
sdk "github.com/cosmos/cosmos-sdk/types"
"github.com/cosmos/cosmos-sdk/types/module"
"github.com/cosmos/cosmos-sdk/x/auth"
"github.com/cosmos/cosmos-sdk/x/auth/vesting"
"github.com/cosmos/cosmos-sdk/x/authz"
banktypes "github.com/cosmos/cosmos-sdk/x/bank/types"
"github.com/cosmos/cosmos-sdk/x/capability"
"github.com/cosmos/cosmos-sdk/x/crisis"
"github.com/cosmos/cosmos-sdk/x/distribution"
"github.com/cosmos/cosmos-sdk/x/evidence"
feegrant "github.com/cosmos/cosmos-sdk/x/feegrant"
"github.com/cosmos/cosmos-sdk/x/genutil"
"github.com/cosmos/cosmos-sdk/x/gov"
transfer "github.com/cosmos/cosmos-sdk/x/ibc/applications/transfer"
ibc "github.com/cosmos/cosmos-sdk/x/ibc/core"
"github.com/cosmos/cosmos-sdk/x/mint"
"github.com/cosmos/cosmos-sdk/x/params"
"github.com/cosmos/cosmos-sdk/x/slashing"
"github.com/cosmos/cosmos-sdk/x/staking"
"github.com/cosmos/cosmos-sdk/x/upgrade"
)

func TestSimAppExportAndBlockedAddrs(t *testing.T) {
Expand Down Expand Up @@ -52,11 +71,33 @@ func TestGetMaccPerms(t *testing.T) {
func TestRunMigrations(t *testing.T) {
db := dbm.NewMemDB()
encCfg := MakeTestEncodingConfig()
app := NewSimApp(log.NewTMLogger(log.NewSyncWriter(os.Stdout)), db, nil, true, map[int64]bool{}, DefaultNodeHome, 0, encCfg, EmptyAppOptions{})

// Create a new configurator for the purpose of this test.
logger := log.NewTMLogger(log.NewSyncWriter(os.Stdout))
app := NewSimApp(logger, db, nil, true, map[int64]bool{}, DefaultNodeHome, 0, encCfg, EmptyAppOptions{})

// Create a new baseapp and configurator for the purpose of this test.
bApp := baseapp.NewBaseApp(appName, logger, db, encCfg.TxConfig.TxDecoder())
bApp.SetCommitMultiStoreTracer(nil)
bApp.SetInterfaceRegistry(encCfg.InterfaceRegistry)
app.BaseApp = bApp
app.configurator = module.NewConfigurator(app.MsgServiceRouter(), app.GRPCQueryRouter())

// We register all modules on the Configurator, except x/bank. x/bank will
// serve as the test subject on which we run the migration tests.
//
// The loop below is the same as calling `RegisterServices` on
// ModuleManager, except that we skip x/bank.
for _, module := range app.mm.Modules {
if module.Name() == banktypes.ModuleName {
continue
}

module.RegisterServices(app.configurator)
}

// Initialize the chain
app.InitChain(abci.RequestInitChain{})
app.Commit()

testCases := []struct {
name string
moduleName string
Expand Down Expand Up @@ -115,12 +156,30 @@ func TestRunMigrations(t *testing.T) {
}
require.NoError(t, err)

// Run migrations only for bank. That's why we put the initial
// version for bank as 1, and for all other modules, we put as
// their latest ConsensusVersion.
err = app.RunMigrations(
app.NewContext(true, tmproto.Header{Height: app.LastBlockHeight()}),
module.MigrationMap{
"auth": 1, "authz": 1, "bank": 1, "staking": 1, "mint": 1, "distribution": 1,
"slashing": 1, "gov": 1, "params": 1, "ibc": 1, "upgrade": 1, "vesting": 1,
"feegrant": 1, "transfer": 1, "evidence": 1, "crisis": 1, "genutil": 1, "capability": 1,
"bank": 1,
"auth": auth.AppModule{}.ConsensusVersion(),
"authz": authz.AppModule{}.ConsensusVersion(),
"staking": staking.AppModule{}.ConsensusVersion(),
"mint": mint.AppModule{}.ConsensusVersion(),
"distribution": distribution.AppModule{}.ConsensusVersion(),
"slashing": slashing.AppModule{}.ConsensusVersion(),
"gov": gov.AppModule{}.ConsensusVersion(),
"params": params.AppModule{}.ConsensusVersion(),
"ibc": ibc.AppModule{}.ConsensusVersion(),
"upgrade": upgrade.AppModule{}.ConsensusVersion(),
"vesting": vesting.AppModule{}.ConsensusVersion(),
"feegrant": feegrant.AppModule{}.ConsensusVersion(),
"transfer": transfer.AppModule{}.ConsensusVersion(),
"evidence": evidence.AppModule{}.ConsensusVersion(),
"crisis": crisis.AppModule{}.ConsensusVersion(),
"genutil": genutil.AppModule{}.ConsensusVersion(),
"capability": capability.AppModule{}.ConsensusVersion(),
},
)
if tc.expRunErr {
Expand Down
2 changes: 1 addition & 1 deletion types/module/module.go
Original file line number Diff line number Diff line change
Expand Up @@ -331,7 +331,7 @@ func (m *Manager) ExportGenesis(ctx sdk.Context, cdc codec.JSONMarshaler) map[st
}

// MigrationHandler is the migration function that each module registers.
type MigrationHandler func(store sdk.Context) error
type MigrationHandler func(sdk.Context) error

// MigrationMap is a map of moduleName -> version, where version denotes the
// version from which we should perform the migration for each module.
Expand Down
19 changes: 10 additions & 9 deletions x/bank/keeper/migrations.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,17 @@ import (
v042 "github.com/cosmos/cosmos-sdk/x/bank/legacy/v042"
)

// MigrationKeeper is an interface that the keeper implements for handling
// in-place store migrations.
type MigrationKeeper interface {
// Migrate1 migrates the store from version 1 to 2.
Migrate1(ctx sdk.Context) error
// Migrator is a struct for handling in-place store migrations.
type Migrator struct {
keeper BaseKeeper
}

var _ MigrationKeeper = (*BaseKeeper)(nil)
// NewMigrator returns a new Migrator.
func NewMigrator(keeper BaseKeeper) Migrator {
return Migrator{keeper: keeper}
}

// Migrate1 implements MigrationKeeper.Migrate1 method.
func (keeper BaseKeeper) Migrate1(ctx sdk.Context) error {
return v042.MigrateStore(ctx, keeper.storeKey)
// Migrate1to2 migrates from version 1 to 2.
func (m Migrator) Migrate1to2(ctx sdk.Context) error {
return v042.MigrateStore(ctx, m.keeper.storeKey)
}
2 changes: 2 additions & 0 deletions x/bank/legacy/v040/keys.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
// Package v040 is copy-pasted from:
// https://github.com/cosmos/cosmos-sdk/blob/v0.41.0/x/bank/types/key.go
package v040

import (
Expand Down
50 changes: 0 additions & 50 deletions x/bank/legacy/v042/keys.go

This file was deleted.

3 changes: 2 additions & 1 deletion x/bank/legacy/v042/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (
sdk "github.com/cosmos/cosmos-sdk/types"
v040auth "github.com/cosmos/cosmos-sdk/x/auth/legacy/v040"
v040bank "github.com/cosmos/cosmos-sdk/x/bank/legacy/v040"
"github.com/cosmos/cosmos-sdk/x/bank/types"
)

// MigrateStore performs in-place store migrations from v0.40 to v0.42. The
Expand All @@ -27,7 +28,7 @@ func MigrateStore(ctx sdk.Context, storeKey sdk.StoreKey) error {
for ; oldStoreIter.Valid(); oldStoreIter.Next() {
addr := v040bank.AddressFromBalancesStore(oldStoreIter.Key())
denom := oldStoreIter.Key()[v040auth.AddrLen:]
newStoreKey := append(CreateAccountBalancesPrefix(addr), denom...)
newStoreKey := append(types.CreateAccountBalancesPrefix(addr), denom...)

// Set new key on store. Values don't change.
store.Set(newStoreKey, oldStoreIter.Value())
Expand Down
3 changes: 2 additions & 1 deletion x/bank/legacy/v042/store_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (
sdk "github.com/cosmos/cosmos-sdk/types"
v040bank "github.com/cosmos/cosmos-sdk/x/bank/legacy/v040"
v042bank "github.com/cosmos/cosmos-sdk/x/bank/legacy/v042"
"github.com/cosmos/cosmos-sdk/x/bank/types"
)

func TestStoreMigration(t *testing.T) {
Expand All @@ -27,7 +28,7 @@ func TestStoreMigration(t *testing.T) {
err := v042bank.MigrateStore(ctx, bankKey)
require.NoError(t, err)

newKey := append(v042bank.CreateAccountBalancesPrefix(addr), denom...)
newKey := append(types.CreateAccountBalancesPrefix(addr), denom...)
// -7 because we replaced "balances" with 0x02,
// +1 because we added length-prefix to address.
require.Equal(t, len(oldKey)-7+1, len(newKey))
Expand Down
6 changes: 3 additions & 3 deletions x/bank/module.go
Original file line number Diff line number Diff line change
Expand Up @@ -98,9 +98,9 @@ type AppModule struct {
func (am AppModule) RegisterServices(cfg module.Configurator) {
types.RegisterMsgServer(cfg.MsgServer(), keeper.NewMsgServerImpl(am.keeper))
types.RegisterQueryServer(cfg.QueryServer(), am.keeper)
cfg.RegisterMigration(types.ModuleName, 0, func(ctx sdk.Context) error {
return am.keeper.(keeper.MigrationKeeper).Migrate1(ctx)
})

m := keeper.NewMigrator(am.keeper.(keeper.BaseKeeper))
cfg.RegisterMigration(types.ModuleName, 1, m.Migrate1to2)
}

// NewAppModule creates a new AppModule object
Expand Down
21 changes: 21 additions & 0 deletions x/distribution/keeper/migrations.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
package keeper

import (
sdk "github.com/cosmos/cosmos-sdk/types"
v042 "github.com/cosmos/cosmos-sdk/x/distribution/legacy/v042"
)

// Migrator is a struct for handling in-place store migrations.
type Migrator struct {
keeper Keeper
}
amaury1093 marked this conversation as resolved.
Show resolved Hide resolved

// NewMigrator returns a new Migrator.
func NewMigrator(keeper Keeper) Migrator {
return Migrator{keeper: keeper}
}

// Migrate1to2 migrates from version 1 to 2.
func (m Migrator) Migrate1to2(ctx sdk.Context) error {
return v042.MigrateStore(ctx, m.keeper.storeKey)
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of enlarging the Keeper interface and responsibilities, let's create a Migrator structure:

type Migrator struct {
   keeper Keeper
}

// in RegisterServies
m := Migrator(keeper)
cfg.RegisterMigration(types.ModuleName, 1, m.Migrate1)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other idea is just to make this functions directly:

cfg.RegisterMigration(types.ModuleName, 1,
    func (ctx) {return v42.MigrateStore(ctx, keeper.storeKey)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

func (ctx) {return v42.MigrateStore(ctx, keeper.storeKey)

This would be ideal, but we can't access private keeper.storeKey there.

OK for the Migrator way.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loading