Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fatal error: concurrent map iteration and map write #5001

Closed
heitorPB opened this issue Apr 25, 2023 · 5 comments
Closed

fatal error: concurrent map iteration and map write #5001

heitorPB opened this issue Apr 25, 2023 · 5 comments

Comments

@heitorPB
Copy link

System information

Osmosis version: 15.0.0
OS & Version: Ubuntu 22.04
Commit hash:

Expected behaviour

No crash.

Actual behaviour

Osmosis crashing.

Steps to reproduce the behaviour

I could not identify a way to make the system crash, but our service crashed a few times per day. The first line of the error in the logs was always fatal error: concurrent map iteration and map write.

Backtrace

The first "few" lines of our error logs:

6:31PM INF received complete proposal block hash=172527154A276A97AD36371C5C8EA881EB6F7393080AEFFAAD8DEDF926F350BD height=10264202 module=consensus
6:31PM INF finalizing commit of block hash={} height=10264202 module=consensus num_txs=3 root=1997EAF5CEF0831FC6C14E01A0A94A0A751B260CA60F17E292B7CE860F7A0A02
6:31PM INF executed block height=10264202 module=state num_invalid_txs=0 num_valid_txs=3
fatal error: concurrent map iteration and map write

goroutine 636538 [running, locked to thread]:
github.com/cosmos/iavl.NewUnsavedFastIterator({0xc0efc224e0, 0x2e, 0x2e}, {0xc0efc22510, 0x25, 0x25}, 0x1, 0xc00103ec80, 0xc0f78e3b60, 0xc0f78e3b90)
        github.com/cosmos/[email protected]/unsaved_fast_iterator.go:55 +0x227
github.com/cosmos/iavl.(*MutableTree).Iterator(0xc00414c0a0, {0xc0efc224e0, 0x2e, 0x2e}, {0xc0efc22510, 0x25, 0x25}, 0xd3?)
        github.com/cosmos/[email protected]/mutable_tree.go:216 +0x10c
github.com/cosmos/cosmos-sdk/store/iavl.(*Store).Iterator(0x45e8d6?, {0xc0efc224e0?, 0xc000076a00?, 0xc0f021ce88?}, {0xc0efc22510?, 0xc09af98ea0?, 0xc0f021cdb0?})
        github.com/cosmos/[email protected]/store/iavl/store.go:249 +0x31
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc06bd84280, {0xc0efc224e0, 0x2e, 0x2e}, {0xc0efc22510, 0x25, 0x25}, 0x1)
        github.com/cosmos/[email protected]/store/cachekv/store.go:189 +0x126
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc0ef86c760?, {0xc0efc224e0?, 0x42f69f?, 0x425192?}, {0xc0efc22510?, 0x4314c6?, 0x24c4c20?})
        github.com/cosmos/[email protected]/store/cachekv/store.go:174 +0x29
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc0ed648c80, {0xc0efc224e0, 0x2e, 0x2e}, {0xc0efc22510, 0x25, 0x25}, 0x1)
        github.com/cosmos/[email protected]/store/cachekv/store.go:189 +0x126
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc005091b60?, {0xc0efc224e0?, 0xee0d66?, 0x203043?}, {0xc0efc22510?, 0x14?, 0x0?})
        github.com/cosmos/[email protected]/store/cachekv/store.go:174 +0x29
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc0ed666380, {0xc0efc224e0, 0x2e, 0x2e}, {0xc0efc22510, 0x25, 0x25}, 0x1)
        github.com/cosmos/[email protected]/store/cachekv/store.go:189 +0x126
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0x0?, {0xc0efc224e0?, 0xc02a01d0d0?, 0x7fbf3e9c80b8?}, {0xc0efc22510?, 0x7fbf8dafa688?, 0xd0?})
        github.com/cosmos/[email protected]/store/cachekv/store.go:174 +0x29
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc0ef952980, {0xc0efc224e0, 0x2e, 0x2e}, {0xc0efc22510, 0x25, 0x25}, 0x1)
        github.com/cosmos/[email protected]/store/cachekv/store.go:189 +0x126
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0x0?, {0xc0efc224e0?, 0xc0f021d248?, 0x42f9e7?}, {0xc0efc22510?, 0x42f69f?, 0x42f69f?})
        github.com/cosmos/[email protected]/store/cachekv/store.go:174 +0x29
github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).iterator(0xc106214c00, {0xc0efc224e0?, 0x0?, 0xc0f021d300?}, {0xc0efc22510?, 0x21?, 0x1?}, 0x10?)
        github.com/cosmos/[email protected]/store/gaskv/store.go:107 +0x56
github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Iterator(0x4265ff?, {0xc0efc224e0?, 0xc10d6af4c0?, 0xc0ef911440?}, {0xc0efc22510?, 0x100c0ef952980?, 0x7fbf3e2f11b0?})
        github.com/cosmos/[email protected]/store/gaskv/store.go:78 +0x29
github.com/cosmos/cosmos-sdk/store/prefix.Store.Iterator({{0x2eedfb8, 0xc106214c00}, {0xc0ef911440, 0x21, 0x30}}, {0xc0a86377d0, 0xd, 0xd}, {0xc0a86377cc, 0x4, ...})
        github.com/cosmos/[email protected]/store/prefix/store.go:101 +0x239
github.com/CosmWasm/wasmvm/internal/api.cScan(0xc0ed7ff698, 0x140617c?, 0x7fbdc37bc298, {0x0?, 0x7fbf41f95f40?, 0x7fbf4197a770?}, {0x58?, 0x7fbf4197a890?, 0xc21881b040?}, 0x1, ...)
        github.com/CosmWasm/[email protected]/internal/api/callbacks.go:280 +0x236
github.com/CosmWasm/wasmvm/internal/api._C2func_execute(0x7fbf65efa820, {0x0, 0xc0ef8c91c0, 0x20}, {0x0, 0xc04da65740, 0xbf}, {0x0, 0xc113db05a0, 0x57}, ...)
        _cgo_gotypes.go:294 +0x85
github.com/CosmWasm/wasmvm/internal/api.Execute.func1({0xc118f295d0?}, {0x28?, 0xc0ef8c91c0?, 0xc0f021db08?}, {0x8c?, 0xc04da65740?, 0x24f1bc0?}, {0x0, 0xc113db05a0, 0x57}, ...)
        github.com/CosmWasm/[email protected]/internal/api/lib.go:211 +0x247
github.com/CosmWasm/wasmvm/internal/api.Execute({0x20?}, {0xc0ef8c91c0?, 0xc0?, 0x203042?}, {0xc04da65740?, 0x203042?, 0xc10829d7e0?}, {0xc113db05a0, 0x57, 0x60}, ...)
        github.com/CosmWasm/[email protected]/internal/api/lib.go:211 +0x6f1
github.com/CosmWasm/wasmvm.(*VM).Execute(0xc00150e7a0, {0xc0ef8c91c0, 0x20, 0x20}, {{0x9c9e89, 0x1758f2761f7950c4, {0xc07abf2260, 0xb}}, 0x0, {{0xc00ca8d0c0, ...}}}, ...)
        github.com/CosmWasm/[email protected]/lib.go:186 +0x2e5
github.com/CosmWasm/wasmd/x/wasm/keeper.Keeper.execute({{0x2ebf738, 0xc00150e350}, {0x2ef5a40, 0xc0002b0c70}, {0x2ec6b30, 0xc00026a6c0}, {0x2ebae20, 0xc00150e7c0}, {0x2eb7c00, 0xc0002e8bf8}, ...}, ...)
        github.com/CosmWasm/[email protected]/x/wasm/keeper/keeper.go:425 +0xa3d
github.com/CosmWasm/wasmd/x/wasm/keeper.PermissionedKeeper.Execute(...)
        github.com/CosmWasm/[email protected]/x/wasm/keeper/contract_keeper.go:95
github.com/CosmWasm/wasmd/x/wasm/keeper.msgServer.ExecuteContract({{0x2ef0878?, 0xc00130b920?}}, {0x2edb398, 0xc118f294a0}, 0xc0ef8ccfa0)
        github.com/CosmWasm/[email protected]/x/wasm/keeper/msg_server.go:139 +0x742
github.com/CosmWasm/wasmd/x/wasm/types._Msg_ExecuteContract_Handler.func1({0x2edb398, 0xc118f294a0}, {0x2768740?, 0xc0ef8ccfa0})
        github.com/CosmWasm/[email protected]/x/wasm/types/tx.pb.go:1005 +0x78
github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2.1({0x2edb398, 0xc118f29440}, {0x4dd986?, 0x42d0eb?}, 0x27c5e40?, 0xc0ed7ff680)
        github.com/cosmos/[email protected]/baseapp/msg_service_router.go:113 +0xd2
github.com/CosmWasm/wasmd/x/wasm/types._Msg_ExecuteContract_Handler({0x264bdc0?, 0xc00150fe20}, {0x2edb398, 0xc118f29440}, 0x2c158b0, 0xc10829d660)
        github.com/CosmWasm/[email protected]/x/wasm/types/tx.pb.go:1007 +0x138
github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2({{0x2edb328, 0xc000056050}, {0x2ef1e80, 0xc0ef952ac0}, {{0xb, 0x10}, {0xc07abf2260, 0xb}, 0x9c9e89, {0x39e016c4, ...}, ...}, ...}, ...)
        github.com/cosmos/[email protected]/baseapp/msg_service_router.go:135 +0x4ad
github.com/CosmWasm/wasmd/x/wasm/keeper.SDKMessageHandler.handleSdkMessage({{_, _}, {_, _}}, {{0x2edb328, 0xc000056050}, {0x2ef1e80, 0xc0ef952ac0}, {{0xb, 0x10}, ...}, ...}, ...)
        github.com/CosmWasm/[email protected]/x/wasm/keeper/handler_plugin.go:98 +0x1ac
github.com/CosmWasm/wasmd/x/wasm/keeper.SDKMessageHandler.DispatchMsg({{_, _}, {_, _}}, {{0x2edb328, 0xc000056050}, {0x2ef1e80, 0xc0ef952ac0}, {{0xb, 0x10}, ...}, ...}, ...)
        github.com/CosmWasm/[email protected]/x/wasm/keeper/handler_plugin.go:68 +0x29b
github.com/CosmWasm/wasmd/x/wasm/keeper.MessageHandlerChain.DispatchMsg({{_, _, _}}, {{0x2edb328, 0xc000056050}, {0x2ef1e80, 0xc0ef952ac0}, {{0xb, 0x10}, {0xc07abf2260, ...}, ...}, ...}, ...)
        github.com/CosmWasm/[email protected]/x/wasm/keeper/handler_plugin.go:130 +0x168
github.com/osmosis-labs/osmosis/v15/wasmbinding.(*CustomMessenger).DispatchMsg(_, {{0x2edb328, 0xc000056050}, {0x2ef1e80, 0xc0ef952ac0}, {{0xb, 0x10}, {0xc07abf2260, 0xb}, 0x9c9e89, ...}, ...}, ...)
        github.com/osmosis-labs/osmosis/v15/wasmbinding/message_plugin.go:59 +0x19e
github.com/CosmWasm/wasmd/x/wasm/keeper.MessageDispatcher.DispatchSubmessages({{_, _}, {_, _}}, {{0x2edb328, 0xc000056050}, {0x2ef1e80, 0xc0ed666440}, {{0xb, 0x10}, ...}, ...}, ...)
        github.com/CosmWasm/[email protected]/x/wasm/keeper/msg_dispatcher.go:103 +0x506
github.com/CosmWasm/wasmd/x/wasm/keeper.DefaultWasmVMContractResponseHandler.Handle({{_, _}}, {{0x2edb328, 0xc000056050}, {0x2ef1e80, 0xc0ed666440}, {{0xb, 0x10}, {0xc07abf2260, 0xb}, ...}, ...}, ...)
        github.com/CosmWasm/[email protected]/x/wasm/keeper/keeper.go:1220 +0xa2
github.com/CosmWasm/wasmd/x/wasm/keeper.(*Keeper).handleContractResponse(_, {{0x2edb328, 0xc000056050}, {0x2ef1e80, 0xc0ed666440}, {{0xb, 0x10}, {0xc07abf2260, 0xb}, 0x9c9e89, ...}, ...}, ...)
        github.com/CosmWasm/[email protected]/x/wasm/keeper/keeper.go:1009 +0x4b0
github.com/CosmWasm/wasmd/x/wasm/keeper.Keeper.execute({{0x2ebf738, 0xc00150e350}, {0x2ef5a40, 0xc0002b0c70}, {0x2ec6b30, 0xc00026a6c0}, {0x2ebae20, 0xc00150e7c0}, {0x2eb7c00, 0xc0002e8bf8}, ...}, ...)
        github.com/CosmWasm/[email protected]/x/wasm/keeper/keeper.go:436 +0xf5b
github.com/CosmWasm/wasmd/x/wasm/keeper.PermissionedKeeper.Execute(...)
        github.com/CosmWasm/[email protected]/x/wasm/keeper/contract_keeper.go:95
github.com/CosmWasm/wasmd/x/wasm/keeper.msgServer.ExecuteContract({{0x2ef0878?, 0xc00130b920?}}, {0x2edb398, 0xc112c39d10}, 0xc0ea363bd0)
        github.com/CosmWasm/[email protected]/x/wasm/keeper/msg_server.go:139 +0x742
github.com/CosmWasm/wasmd/x/wasm/types._Msg_ExecuteContract_Handler.func1({0x2edb398, 0xc112c39d10}, {0x2768740?, 0xc0ea363bd0})
        github.com/CosmWasm/[email protected]/x/wasm/types/tx.pb.go:1005 +0x78
github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2.1({0x2edb398, 0xc112c39cb0}, {0x4dd986?, 0x42d0eb?}, 0x27c5e40?, 0xc0ea371b60)
        github.com/cosmos/[email protected]/baseapp/msg_service_router.go:113 +0xd2
github.com/CosmWasm/wasmd/x/wasm/types._Msg_ExecuteContract_Handler({0x264bdc0?, 0xc00150fe20}, {0x2edb398, 0xc112c39cb0}, 0x2c158b0, 0xc10a21c380)
        github.com/CosmWasm/[email protected]/x/wasm/types/tx.pb.go:1007 +0x138
github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2({{0x2edb328, 0xc000056050}, {0x2ef1e80, 0xc0ed666440}, {{0xb, 0x10}, {0xc07abf2260, 0xb}, 0x9c9e89, {0x39e016c4, ...}, ...}, ...}, ...)
        github.com/cosmos/[email protected]/baseapp/msg_service_router.go:135 +0x4ad
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x2edb328, 0xc000056050}, {0x2ef1e80, 0xc0ed666440}, {{0xb, 0x10}, {0xc07abf2260, 0xb}, 0x9c9e89, ...}, ...}, ...)
        github.com/cosmos/[email protected]/baseapp/baseapp.go:801 +0x5a5
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0xc0002a96c0, 0x2, {0xc1090558c0, 0x231, 0x240})
        github.com/cosmos/[email protected]/baseapp/baseapp.go:743 +0xc05
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Simulate(0x42f9e7?, {0xc1090558c0?, 0x2602160?, 0xc0ed648701?})
        github.com/cosmos/[email protected]/baseapp/test_helpers.go:23 +0x2c
github.com/cosmos/cosmos-sdk/x/auth/tx.txServer.Simulate({{{0x0, 0x0, 0x0}, {0x2efffd0, 0xc00ac7d1a0}, {0x0, 0x0}, {0x2ee4070, 0xc0014a8dc0}, 0x0, ...}, ...}, ...)
        github.com/cosmos/[email protected]/x/auth/tx/service.go:117 +0x129
github.com/cosmos/cosmos-sdk/types/tx._Service_Simulate_Handler.func1({0x2edb398, 0xc112c30060}, {0x270dd40?, 0xc2125451a0})
        github.com/cosmos/[email protected]/types/tx/service.pb.go:752 +0x78
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func1({0x2edb398, 0xc11818ba70}, {0x270dd40, 0xc2125451a0}, 0xc2125451c0?, 0xc0ea370708)
        github.com/cosmos/[email protected]/baseapp/grpcserver.go:66 +0x3d5
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2edb398?, 0xc11818ba70?}, {0x270dd40?, 0xc2125451a0?})

...

This is the longest stack trace I've ever seen (~50k lines). I can send you the full logs if you want.
@github-project-automation github-project-automation bot moved this to Needs Triage 🔍 in Osmosis Chain Development Apr 25, 2023
@p0mvn
Copy link
Member

p0mvn commented Apr 26, 2023

Hey @heitorPB . Thanks for the issue. Do you have concurrency = false in app.toml?

@heitorPB
Copy link
Author

Yes, we have concurrency = false in there.

@p0mvn
Copy link
Member

p0mvn commented Apr 26, 2023

Have you tried upgrading to v15.1.0?
https://github.com/osmosis-labs/osmosis/releases/tag/v15.1.0

There was a security advisory and a wasmvm upgrade. I'm seeing wasmvm in the call stack so might be related

@heitorPB
Copy link
Author

Oh, nice. It's on my TODO list but I haven't had time to update it yet. Will update today.

Do you want more lines of the errors? It is quite long, but I can find a way to send you.

@mattverse mattverse mentioned this issue Jul 31, 2023
Closed
@ValarDragon
Copy link
Member

This should now be fixed

@github-project-automation github-project-automation bot moved this from Needs Triage 🔍 to Done ✅ in Osmosis Chain Development Mar 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
Osmosis Chain Development
Archived in project
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ValarDragon @heitorPB @p0mvn