Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: consensus node panics in concurrent map read-writes #2840

Open
4 tasks
Bidon15 opened this issue Nov 14, 2023 · 5 comments
Open
4 tasks

bug: consensus node panics in concurrent map read-writes #2840

Bidon15 opened this issue Nov 14, 2023 · 5 comments
Labels
bug Something isn't working WS: Maintenance 🔧 includes bugs, refactors, flakes, and tech debt etc

Comments

@Bidon15
Copy link
Member

Bidon15 commented Nov 14, 2023

Summary of Bug

Happened on mainnet for devops infra

RAM consumption was 8Gib which was fine

Version

v1.3.0

Steps to Reproduce

run a node for a while

https://celestia.grafana.net/goto/ath5JDSIg?orgId=1



2023-11-14 08:32:47.0697:32AM INF received complete proposal block hash=48EA0DD7CFEB4EABD956B14B159D20170FCD4402583B2B31CBCBE45ACDC17B0A height=101331 module=consensus | 2023-11-14 08:32:47.069 | 7:32AM INF received complete proposal block hash=48EA0DD7CFEB4EABD956B14B159D20170FCD4402583B2B31CBCBE45ACDC17B0A height=101331 module=consensus |  
-- | -- | -- | --
2023-11-14 08:32:47.069 | 7:32AM INF received complete proposal block hash=48EA0DD7CFEB4EABD956B14B159D20170FCD4402583B2B31CBCBE45ACDC17B0A height=101331 module=consensus |  
2023-11-14 08:32:47.6027:32AM INF finalizing commit of block hash={} height=101331 module=consensus num_txs=1 root=B9CFF4FA62EB8B44DEE706C505A064C5056E02574D3DC06B0013F4981C67BAAF2023-11-14 08:32:47.6377:32AM INF minted coins from module account amount=29840334utia from=mint module=x/bank2023-11-14 08:32:47.6517:32AM INF executed block height=101331 module=state num_invalid_txs=0 num_valid_txs=12023-11-14 08:32:47.781fatal error: concurrent map iteration and map write2023-11-14 08:32:47.7852023-11-14 08:32:47.785goroutine 15580584 [running]:2023-11-14 08:32:47.785github.com/cosmos/iavl.NewUnsavedFastIterator({0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1, 0xc00098f600, 0xc0526eede0, 0xc0526eee70)2023-11-14 08:32:47.785	/go/pkg/mod/github.com/cosmos/[email protected]/unsaved_fast_iterator.go:55 +0x2192023-11-14 08:32:47.785github.com/cosmos/iavl.(*MutableTree).Iterator(0xc0019bcf50, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x60?)2023-11-14 08:32:47.785	/go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:216 +0x1052023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/iavl.(*Store).Iterator(0xc111643f80?, {0xc0474ca4e0?, 0x20?, 0xc111643fc0?}, {0xc0474ca800?, 0x20?, 0xc03f3f4000?})2023-11-14 08:32:47.785	/go/pkg/mod/github.com/celestiaorg/[email protected]/store/iavl/store.go:249 +0x2b2023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc0c107a680, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1)2023-11-14 08:32:47.785	/go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:185 +0x11d2023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0x3?, {0xc0474ca4e0?, 0x1e?, 0x1e?}, {0xc0474ca800?, 0x64?, 0x64?})2023-11-14 08:32:47.785	/go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:170 +0x252023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc03ffdb480, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1)2023-11-14 08:32:47.785	/go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:185 +0x11d2023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc111643f20?, {0xc0474ca4e0?, 0x20?, 0xc111643f40?}, {0xc0474ca800?, 0x20?, 0xc111643f60?})2023-11-14 08:32:47.785	/go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:170 +0x252023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc03520e1c0, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1)2023-11-14 08:32:47.785	/go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:185 +0x11d2023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc000581000?, {0xc0474ca4e0?, 0x0?, 0xc14afe90e0?}, {0xc0474ca800?, 0x20?, 0x0?})2023-11-14 08:32:47.785	/go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:170 +0x252023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).iterator(0xc0d3f5dec0, {0xc0474ca4e0?, 0x64?, 0x15?}, {0xc0474ca800?, 0x20?, 0xd38b010000000000?}, 0x0?)2023-11-14 08:32:47.785	/go/pkg/mod/github.com/celestiaorg/[email protected]/store/gaskv/store.go:98 +0x482023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Iterator(0xc021cb62e8?, {0xc0474ca4e0?, 0x3045a50?, 0xc03520e240?}, {0xc0474ca800?, 0x1?, 0xc0776c68a0?})2023-11-14 08:32:47.785	/go/pkg/mod/github.com/celestiaorg/[email protected]/store/gaskv/store.go:74 +0x252023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.IterateValidatorSlashEventsBetween({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...)2023-11-14 08:32:47.785	/go/pkg/mod/github.com/celestiaorg/[email protected]/x/distribution/keeper/store.go:337 +0x1472023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.CalculateDelegationRewards({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...)2023-11-14 08:32:47.785	/go/pkg/mod/github.com/celestiaorg/[email protected]/x/distribution/keeper/delegation.go:80 +0x2d92023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...) |   |   | 2023-11-14 08:32:47.602 | 7:32AM INF finalizing commit of block hash={} height=101331 module=consensus num_txs=1 root=B9CFF4FA62EB8B44DEE706C505A064C5056E02574D3DC06B0013F4981C67BAAF |   |   |   | 2023-11-14 08:32:47.637 | 7:32AM INF minted coins from module account amount=29840334utia from=mint module=x/bank |   |   |   | 2023-11-14 08:32:47.651 | 7:32AM INF executed block height=101331 module=state num_invalid_txs=0 num_valid_txs=1 |   |   |   | 2023-11-14 08:32:47.781 | fatal error: concurrent map iteration and map write |   |   |   | 2023-11-14 08:32:47.785 |   |   |   |   | 2023-11-14 08:32:47.785 | goroutine 15580584 [running]: |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/iavl.NewUnsavedFastIterator({0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1, 0xc00098f600, 0xc0526eede0, 0xc0526eee70) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/cosmos/[email protected]/unsaved_fast_iterator.go:55 +0x219 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/iavl.(*MutableTree).Iterator(0xc0019bcf50, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x60?) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:216 +0x105 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/iavl.(*Store).Iterator(0xc111643f80?, {0xc0474ca4e0?, 0x20?, 0xc111643fc0?}, {0xc0474ca800?, 0x20?, 0xc03f3f4000?}) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/iavl/store.go:249 +0x2b |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc0c107a680, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:185 +0x11d |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0x3?, {0xc0474ca4e0?, 0x1e?, 0x1e?}, {0xc0474ca800?, 0x64?, 0x64?}) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:170 +0x25 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc03ffdb480, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:185 +0x11d |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc111643f20?, {0xc0474ca4e0?, 0x20?, 0xc111643f40?}, {0xc0474ca800?, 0x20?, 0xc111643f60?}) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:170 +0x25 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc03520e1c0, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:185 +0x11d |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc000581000?, {0xc0474ca4e0?, 0x0?, 0xc14afe90e0?}, {0xc0474ca800?, 0x20?, 0x0?}) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:170 +0x25 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).iterator(0xc0d3f5dec0, {0xc0474ca4e0?, 0x64?, 0x15?}, {0xc0474ca800?, 0x20?, 0xd38b010000000000?}, 0x0?) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/gaskv/store.go:98 +0x48 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Iterator(0xc021cb62e8?, {0xc0474ca4e0?, 0x3045a50?, 0xc03520e240?}, {0xc0474ca800?, 0x1?, 0xc0776c68a0?}) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/gaskv/store.go:74 +0x25 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.IterateValidatorSlashEventsBetween({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/x/distribution/keeper/store.go:337 +0x147 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.CalculateDelegationRewards({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/x/distribution/keeper/delegation.go:80 +0x2d9 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...)
  |   | 2023-11-14 08:32:47.602 | 7:32AM INF finalizing commit of block hash={} height=101331 module=consensus num_txs=1 root=B9CFF4FA62EB8B44DEE706C505A064C5056E02574D3DC06B0013F4981C67BAAF |  
  |   | 2023-11-14 08:32:47.637 | 7:32AM INF minted coins from module account amount=29840334utia from=mint module=x/bank |  
  |   | 2023-11-14 08:32:47.651 | 7:32AM INF executed block height=101331 module=state num_invalid_txs=0 num_valid_txs=1 |  
  |   | 2023-11-14 08:32:47.781 | fatal error: concurrent map iteration and map write |  
  |   | 2023-11-14 08:32:47.785 |   |  
  |   | 2023-11-14 08:32:47.785 | goroutine 15580584 [running]: |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/iavl.NewUnsavedFastIterator({0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1, 0xc00098f600, 0xc0526eede0, 0xc0526eee70) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/cosmos/[email protected]/unsaved_fast_iterator.go:55 +0x219 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/iavl.(*MutableTree).Iterator(0xc0019bcf50, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x60?) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:216 +0x105 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/iavl.(*Store).Iterator(0xc111643f80?, {0xc0474ca4e0?, 0x20?, 0xc111643fc0?}, {0xc0474ca800?, 0x20?, 0xc03f3f4000?}) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/iavl/store.go:249 +0x2b |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc0c107a680, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:185 +0x11d |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0x3?, {0xc0474ca4e0?, 0x1e?, 0x1e?}, {0xc0474ca800?, 0x64?, 0x64?}) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:170 +0x25 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc03ffdb480, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:185 +0x11d |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc111643f20?, {0xc0474ca4e0?, 0x20?, 0xc111643f40?}, {0xc0474ca800?, 0x20?, 0xc111643f60?}) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:170 +0x25 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc03520e1c0, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:185 +0x11d |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc000581000?, {0xc0474ca4e0?, 0x0?, 0xc14afe90e0?}, {0xc0474ca800?, 0x20?, 0x0?}) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/cachekv/store.go:170 +0x25 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).iterator(0xc0d3f5dec0, {0xc0474ca4e0?, 0x64?, 0x15?}, {0xc0474ca800?, 0x20?, 0xd38b010000000000?}, 0x0?) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/gaskv/store.go:98 +0x48 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Iterator(0xc021cb62e8?, {0xc0474ca4e0?, 0x3045a50?, 0xc03520e240?}, {0xc0474ca800?, 0x1?, 0xc0776c68a0?}) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/store/gaskv/store.go:74 +0x25 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.IterateValidatorSlashEventsBetween({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/x/distribution/keeper/store.go:337 +0x147 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.CalculateDelegationRewards({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/[email protected]/x/distribution/keeper/delegation.go:80 +0x2d9 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...)

For Admin Use

  • Not duplicate issue
  • Appropriate labels applied
  • Appropriate contributors tagged
  • Contributor assigned/self-assigned
@Bidon15 Bidon15 added the bug Something isn't working label Nov 14, 2023
@evan-forbes
Copy link
Member

to add more context from a sync conversation, this seems to occur more frequently the more a node reads from state.

@evan-forbes
Copy link
Member

evan-forbes commented Nov 15, 2023 •
edited
Loading

seems related to cosmos/cosmos-sdk#10568

we might actually need to go in and add a mutex somewhere in the read path

I'm curious if this error is always with the distribution keeper or do we see it elsewhere as well?

@cmwaters
Copy link
Contributor

cmwaters commented Dec 5, 2023

seems related to cosmos/cosmos-sdk#10568

Are you sure? It looks like this a concurrency problem in the IAVL not the gRPC server

@evan-forbes
Copy link
Member

evan-forbes commented Dec 5, 2023 •
edited
Loading

Are you sure? It looks like this a concurrency problem in the IAVL not the gRPC server

yeah I think so, the title of that issue is misleading. its encountering a similar stack trace both from iavl. both scenario's are using the gprc server to read from iavl

to add more context from a sync conversation, this seems to occur more frequently the more a node reads from state.

@evan-forbes
Copy link
Member

cosmos/cosmos-sdk#10568 (comment)

@evan-forbes evan-forbes added WS: Maintenance 🔧 includes bugs, refactors, flakes, and tech debt etc needs:triage labels May 17, 2024
@evan-forbes evan-forbes mentioned this issue May 22, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working WS: Maintenance 🔧 includes bugs, refactors, flakes, and tech debt etc
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Bidon15 @cmwaters @evan-forbes