Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick upgrade-14 cosmos changes #8869

Merged
merged 70 commits into from
Feb 19, 2024

Conversation

mhofman
Copy link
Member

@mhofman mhofman commented Feb 7, 2024

Description

These are almost all the changes to golang/cosmos into upgrade-14. Most notably, the core of the split-vm work is still excluded, but its context changes are included, which causes some conflict resolution in a few places.

Compare golang/cosmos from this branch with master: master...mhofman/cherry-pick-ibc-updates

Constructed from the following rebase-todo:

# Branch zmanian-zaki-ledger
label base-zmanian-zaki-ledger
pick e955047f6 Fix for MacOS Sonoma Ledger detection issues
label zmanian-zaki-ledger
reset base-zmanian-zaki-ledger
merge -C 780a4613a zmanian-zaki-ledger # Merge pull request #8462 from zmanian/zaki/ledger

# Branch Agoric-8767-ibc-4-proto
label base-Agoric-8767-ibc-4-proto
pick 9cdfaefcf fix: update protobuf download and generation for ibc-go v4
label Agoric-8767-ibc-4-proto
reset base-Agoric-8767-ibc-4-proto
merge -C e0447b837 Agoric-8767-ibc-4-proto # Merge pull request #8769 from Agoric/8767-ibc-4-proto

# Branch Agoric-8224-ibc5
label base-Agoric-8224-ibc5
pick 302f6a02e chore: update proto dependencies and scripts to produce/use them
pick 045d051c7 chore: update ibc-go import versions
pick 5f8deb41b chore: update for cosmos-sdk API changes
pick 20a89f060 fix: unwrap account keeper for app module
pick 680e5274f build: use temporary tag for cosmos-sdk
pick b6abb10ba build: clean up no-op go.mod replace directive
pick cb12a5342 feat: expose node service to retrieve operator config
pick 38f6c3f8d fix: pick up snapshot initiation fix
pick a2bfb34d1 fix: avoid broken goleveldb
pick d2602b3be fix: exempt more 3rd-party protos from link check
pick 0345c4d20 chore: use more cosmos-sdk replace directives and align formatting
pick 34d505671 fix: govulncheck updates
pick 2e4d8d33a feat: tolerate missing files in gaia 3-way diff
pick 6e290612d docs: typo
pick 5a02d321c build: update cosmos-sdk fork to final tag
label Agoric-8224-ibc5
reset base-Agoric-8224-ibc5
merge -C 64cb69fcc Agoric-8224-ibc5 # Merge pull request #8791 from Agoric/8224-ibc5

# Branch Agoric-8224-ibc6
label base-Agoric-8224-ibc6
pick fa4695dce feat: upgrade ibc-go to v6.2.1
pick 9e12a7ce3 docs: update note on unwrapping of account keeper
label Agoric-8224-ibc6
reset base-Agoric-8224-ibc6
merge -C 7ffae88ae Agoric-8224-ibc6 # Merge pull request #8865 from Agoric/8224-ibc6

# Branch Agoric-mfig-silence-golint-errors
label base-Agoric-mfig-silence-golint-errors
pick 9f0250878 ci(golangci-lint): ignore deprecations until we have none
pick c226455af chore(cosmos): panic instead of ignoring export errors
# pick d5484e3ff ci(integration): fix `agd` wrapper and `Dockerfile.sdk`
label Agoric-mfig-silence-golint-errors
reset base-Agoric-mfig-silence-golint-errors
merge -C d4925683c Agoric-mfig-silence-golint-errors # Merge pull request #8866 from Agoric/mfig-silence-golint-errors

# Branch Agoric-mhofman-8887-fix-governance
label base-Agoric-mhofman-8887-fix-governance
pick 4887ecb7c build(cosmos): update cosmos-sdk with gov fixes
pick 7cfae88a1 fix(cosmos): add action context to core evals
pick 003f0c223 Revert "fix(cosmos): don't log expected missing tx context of CORE_EVAL"
pick d8efae59e chore: handle new agd query gov proposals response
pick ac804f448 chore: handle failed proposals in test script
pick d13427ce5 chore: debug faucet script
# pick 229b36ff1 chore(a3p): bump synthetic-chain to 0.0.6-4
label Agoric-mhofman-8887-fix-governance
reset base-Agoric-mhofman-8887-fix-governance
merge -C 61af4a89c Agoric-mhofman-8887-fix-governance # Merge pull request #8902 from Agoric/mhofman/8887-fix-governance

# Branch Agoric-gibson-7581-rpc-capdata-formatting
label base-Agoric-gibson-7581-rpc-capdata-formatting
pick 8bdd7cb91 feat(cosmos): Add a vstorage package for decoding CapData
edit 8943f2f85 feat(cosmos): Add a "CapData" vstorage RPC endpoint # fix grpc_query.go HasValue merge order "conflict"
pick b68a5baa7 chore(cosmos): Regenerate protobuf code
pick 22c7d49e4 fixup! feat(cosmos): Add a "CapData" vstorage RPC endpoint
pick 935650975 chore(cosmos): Improve vstorage capdata error messages
pick 36b921d58 test(cosmos): Test vstorage capdata
edit a7f17febb test(cosmos): Test gRPC endpoint "/agoric.vstorage.Query/CapData" # fix keeper_grpc_test.go agoric.NewKVEntry* merge order "conflict"
pick e4bfad38b chore(cosmos): Improve vstorage/keeper/grpc_query.go doc comments
pick dff7e9329 chore(cosmos): Add vstorage README
pick b4fdcc900 chore(cosmos): Avoid special treatment of HTML characters in vstorage JSON encoding
pick e2cbffacc feat(cosmos): Always include alleged name and slot id in vstorage CapData remotable representations
label Agoric-gibson-7581-rpc-capdata-formatting
reset base-Agoric-gibson-7581-rpc-capdata-formatting
merge -C 91f5e93d5 Agoric-gibson-7581-rpc-capdata-formatting # Merge pull request #8056 from Agoric/gibson-7581-rpc-capdata-formatting

# Branch Agoric-gibson-8056-followup
label base-Agoric-gibson-8056-followup
pick f1cb453ae chore(cosmos): Update doc comments for accuracy
pick 749a73719 docs(cosmos): Fix hyperlinks
label Agoric-gibson-8056-followup
reset base-Agoric-gibson-8056-followup
merge -C d55dda861 Agoric-gibson-8056-followup # Merge pull request #8119 from Agoric/gibson-8056-followup

# Branch Agoric-gibson-8056-followup-2
label base-Agoric-gibson-8056-followup-2
pick 68f158677 docs(cosmos/x/vstorage): Add REAME link to vstorage/query.proto
label Agoric-gibson-8056-followup-2
reset base-Agoric-gibson-8056-followup-2
merge -C 4264b77fd Agoric-gibson-8056-followup-2 # Merge pull request #8130 from Agoric/gibson-8056-followup-2

# Branch Agoric-gibson-8056-followup-3
label base-Agoric-gibson-8056-followup-3
pick 8f7e4b4d2 docs(cosmos/x/vstorage): Add README section for "REST" API interface
pick 7a70695b6 docs(cosmos/x/vstorage): Add README section for CLI
label Agoric-gibson-8056-followup-3
reset base-Agoric-gibson-8056-followup-3
merge -C b4ce02d92 Agoric-gibson-8056-followup-3 # Merge pull request #8134 from Agoric/gibson-8056-followup-3

# Branch Agoric-gibson-2023-10-vstorage-protobuf-example
label base-Agoric-gibson-2023-10-vstorage-protobuf-example
pick a213b41a2 docs(vstorage): Add an example external protobuf interface interaction
label Agoric-gibson-2023-10-vstorage-protobuf-example
reset base-Agoric-gibson-2023-10-vstorage-protobuf-example
merge -C 9d19b124a Agoric-gibson-2023-10-vstorage-protobuf-example # Merge pull request #8475 from Agoric/gibson-2023-10-vstorage-protobuf-example

# Branch Agoric-ta-vstorage-cli
label base-Agoric-ta-vstorage-cli
pick 2b56fc663 feat: agd vstorage 'path' for data or children
pick d97935883 docs: recommend 'path' style of vstorage query
label Agoric-ta-vstorage-cli
reset base-Agoric-ta-vstorage-cli
merge -C 624ac4ada Agoric-ta-vstorage-cli # Merge pull request #8637 from Agoric/ta/vstorage-cli

# Pull Request #8674
pick 9380cfe04 chore: Avoid out-of-bounds reads in go-lang code (#8674)

# Pull Request #8677
pick a0d822952 chore(golang): Clarify error messages (#8677)

# Branch Agoric-gibson-2023-12-golang-cli-commands
label base-Agoric-gibson-2023-12-golang-cli-commands
pick f3ca9249d style(golang): Align CLI command fields with upstream recommendations
pick 90f4ba893 chore(agd testnet): Deprecate --v in favor of --validator-count, -n
pick 088764597 chore(x/swingset): Adopt and explain conventions for file/inline CLI arguments
label Agoric-gibson-2023-12-golang-cli-commands
reset base-Agoric-gibson-2023-12-golang-cli-commands
merge -C 853cca53c Agoric-gibson-2023-12-golang-cli-commands # Merge pull request #8678 from Agoric/gibson-2023-12-golang-cli-commands

# Branch Agoric-gibson-gh-8703-followup
label base-Agoric-gibson-gh-8703-followup
pick 89a2d9fcb docs(golang): Improve the description of PopulateAction
pick 7665aafa7 style(golang): Rename PopulateAction variables for clarity
pick ca42f47f3 style(golang): Rename a test helper function for clarity
label Agoric-gibson-gh-8703-followup
reset base-Agoric-gibson-gh-8703-followup
merge -C 121b67747 Agoric-gibson-gh-8703-followup # Merge pull request #8855 from Agoric/gibson-gh-8703-followup

# From pull request #8001
pick fa1754c79 feat(cosmos)!: make vm comms use `context.Context`
pick 158d83156 feat(cosmos): clean up `OnStartHook` and `OnExitHook` signatures

Security Considerations

None specific to this cherry-pick

Scaling Considerations

None specific to the cherry-pick

Documentation Considerations

Release info should make a mention of this notable change

Testing Considerations

TBD @JimLarson

Core eval logic tested in #8942

Upgrade Considerations

Chain software change deployed as part of chain software upgrade

@mhofman
Copy link
Member Author

mhofman commented Feb 7, 2024

@JimLarson are the golint failures expected?

@mhofman mhofman added the force:integration Force integration tests to run on PR label Feb 7, 2024
@mhofman mhofman force-pushed the mhofman/cherry-pick-ibc-updates branch from 04db7e9 to 5c0ff3c Compare February 7, 2024 03:13
@mhofman mhofman marked this pull request as ready for review February 7, 2024 03:13
@mhofman mhofman requested a review from gibson042 February 7, 2024 03:13
@JimLarson
Copy link
Contributor

Yes, the golint failures are expected. See #8866 for a fix from @michaelfig .

@JimLarson
Copy link
Contributor

See also #8803 for a cleanup of the deprecated APIs.

@mhofman
Copy link
Member Author

mhofman commented Feb 7, 2024

Getting the following error in the integration test:

stage-0: chain: 3:24AM INF Connection is closed @ recvRoutine (likely by the other side) conn={"Logger":{}} module=p2p peer={"id":"15e97080abf3b86c1c5170e4c4df65fcc09c1651","ip":"172.17.0.2","port":26656}
stage-0: chain: 3:24AM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"15e97080abf3b86c1c5170e4c4df65fcc09c1651","ip":"172.17.0.2","port":26656}
stage-0: chain: 3:24AM ERR Stopping peer for error err=EOF module=p2p peer={"Data":{},"Logger":{}}
stage-0: chain: 3:24AM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"15e97080abf3b86c1c5170e4c4df65fcc09c1651","ip":"172.17.0.2","port":26656}
stage-0: chain: 3:24AM INF Reconnecting to peer addr={"id":"15e97080abf3b86c1c5170e4c4df65fcc09c1651","ip":"172.17.0.2","port":26656} module=p2p
stage-0: chain: 3:24AM INF Error reconnecting to peer. Trying again addr={"id":"15e97080abf3b86c1c5170e4c4df65fcc09c1651","ip":"172.17.0.2","port":26656} err="dial tcp 172.17.0.2:26656: connect: connection refused" module=p2p tries=0
stage-0: chain: 3:24AM ERR Error dialing peer err="dial tcp 172.17.0.2:26656: connect: connection refused" module=p2p
stage-0: chain: 3:24AM INF Error reconnecting to peer. Trying again addr={"id":"15e97080abf3b86c1c5170e4c4df65fcc09c1651","ip":"172.17.0.2","port":26656} err="dial tcp 172.17.0.2:26656: connect: connection refused" module=p2p tries=1
stage-0: chain: 3:24AM INF Error reconnecting to peer. Trying again addr={"id":"15e97080abf3b86c1c5170e4c4df65fcc09c1651","ip":"172.17.0.2","port":26656} err="dial tcp 172.17.0.2:26656: connect: connection refused" module=p2p tries=2
stage-0: chain: 3:24AM INF sync any module=statesync msg={}
stage-0: chain: 3:24AM INF Discovered new snapshot format=2 hash="\x04L4j\x1b�z\v�1-`�\x1aNP\x1a�x�\x10��,���翘\x1d\x14" height=20 module=statesync
stage-0: chain: 3:24AM INF Error reconnecting to peer. Trying again addr={"id":"15e97080abf3b86c1c5170e4c4df65fcc09c1651","ip":"172.17.0.2","port":26656} err="dial tcp 172.17.0.2:26656: connect: connection refused" module=p2p tries=3
stage-0: chain: 3:25AM INF Error reconnecting to peer. Trying again addr={"id":"15e97080abf3b86c1c5170e4c4df65fcc09c1651","ip":"172.17.0.2","port":26656} err="dial tcp 172.17.0.2:26656: connect: connection refused" module=p2p tries=4
stage-0: chain: 3:25AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=1 numToDial=9
stage-0: chain: 3:25AM INF We need more addresses. Sending pexRequest to random peer module=pex peer={"Data":{},"Logger":{}}
stage-0: chain: 3:25AM INF error from light block request from primary, replacing... error="height requested is too high" height=21 module=light primary={}
stage-0: chain: 3:25AM ERR error on light block request from witness, removing... error="post failed: Post \"[http://172.17.0.2:26657\](http://172.17.0.2:26657/)": dial tcp 172.17.0.2:26657: connect: connection refused" module=light primary={}
stage-0: chain: 3:25AM ERR failed to remove witnesses err="no witnesses connected. please reset light client" module=light witnessesToRemove=[0]
stage-0: chain: 3:25AM INF failed to fetch and verify app hash err="post failed: Post \"[http://172.17.0.2:26657\](http://172.17.0.2:26657/)": dial tcp 172.17.0.2:26657: connect: connection refused" module=statesync
stage-0: chain: 3:25AM INF Snapshot rejected format=2 hash="\x04L4j\x1b�z\v�1-`�\x1aNP\x1a�x�\x10��,���翘\x1d\x14" height=20 module=statesync

It sounds like one of the 2 validator nodes crashed, the chain stopped producing blocks, so when the light client tried to verify the block after the snapshot, it failed because that block didn't exist. Will try to look into the validator node logs to see why it crashed.

@mhofman
Copy link
Member Author

mhofman commented Feb 7, 2024

Yup as I expected, panic while creating the snapshot.

Stack validator1
�[90m3:24AM�[0m �[32mINF�[0m service start �[36mimpl=�[0m"Peer{MConn{172.17.0.1:54300} 79945e03cfc0cdd4f735fb8c2786ec63dd2deeb3 in}" �[36mmodule=�[0mp2p �[36mmsg=�[0m{} �[36mpeer=�[0m{"id":"79945e03cfc0cdd4f735fb8c2786ec63dd2deeb3","ip":"172.17.0.1","port":54300}
�[90m3:24AM�[0m �[32mINF�[0m service start �[36mimpl=�[0mMConn{172.17.0.1:54300} �[36mmodule=�[0mp2p �[36mmsg=�[0m{} �[36mpeer=�[0m{"id":"79945e03cfc0cdd4f735fb8c2786ec63dd2deeb3","ip":"172.17.0.1","port":54300}
runtime: g 77: unexpected return pc for runtime.selectgo called from 0x7fa834917ac4
stack: frame={sp:0xc000503d40, fp:0xc000503e80} stack=[0xc000503800,0xc000504000)
0x000000c000503c40:  0x000000c0001f8780  0x0000000000000060 
0x000000c000503c50:  0x0000000000000058  0x00007fa58eba7440 
0x000000c000503c60:  0x0000000000000000  0x00007fa83002f528 
0x000000c000503c70:  0x00007fa8300aca68  0x0000000000000060 
0x000000c000503c80:  0x000000c00007b800  0x000000c0001f8780 
0x000000c000503c90:  0x000000c000c63860  0x000000c000503cc0 
0x000000c000503ca0:  0x00007fa58c2aeb07 <runtime.newobject+0x0000000000000027>  0xffffffffffffffff 
0x000000c000503cb0:  0x000000000000003c  0x000000c000503d30 
0x000000c000503cc0:  0x00007fa58c311447 <runtime.gcWriteBarrierR9+0x0000000000000007>  0x00007fa58c2dcb12 <runtime.acquireSudog+0x00000000000000b2> 
0x000000c000503cd0:  0x0000000000000000  0x0000000000000000 
0x000000c000503ce0:  0x0000000000000000  0x0000000000000000 
0x000000c000503cf0:  0x0000000000000000  0x000000c0001f86c0 
0x000000c000503d00:  0x000000c0001f86c0  0x000000000000003c 
0x000000c000503d10:  0x000000c000503d30  0x00007fa58c2dc936 <runtime.gopark+0x00000000000000d6> 
0x000000c000503d20:  0x000000000000003c  0x000000c000503e70 
0x000000c000503d30:  0x000000c000503e70  0x00007fa58c2ecffe <runtime.selectgo+0x00000000000007be> 
0x000000c000503d40: <0x000000c000503f28  0x0000000000000002 
0x000000c000503d50:  0x0000000000000000  0x000000c000503ed0 
0x000000c000503d60:  0x0000000000000002  0x0000000000000000 
0x000000c000503d70:  0x0001000100000000  0x0000000000000000 
0x000000c000503d80:  0x0000000000000002  0x0000000000000000 
0x000000c000503d90:  0x0000000000000000  0x0000000000000000 
0x000000c000503da0:  0x0000000000000000  0x0000000000000000 
0x000000c000503db0:  0x0000000000000000  0x0000000000000000 
0x000000c000503dc0:  0x0000000000000000  0x0000000000000001 
0x000000c000503dd0:  0x0000000000000002  0x0000000000000000 
0x000000c000503de0:  0x0000000000000000  0x000000c0001f8760 
0x000000c000503df0:  0x000000c000503ed0  0x000000c000c63860 
0x000000c000503e00:  0x0000000000000000  0x000000c000cefc80 
0x000000c000503e10:  0x000000c000cefc20  0x000000c000cefc80 
0x000000c000503e20:  0x0000000000000000  0x000000c000c638e8 
0x000000c000503e30:  0x000000c000c639a8  0x0000000000000000 
0x000000c000503e40:  0x0000000000000000  0x0000000000000000 
0x000000c000503e50:  0x0000000000000000  0x0000000000000000 
0x000000c000503e60:  0x0000000000000000  0x0000000000000000 
0x000000c000503e70:  0x000000c000503fb8 !0x00007fa834917ac4 
0x000000c000503e80: >0x000000c000503f28  0x0000000000000001 
0x000000c000503e90:  0x000000c000503ebf  0x0000000000000001 
0x000000c000503ea0:  0x000000c000503ef0  0x000000000167b84b 
0x000000c000503eb0:  0x0000000000000000  0x2a00000000000000 
0x000000c000503ec0:  0x000000000532e808  0x0000000000000011 
0x000000c000503ed0:  0x0000000000000000  0x00007fa547ffebf8 
0x000000c000503ee0:  0x0000000000000000  0x000000c0001b4000 
0x000000c000503ef0:  0x000000c0005044d8  0x00007fa834845520 
0x000000c000503f00:  0x0000000000000007  0x0000000000000000 
0x000000c000503f10:  0x000000c001a00000  0x0000000000000000 
0x000000c000503f20:  0x0000000000008000  0x0000000000000000 
0x000000c000503f30:  0x0000000000000000  0x0000000000000000 
0x000000c000503f40:  0x0000000000000202  0x00007fa547ffebf8 
0x000000c000503f50:  0x0000000000000000  0x000000c0001b4000 
0x000000c000503f60:  0x00000000000001aa  0x000000c00007b148 
0x000000c000503f70:  0x0000000000000003  0x000000c0005044d8 
fatal error: unknown caller pc

runtime stack:
runtime.throw({0x7fa58dc8b71f?, 0x7fa58fdf75c0?})
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/panic.go:1047 +0x5f fp=0x7fa54dffc610 sp=0x7fa54dffc5e0 pc=0x7fa58c2d9bbf
runtime.gentraceback(0x8?, 0x5?, 0x0?, 0xc000c63860, 0x0, 0x0, 0x7fffffff, 0x7fa54dffcb48, 0x7fa58c2c8a29?, 0x0)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/traceback.go:270 +0x1bb0 fp=0x7fa54dffc968 sp=0x7fa54dffc610 pc=0x7fa58c301d50
runtime.scanstack(0xc000c63860, 0xc000069c38)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgcmark.go:804 +0x1f2 fp=0x7fa54dffcb70 sp=0x7fa54dffc968 pc=0x7fa58c2c0cd2
runtime.markroot.func1()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgcmark.go:239 +0xb5 fp=0x7fa54dffcbc0 sp=0x7fa54dffcb70 pc=0x7fa58c2bfad5
runtime.markroot(0xc000069c38, 0x7e, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgcmark.go:213 +0x1a5 fp=0x7fa54dffcc60 sp=0x7fa54dffcbc0 pc=0x7fa58c2bf785
runtime.gcDrain(0xc000069c38, 0x7)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgcmark.go:1069 +0x39f fp=0x7fa54dffccc0 sp=0x7fa54dffcc60 pc=0x7fa58c2c187f
runtime.gcBgMarkWorker.func2()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgc.go:1367 +0x7a fp=0x7fa54dffcd10 sp=0x7fa54dffccc0 pc=0x7fa58c2bdd5a
runtime.systemstack()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:496 +0x46 fp=0x7fa54dffcd18 sp=0x7fa54dffcd10 pc=0x7fa58c30f026

goroutine 5 [GC worker (active)]:
runtime.systemstack_switch()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:463 fp=0xc000078750 sp=0xc000078748 pc=0x7fa58c30efc0
runtime.gcBgMarkWorker()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgc.go:1335 +0x209 fp=0xc0000787e0 sp=0xc000078750 pc=0x7fa58c2bda29
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000787e8 sp=0xc0000787e0 pc=0x7fa58c311261
created by runtime.gcBgMarkStartWorkers
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgc.go:1199 +0x25

goroutine 8 [chan receive, 4 minutes]:
runtime.gopark(0x0?, 0xc001563588?, 0x7?, 0xeb?, 0xc001563588?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc001563558 sp=0xc001563538 pc=0x7fa58c2dc936
runtime.chanrecv(0xc000e2e000, 0xc001563638, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/chan.go:583 +0x49d fp=0xc0015635e8 sp=0xc001563558 pc=0x7fa58c2a793d
runtime.chanrecv1(0xc000e2e000?, 0xc001563648?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/chan.go:442 +0x18 fp=0xc001563610 sp=0xc0015635e8 pc=0x7fa58c2a7438
github.com/cosmos/cosmos-sdk/server.WaitForQuitSignals()
	/home/runner/go/pkg/mod/github.com/agoric-labs/[email protected]/server/util.go:363 +0x8a fp=0xc001563678 sp=0xc001563610 pc=0x7fa58dc4ceea
github.com/cosmos/cosmos-sdk/server.wrapCPUProfile(0xc0000bffe0, 0xc000c6bba0)
	/home/runner/go/pkg/mod/github.com/agoric-labs/[email protected]/server/start.go:545 +0x310 fp=0xc001563760 sp=0xc001563678 pc=0x7fa58dc49b50
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc00112c000?, {0xc00132d790?, 0x0?, 0x1?})
	/home/runner/go/pkg/mod/github.com/agoric-labs/[email protected]/server/start.go:146 +0x230 fp=0xc001563c98 sp=0xc001563760 pc=0x7fa58dc46b30
github.com/spf13/cobra.(*Command).execute(0xc00112c000, {0xc00132d770, 0x1, 0x1})
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:916 +0x862 fp=0xc001563dd0 sp=0xc001563c98 pc=0x7fa58ca5b6c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc00111c000)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1044 +0x3bd fp=0xc001563e88 sp=0xc001563dd0 pc=0x7fa58ca5bf3d
github.com/spf13/cobra.(*Command).Execute(...)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:968
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:961
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0xc00053c120?, {0x0, 0x0}, {0xc000e4dd10, 0x26})
	/home/runner/go/pkg/mod/github.com/agoric-labs/[email protected]/server/cmd/execute.go:36 +0x20f fp=0xc001563f30 sp=0xc001563e88 pc=0x7fa58dc4f2af
github.com/Agoric/agoric-sdk/golang/cosmos/daemon.RunWithController(0x0?)
	/home/runner/work/agoric-sdk/agoric-sdk/agoric-sdk/golang/cosmos/daemon/main.go:45 +0x110 fp=0xc001563fb0 sp=0xc001563f30 pc=0x7fa58dc66450
main.RunAgCosmosDaemon.func2()
	/home/runner/work/agoric-sdk/agoric-sdk/agoric-sdk/golang/cosmos/cmd/libdaemon/main.go:86 +0x87 fp=0xc001563fe0 sp=0xc001563fb0 pc=0x7fa58dc66aa7
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001563fe8 sp=0xc001563fe0 pc=0x7fa58c311261
created by main.RunAgCosmosDaemon
	/home/runner/work/agoric-sdk/agoric-sdk/agoric-sdk/golang/cosmos/cmd/libdaemon/main.go:77 +0x1f6

goroutine 2 [force gc (idle), 4 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000076fb0 sp=0xc000076f90 pc=0x7fa58c2dc936
runtime.goparkunlock(...)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:387
runtime.forcegchelper()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:305 +0xb0 fp=0xc000076fe0 sp=0xc000076fb0 pc=0x7fa58c2dc770
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000076fe8 sp=0xc000076fe0 pc=0x7fa58c311261
created by runtime.init.6
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:293 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000077780 sp=0xc000077760 pc=0x7fa58c2dc936
runtime.goparkunlock(...)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:387
runtime.bgsweep(0x0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgcsweep.go:319 +0xde fp=0xc0000777c8 sp=0xc000077780 pc=0x7fa58c2c693e
runtime.gcenable.func1()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgc.go:178 +0x26 fp=0xc0000777e0 sp=0xc0000777c8 pc=0x7fa58c2bbba6
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000777e8 sp=0xc0000777e0 pc=0x7fa58c311261
created by runtime.gcenable
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc00005c070?, 0x7fa58e6fc5b0?, 0x0?, 0x0?, 0x0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000077f70 sp=0xc000077f50 pc=0x7fa58c2dc936
runtime.goparkunlock(...)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:387
runtime.(*scavengerState).park(0x7fa58fe74ee0)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc000077fa0 sp=0xc000077f70 pc=0x7fa58c2c4813
runtime.bgscavenge(0x0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc000077fc8 sp=0xc000077fa0 pc=0x7fa58c2c4e05
runtime.gcenable.func2()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgc.go:179 +0x26 fp=0xc000077fe0 sp=0xc000077fc8 pc=0x7fa58c2bbb46
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0x7fa58c311261
created by runtime.gcenable
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgc.go:179 +0xaa

goroutine 18 [finalizer wait, 4 minutes]:
runtime.gopark(0x1a0?, 0x7fa58fe76660?, 0xe0?, 0x24?, 0xc000076770?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000076628 sp=0xc000076608 pc=0x7fa58c2dc936
runtime.runfinq()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mfinal.go:193 +0x107 fp=0xc0000767e0 sp=0xc000076628 pc=0x7fa58c2babc7
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000767e8 sp=0xc0000767e0 pc=0x7fa58c311261
created by runtime.createfing
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mfinal.go:163 +0x45

goroutine 19 [GC worker (idle), 1 minutes]:
runtime.gopark(0x88e9e0e688?, 0x0?, 0x0?, 0x0?, 0x0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000072750 sp=0xc000072730 pc=0x7fa58c2dc936
runtime.gcBgMarkWorker()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000727e0 sp=0xc000072750 pc=0x7fa58c2bd911
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000727e8 sp=0xc0000727e0 pc=0x7fa58c311261
created by runtime.gcBgMarkStartWorkers
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgc.go:1199 +0x25

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x7fa58feabaa0?, 0x1?, 0x23?, 0x85?, 0x0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000504750 sp=0xc000504730 pc=0x7fa58c2dc936
runtime.gcBgMarkWorker()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0005047e0 sp=0xc000504750 pc=0x7fa58c2bd911
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005047e8 sp=0xc0005047e0 pc=0x7fa58c311261
created by runtime.gcBgMarkStartWorkers
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgc.go:1199 +0x25

goroutine 50 [GC worker (idle)]:
runtime.gopark(0x9abb31515a?, 0xc000508020?, 0x1a?, 0x14?, 0x0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000500750 sp=0xc000500730 pc=0x7fa58c2dc936
runtime.gcBgMarkWorker()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0005007e0 sp=0xc000500750 pc=0x7fa58c2bd911
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005007e8 sp=0xc0005007e0 pc=0x7fa58c311261
created by runtime.gcBgMarkStartWorkers
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgc.go:1199 +0x25

goroutine 6 [select, 1 minutes]:
runtime.gopark(0xc000079f88?, 0x3?, 0x90?, 0x0?, 0xc000079f72?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000079df8 sp=0xc000079dd8 pc=0x7fa58c2dc936
runtime.selectgo(0xc000079f88, 0xc000079f6c, 0xc000ccaf00?, 0x0, 0x7fa58c2a66dd?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc000079f38 sp=0xc000079df8 pc=0x7fa58c2ecffe
go.opencensus.io/stats/view.(*worker).start(0xc000ccaf00)
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:292 +0xad fp=0xc000079fc8 sp=0xc000079f38 pc=0x7fa58d4a594d
go.opencensus.io/stats/view.init.0.func1()
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x26 fp=0xc000079fe0 sp=0xc000079fc8 pc=0x7fa58d4a4b86
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0x7fa58c311261
created by go.opencensus.io/stats/view.init.0
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x96

goroutine 7 [select, 4 minutes]:
runtime.gopark(0xc0000797b0?, 0x2?, 0x0?, 0x0?, 0xc00007978c?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000079600 sp=0xc0000795e0 pc=0x7fa58c2dc936
runtime.selectgo(0xc0000797b0, 0xc000079788, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc000079740 sp=0xc000079600 pc=0x7fa58c2ecffe
github.com/desertbit/timer.timerRoutine()
	/home/runner/go/pkg/mod/github.com/desertbit/[email protected]/timers.go:119 +0xba fp=0xc0000797e0 sp=0xc000079740 pc=0x7fa58dbfd71a
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000797e8 sp=0xc0000797e0 pc=0x7fa58c311261
created by github.com/desertbit/timer.init.0
	/home/runner/go/pkg/mod/github.com/desertbit/[email protected]/timers.go:15 +0x25

goroutine 20 [select, 2 minutes, locked to thread]:
runtime.gopark(0xc000500fa8?, 0x2?, 0xb2?, 0xcc?, 0xc000500fa4?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000500e18 sp=0xc000500df8 pc=0x7fa58c2dc936
runtime.selectgo(0xc000500fa8, 0xc000500fa0, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc000500f58 sp=0xc000500e18 pc=0x7fa58c2ecffe
runtime.ensureSigM.func1()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/signal_unix.go:1004 +0x1b4 fp=0xc000500fe0 sp=0xc000500f58 pc=0x7fa58c307774
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000500fe8 sp=0xc000500fe0 pc=0x7fa58c311261
created by runtime.ensureSigM
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/signal_unix.go:987 +0xbd

goroutine 21 [syscall, 4 minutes]:
runtime.notetsleepg(0x0?, 0x0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/lock_futex.go:236 +0x34 fp=0xc0005017a0 sp=0xc000501768 pc=0x7fa58c2ad214
os/signal.signal_recv()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/sigqueue.go:152 +0x2f fp=0xc0005017c0 sp=0xc0005017a0 pc=0x7fa58c30d2ef
os/signal.loop()
	/opt/hostedtoolcache/go/1.20.13/x64/src/os/signal/signal_unix.go:23 +0x19 fp=0xc0005017e0 sp=0xc0005017c0 pc=0x7fa58c978099
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005017e8 sp=0xc0005017e0 pc=0x7fa58c311261
created by os/signal.Notify.func1.1
	/opt/hostedtoolcache/go/1.20.13/x64/src/os/signal/signal.go:151 +0x2a

goroutine 51 [chan receive, 4 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000545f08 sp=0xc000545ee8 pc=0x7fa58c2dc936
runtime.chanrecv(0xc0001f8000, 0x0, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/chan.go:583 +0x49d fp=0xc000545f98 sp=0xc000545f08 pc=0x7fa58c2a793d
runtime.chanrecv1(0x0?, 0x0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/chan.go:442 +0x18 fp=0xc000545fc0 sp=0xc000545f98 pc=0x7fa58c2a7438
github.com/Agoric/agoric-sdk/golang/cosmos/daemon.RunWithController.func1()
	/home/runner/work/agoric-sdk/agoric-sdk/agoric-sdk/golang/cosmos/daemon/main.go:36 +0x1f fp=0xc000545fe0 sp=0xc000545fc0 pc=0x7fa58dc664bf
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000545fe8 sp=0xc000545fe0 pc=0x7fa58c311261
created by github.com/Agoric/agoric-sdk/golang/cosmos/daemon.RunWithController
	/home/runner/work/agoric-sdk/agoric-sdk/agoric-sdk/golang/cosmos/daemon/main.go:35 +0xd6

goroutine 52 [chan receive, 1 minutes]:
runtime.gopark(0xc000504f98?, 0x7fa58cd99357?, 0x30?, 0x4f?, 0x0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000504ef0 sp=0xc000504ed0 pc=0x7fa58c2dc936
runtime.chanrecv(0xc00012d380, 0x0, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/chan.go:583 +0x49d fp=0xc000504f80 sp=0xc000504ef0 pc=0x7fa58c2a793d
runtime.chanrecv1(0x0?, 0x0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/chan.go:442 +0x18 fp=0xc000504fa8 sp=0xc000504f80 pc=0x7fa58c2a7438
github.com/rcrowley/go-metrics.(*meterArbiter).tick(0x7fa58fe6e9e0)
	/home/runner/go/pkg/mod/github.com/rcrowley/[email protected]/meter.go:239 +0x2a fp=0xc000504fc8 sp=0xc000504fa8 pc=0x7fa58cd9922a
github.com/rcrowley/go-metrics.NewMeter.func2()
	/home/runner/go/pkg/mod/github.com/rcrowley/[email protected]/meter.go:46 +0x26 fp=0xc000504fe0 sp=0xc000504fc8 pc=0x7fa58cd987e6
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000504fe8 sp=0xc000504fe0 pc=0x7fa58c311261
created by github.com/rcrowley/go-metrics.NewMeter
	/home/runner/go/pkg/mod/github.com/rcrowley/[email protected]/meter.go:46 +0xd3

goroutine 53 [chan receive, 1 minutes]:
runtime.gopark(0x0?, 0xc007ec0650?, 0x7?, 0xeb?, 0xc007ec0650?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc007ec0620 sp=0xc007ec0600 pc=0x7fa58c2dc936
runtime.chanrecv(0xc00053dd40, 0xc007ec0700, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/chan.go:583 +0x49d fp=0xc007ec06b0 sp=0xc007ec0620 pc=0x7fa58c2a793d
runtime.chanrecv1(0xc00053dd40?, 0xc007ec0710?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/chan.go:442 +0x18 fp=0xc007ec06d8 sp=0xc007ec06b0 pc=0x7fa58c2a7438
github.com/cosmos/cosmos-sdk/server.WaitForQuitSignals()
	/home/runner/go/pkg/mod/github.com/agoric-labs/[email protected]/server/util.go:363 +0x8a fp=0xc007ec0740 sp=0xc007ec06d8 pc=0x7fa58dc4ceea
github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x7fa58ec67c90, 0xc003a68960}, 0x0, {0xc00132ec00, 0xb}, {0x7fa58ec60818, ...}, ...}, ...)
	/home/runner/go/pkg/mod/github.com/agoric-labs/[email protected]/server/start.go:501 +0x1a15 fp=0xc007ec1de8 sp=0xc007ec0740 pc=0x7fa58dc48f35
github.com/cosmos/cosmos-sdk/server.StartCmd.func2.2()
	/home/runner/go/pkg/mod/github.com/agoric-labs/[email protected]/server/start.go:147 +0x58 fp=0xc007ec1fa8 sp=0xc007ec1de8 pc=0x7fa58dc46d38
github.com/cosmos/cosmos-sdk/server.wrapCPUProfile.func2()
	/home/runner/go/pkg/mod/github.com/agoric-labs/[email protected]/server/start.go:535 +0x29 fp=0xc007ec1fe0 sp=0xc007ec1fa8 pc=0x7fa58dc49c49
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc007ec1fe8 sp=0xc007ec1fe0 pc=0x7fa58c311261
created by github.com/cosmos/cosmos-sdk/server.wrapCPUProfile
	/home/runner/go/pkg/mod/github.com/agoric-labs/[email protected]/server/start.go:534 +0x24a

goroutine 54 [select, 1 minutes]:
runtime.gopark(0xc001577d90?, 0x7?, 0x1?, 0x0?, 0xc0015779ba?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc0015777f0 sp=0xc0015777d0 pc=0x7fa58c2dc936
runtime.selectgo(0xc001577d90, 0xc0015779ac, 0x1?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc001577930 sp=0xc0015777f0 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc000e8dd10)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session_util.go:189 +0x59b fp=0xc001577fc8 sp=0xc001577930 pc=0x7fa58cb74f7b
github.com/syndtr/goleveldb/leveldb.newSession.func1()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:93 +0x26 fp=0xc001577fe0 sp=0xc001577fc8 pc=0x7fa58cb6eaa6
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001577fe8 sp=0xc001577fe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.newSession
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:93 +0x2e5

goroutine 55 [select, 1 minutes]:
runtime.gopark(0xc000505728?, 0x2?, 0x0?, 0x0?, 0xc0005056d0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000505540 sp=0xc000505520 pc=0x7fa58c2dc936
runtime.selectgo(0xc000505728, 0xc0005056cc, 0xc00132ec20?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc000505680 sp=0xc000505540 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc000f29180)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:91 +0x137 fp=0xc0005057c8 sp=0xc000505680 pc=0x7fa58cb5d4f7
github.com/syndtr/goleveldb/leveldb.openDB.func1()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:148 +0x26 fp=0xc0005057e0 sp=0xc0005057c8 pc=0x7fa58cb57546
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005057e8 sp=0xc0005057e0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:148 +0x4ea

goroutine 56 [select, 1 minutes]:
runtime.gopark(0xc000505f98?, 0x2?, 0x4?, 0x0?, 0xc000505f64?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000505de8 sp=0xc000505dc8 pc=0x7fa58c2dc936
runtime.selectgo(0xc000505f98, 0xc000505f60, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc000505f28 sp=0xc000505de8 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc000f29180)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_state.go:101 +0xa8 fp=0xc000505fc8 sp=0xc000505f28 pc=0x7fa58cb66c48
github.com/syndtr/goleveldb/leveldb.openDB.func2()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:149 +0x26 fp=0xc000505fe0 sp=0xc000505fc8 pc=0x7fa58cb574e6
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000505fe8 sp=0xc000505fe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:149 +0x52a

goroutine 57 [select, 1 minutes]:
runtime.gopark(0xc000089f10?, 0x3?, 0x0?, 0x0?, 0xc000089e5e?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000089ca0 sp=0xc000089c80 pc=0x7fa58c2dc936
runtime.selectgo(0xc000089f10, 0xc000089e58, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc000089de0 sp=0xc000089ca0 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc000f29180)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:836 +0x6b7 fp=0xc000089fc8 sp=0xc000089de0 pc=0x7fa58cb62e97
github.com/syndtr/goleveldb/leveldb.openDB.func3()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:155 +0x26 fp=0xc000089fe0 sp=0xc000089fc8 pc=0x7fa58cb57486
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000089fe8 sp=0xc000089fe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:155 +0x59b

goroutine 58 [select, 1 minutes]:
runtime.gopark(0xc009ffff90?, 0x2?, 0x0?, 0x60?, 0xc009ffff54?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc009925dd8 sp=0xc009925db8 pc=0x7fa58c2dc936
runtime.selectgo(0xc009925f90, 0xc009ffff50, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc009925f18 sp=0xc009925dd8 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc000f29180)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:773 +0x113 fp=0xc009925fc8 sp=0xc009925f18 pc=0x7fa58cb62653
github.com/syndtr/goleveldb/leveldb.openDB.func4()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:156 +0x26 fp=0xc009925fe0 sp=0xc009925fc8 pc=0x7fa58cb57426
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc009925fe8 sp=0xc009925fe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:156 +0x5d8

goroutine 9 [select, 4 minutes]:
runtime.gopark(0xc00008ed90?, 0x7?, 0x0?, 0x0?, 0xc00008e9ba?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc00008e7f0 sp=0xc00008e7d0 pc=0x7fa58c2dc936
runtime.selectgo(0xc00008ed90, 0xc00008e9ac, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc00008e930 sp=0xc00008e7f0 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc000e8c000)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session_util.go:189 +0x59b fp=0xc00008efc8 sp=0xc00008e930 pc=0x7fa58cb74f7b
github.com/syndtr/goleveldb/leveldb.newSession.func1()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:93 +0x26 fp=0xc00008efe0 sp=0xc00008efc8 pc=0x7fa58cb6eaa6
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00008efe8 sp=0xc00008efe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.newSession
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:93 +0x2e5

goroutine 10 [select, 4 minutes]:
runtime.gopark(0xc000078f28?, 0x2?, 0x0?, 0x0?, 0xc000078ed0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000078d40 sp=0xc000078d20 pc=0x7fa58c2dc936
runtime.selectgo(0xc000078f28, 0xc000078ecc, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc000078e80 sp=0xc000078d40 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc0005ca380)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:91 +0x137 fp=0xc000078fc8 sp=0xc000078e80 pc=0x7fa58cb5d4f7
github.com/syndtr/goleveldb/leveldb.openDB.func1()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:148 +0x26 fp=0xc000078fe0 sp=0xc000078fc8 pc=0x7fa58cb57546
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000078fe8 sp=0xc000078fe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:148 +0x4ea

goroutine 11 [select, 1 minutes]:
runtime.gopark(0xc000072f98?, 0x2?, 0x1?, 0x0?, 0xc000072f64?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000072de8 sp=0xc000072dc8 pc=0x7fa58c2dc936
runtime.selectgo(0xc000072f98, 0xc000072f60, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc000072f28 sp=0xc000072de8 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc0005ca380)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_state.go:101 +0xa8 fp=0xc000072fc8 sp=0xc000072f28 pc=0x7fa58cb66c48
github.com/syndtr/goleveldb/leveldb.openDB.func2()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:149 +0x26 fp=0xc000072fe0 sp=0xc000072fc8 pc=0x7fa58cb574e6
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000072fe8 sp=0xc000072fe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:149 +0x52a

goroutine 12 [select, 4 minutes]:
runtime.gopark(0xc000073710?, 0x3?, 0x0?, 0x0?, 0xc00007365e?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc001574ca0 sp=0xc001574c80 pc=0x7fa58c2dc936
runtime.selectgo(0xc001574f10, 0xc000073658, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc001574de0 sp=0xc001574ca0 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc0005ca380)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:836 +0x6b7 fp=0xc001574fc8 sp=0xc001574de0 pc=0x7fa58cb62e97
github.com/syndtr/goleveldb/leveldb.openDB.func3()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:155 +0x26 fp=0xc001574fe0 sp=0xc001574fc8 pc=0x7fa58cb57486
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001574fe8 sp=0xc001574fe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:155 +0x59b

goroutine 13 [select, 4 minutes]:
runtime.gopark(0xc000073f90?, 0x2?, 0x0?, 0x0?, 0xc000073f54?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000073dd8 sp=0xc000073db8 pc=0x7fa58c2dc936
runtime.selectgo(0xc000073f90, 0xc000073f50, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc000073f18 sp=0xc000073dd8 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc0005ca380)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:773 +0x113 fp=0xc000073fc8 sp=0xc000073f18 pc=0x7fa58cb62653
github.com/syndtr/goleveldb/leveldb.openDB.func4()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:156 +0x26 fp=0xc000073fe0 sp=0xc000073fc8 pc=0x7fa58cb57426
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000073fe8 sp=0xc000073fe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:156 +0x5d8

goroutine 14 [select, 4 minutes]:
runtime.gopark(0xc00008fd90?, 0x7?, 0x0?, 0x0?, 0xc00008f9ba?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc00008f7f0 sp=0xc00008f7d0 pc=0x7fa58c2dc936
runtime.selectgo(0xc00008fd90, 0xc00008f9ac, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc00008f930 sp=0xc00008f7f0 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc000e8c3c0)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session_util.go:189 +0x59b fp=0xc00008ffc8 sp=0xc00008f930 pc=0x7fa58cb74f7b
github.com/syndtr/goleveldb/leveldb.newSession.func1()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:93 +0x26 fp=0xc00008ffe0 sp=0xc00008ffc8 pc=0x7fa58cb6eaa6
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00008ffe8 sp=0xc00008ffe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.newSession
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:93 +0x2e5

goroutine 15 [select, 4 minutes]:
runtime.gopark(0xc000074728?, 0x2?, 0x0?, 0x0?, 0xc0000746d0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000074540 sp=0xc000074520 pc=0x7fa58c2dc936
runtime.selectgo(0xc000074728, 0xc0000746cc, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc000074680 sp=0xc000074540 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc0005cafc0)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:91 +0x137 fp=0xc0000747c8 sp=0xc000074680 pc=0x7fa58cb5d4f7
github.com/syndtr/goleveldb/leveldb.openDB.func1()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:148 +0x26 fp=0xc0000747e0 sp=0xc0000747c8 pc=0x7fa58cb57546
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000747e8 sp=0xc0000747e0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:148 +0x4ea

goroutine 16 [select, 1 minutes]:
runtime.gopark(0xc000074f98?, 0x2?, 0x38?, 0x4e?, 0xc000074f64?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000074de8 sp=0xc000074dc8 pc=0x7fa58c2dc936
runtime.selectgo(0xc000074f98, 0xc000074f60, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc000074f28 sp=0xc000074de8 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc0005cafc0)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_state.go:101 +0xa8 fp=0xc000074fc8 sp=0xc000074f28 pc=0x7fa58cb66c48
github.com/syndtr/goleveldb/leveldb.openDB.func2()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:149 +0x26 fp=0xc000074fe0 sp=0xc000074fc8 pc=0x7fa58cb574e6
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000074fe8 sp=0xc000074fe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:149 +0x52a

goroutine 67 [select, 4 minutes]:
runtime.gopark(0xc000075710?, 0x3?, 0x0?, 0x0?, 0xc00007565e?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc001570ca0 sp=0xc001570c80 pc=0x7fa58c2dc936
runtime.selectgo(0xc001570f10, 0xc000075658, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc001570de0 sp=0xc001570ca0 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc0005cafc0)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:836 +0x6b7 fp=0xc001570fc8 sp=0xc001570de0 pc=0x7fa58cb62e97
github.com/syndtr/goleveldb/leveldb.openDB.func3()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:155 +0x26 fp=0xc001570fe0 sp=0xc001570fc8 pc=0x7fa58cb57486
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001570fe8 sp=0xc001570fe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:155 +0x59b

goroutine 68 [select, 4 minutes]:
runtime.gopark(0xc000075f90?, 0x2?, 0x0?, 0x0?, 0xc000075f54?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000075dd8 sp=0xc000075db8 pc=0x7fa58c2dc936
runtime.selectgo(0xc000075f90, 0xc000075f50, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc000075f18 sp=0xc000075dd8 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc0005cafc0)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:773 +0x113 fp=0xc000075fc8 sp=0xc000075f18 pc=0x7fa58cb62653
github.com/syndtr/goleveldb/leveldb.openDB.func4()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:156 +0x26 fp=0xc000075fe0 sp=0xc000075fc8 pc=0x7fa58cb57426
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000075fe8 sp=0xc000075fe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:156 +0x5d8

goroutine 69 [select, 4 minutes]:
runtime.gopark(0xc00008ad90?, 0x7?, 0x0?, 0x0?, 0xc00008a9ba?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc00008a7f0 sp=0xc00008a7d0 pc=0x7fa58c2dc936
runtime.selectgo(0xc00008ad90, 0xc00008a9ac, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc00008a930 sp=0xc00008a7f0 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc000e8c4b0)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session_util.go:189 +0x59b fp=0xc00008afc8 sp=0xc00008a930 pc=0x7fa58cb74f7b
github.com/syndtr/goleveldb/leveldb.newSession.func1()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:93 +0x26 fp=0xc00008afe0 sp=0xc00008afc8 pc=0x7fa58cb6eaa6
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00008afe8 sp=0xc00008afe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.newSession
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:93 +0x2e5

goroutine 70 [select, 4 minutes]:
runtime.gopark(0xc000507728?, 0x2?, 0x0?, 0x0?, 0xc0005076d0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc0015bed40 sp=0xc0015bed20 pc=0x7fa58c2dc936
runtime.selectgo(0xc0015bef28, 0xc0005076cc, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc0015bee80 sp=0xc0015bed40 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc0005cb180)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:91 +0x137 fp=0xc0015befc8 sp=0xc0015bee80 pc=0x7fa58cb5d4f7
github.com/syndtr/goleveldb/leveldb.openDB.func1()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:148 +0x26 fp=0xc0015befe0 sp=0xc0015befc8 pc=0x7fa58cb57546
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0015befe8 sp=0xc0015befe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:148 +0x4ea

goroutine 71 [select, 1 minutes]:
runtime.gopark(0xc000507f98?, 0x2?, 0x4?, 0x0?, 0xc000507f64?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000507de8 sp=0xc000507dc8 pc=0x7fa58c2dc936
runtime.selectgo(0xc000507f98, 0xc000507f60, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc000507f28 sp=0xc000507de8 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc0005cb180)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_state.go:101 +0xa8 fp=0xc000507fc8 sp=0xc000507f28 pc=0x7fa58cb66c48
github.com/syndtr/goleveldb/leveldb.openDB.func2()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:149 +0x26 fp=0xc000507fe0 sp=0xc000507fc8 pc=0x7fa58cb574e6
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000507fe8 sp=0xc000507fe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:149 +0x52a

goroutine 72 [select, 4 minutes]:
runtime.gopark(0xc000501f10?, 0x3?, 0x0?, 0x0?, 0xc000501e5e?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc001571ca0 sp=0xc001571c80 pc=0x7fa58c2dc936
runtime.selectgo(0xc001571f10, 0xc000501e58, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc001571de0 sp=0xc001571ca0 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc0005cb180)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:836 +0x6b7 fp=0xc001571fc8 sp=0xc001571de0 pc=0x7fa58cb62e97
github.com/syndtr/goleveldb/leveldb.openDB.func3()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:155 +0x26 fp=0xc001571fe0 sp=0xc001571fc8 pc=0x7fa58cb57486
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001571fe8 sp=0xc001571fe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:155 +0x59b

goroutine 73 [select, 4 minutes]:
runtime.gopark(0xc000502790?, 0x2?, 0x0?, 0x0?, 0xc000502754?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc0005025d8 sp=0xc0005025b8 pc=0x7fa58c2dc936
runtime.selectgo(0xc000502790, 0xc000502750, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc000502718 sp=0xc0005025d8 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc0005cb180)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:773 +0x113 fp=0xc0005027c8 sp=0xc000502718 pc=0x7fa58cb62653
github.com/syndtr/goleveldb/leveldb.openDB.func4()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:156 +0x26 fp=0xc0005027e0 sp=0xc0005027c8 pc=0x7fa58cb57426
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005027e8 sp=0xc0005027e0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.openDB
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:156 +0x5d8

goroutine 74 [select, 4 minutes]:
runtime.gopark(0xc000502f78?, 0x4?, 0x0?, 0x0?, 0xc000502f58?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000502de0 sp=0xc000502dc0 pc=0x7fa58c2dc936
runtime.selectgo(0xc000502f78, 0xc000502f50, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc000502f20 sp=0xc000502de0 pc=0x7fa58c2ecffe
github.com/tendermint/tendermint/proxy.(*multiAppConn).killTMOnClientError(0xc000fd96c0)
	/home/runner/go/pkg/mod/github.com/agoric-labs/[email protected]/proxy/multi_app_conn.go:138 +0x11c fp=0xc000502fc8 sp=0xc000502f20 pc=0x7fa58da5fcfc
github.com/tendermint/tendermint/proxy.(*multiAppConn).OnStart.func1()
	/home/runner/go/pkg/mod/github.com/agoric-labs/[email protected]/proxy/multi_app_conn.go:118 +0x26 fp=0xc000502fe0 sp=0xc000502fc8 pc=0x7fa58da5fb66
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000502fe8 sp=0xc000502fe0 pc=0x7fa58c311261
created by github.com/tendermint/tendermint/proxy.(*multiAppConn).OnStart
	/home/runner/go/pkg/mod/github.com/agoric-labs/[email protected]/proxy/multi_app_conn.go:118 +0x34d

goroutine 75 [chan receive, 1 minutes]:
runtime.gopark(0xc00053c540?, 0xc0011240d0?, 0xd0?, 0x40?, 0x0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc0015bdd40 sp=0xc0015bdd20 pc=0x7fa58c2dc936
runtime.chanrecv(0xc000cef5c0, 0xc0015bdea0, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/chan.go:583 +0x49d fp=0xc0015bddd0 sp=0xc0015bdd40 pc=0x7fa58c2a793d
runtime.chanrecv2(0xc0015bdfc0?, 0x7fa58ea3a680?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/chan.go:447 +0x18 fp=0xc0015bddf8 sp=0xc0015bddd0 pc=0x7fa58c2a7478
github.com/tendermint/tendermint/libs/pubsub.(*Server).loop(0xc000fd3650, {0xc0014984e0, 0xc001498510})
	/home/runner/go/pkg/mod/github.com/agoric-labs/[email protected]/libs/pubsub/pubsub.go:323 +0x8e fp=0xc0015bdfb8 sp=0xc0015bddf8 pc=0x7fa58c96b98e
github.com/tendermint/tendermint/libs/pubsub.(*Server).OnStart.func1()
	/home/runner/go/pkg/mod/github.com/agoric-labs/[email protected]/libs/pubsub/pubsub.go:309 +0x2e fp=0xc0015bdfe0 sp=0xc0015bdfb8 pc=0x7fa58c96b8ae
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0015bdfe8 sp=0xc0015bdfe0 pc=0x7fa58c311261
created by github.com/tendermint/tendermint/libs/pubsub.(*Server).OnStart
	/home/runner/go/pkg/mod/github.com/agoric-labs/[email protected]/libs/pubsub/pubsub.go:309 +0xaa

goroutine 76 [select, 4 minutes]:
runtime.gopark(0xc00008bd90?, 0x7?, 0x0?, 0x0?, 0xc00008b9ba?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc00008b7f0 sp=0xc00008b7d0 pc=0x7fa58c2dc936
runtime.selectgo(0xc00008bd90, 0xc00008b9ac, 0x0?, 0x0, 0x0?, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/select.go:327 +0x7be fp=0xc00008b930 sp=0xc00008b7f0 pc=0x7fa58c2ecffe
github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc000e8c5a0)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session_util.go:189 +0x59b fp=0xc00008bfc8 sp=0xc00008b930 pc=0x7fa58cb74f7b
github.com/syndtr/goleveldb/leveldb.newSession.func1()
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:93 +0x26 fp=0xc00008bfe0 sp=0xc00008bfc8 pc=0x7fa58cb6eaa6
runtime.goexit()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00008bfe8 sp=0xc00008bfe0 pc=0x7fa58c311261
created by github.com/syndtr/goleveldb/leveldb.newSession
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:93 +0x2e5

goroutine 77 [select (scan), 4 minutes]:
runtime.gopark(0xc000503f28?, 0x2?, 0x0?, 0x0?, 0xc000503ed0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000503d40 sp=0xc000503d20 pc=0x7fa58c2dc936
fatal error: unexpected signal during runtime execution
panic during panic
[signal SIGSEGV: segmentation violation code=0x1 addr=0x118 pc=0x7fa58c300a5c]

runtime stack:
runtime.throw({0x7fa58dce3b0b?, 0x7fa58fdf75c0?})
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/panic.go:1047 +0x5f fp=0x7fa54dffbe98 sp=0x7fa54dffbe68 pc=0x7fa58c2d9bbf
runtime.sigpanic()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/signal_unix.go:825 +0x3e9 fp=0x7fa54dffbef8 sp=0x7fa54dffbe98 pc=0x7fa58c2f1089
runtime.gentraceback(0x7fa58f7c7f68?, 0x7fa58fdf75c0?, 0x7fa54dffc2b0?, 0xc000c63860, 0x0, 0x0, 0x64, 0x0, 0x7fa54dffc2d0?, 0x0)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/traceback.go:258 +0x8bc fp=0x7fa54dffc250 sp=0x7fa54dffbef8 pc=0x7fa58c300a5c
runtime.traceback1(0xc000c63860?, 0x7fa58c2db600?, 0x3?, 0xc000c63860, 0x7fa58c30385b?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/traceback.go:776 +0x1b6 fp=0x7fa54dffc410 sp=0x7fa54dffc250 pc=0x7fa58c302b56
runtime.traceback(...)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/traceback.go:723
runtime.tracebackothers.func1(0xc000c63860)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/traceback.go:992 +0xe5 fp=0x7fa54dffc450 sp=0x7fa54dffc410 pc=0x7fa58c303a85
runtime.forEachGRace(0x7fa54dffc4b8)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/proc.go:604 +0x4d fp=0x7fa54dffc480 sp=0x7fa54dffc450 pc=0x7fa58c2dd46d
runtime.tracebackothers(0xc0000cc1a0?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/traceback.go:978 +0xe5 fp=0x7fa54dffc4e8 sp=0x7fa54dffc480 pc=0x7fa58c303965
runtime.dopanic_m(0xc0000cc1a0, 0x7fa58e6fc730?, 0x1?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/panic.go:1273 +0x285 fp=0x7fa54dffc560 sp=0x7fa54dffc4e8 pc=0x7fa58c2da5a5
runtime.fatalthrow.func1()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/panic.go:1127 +0x70 fp=0x7fa54dffc5a0 sp=0x7fa54dffc560 pc=0x7fa58c2d9fb0
runtime.fatalthrow(0x4dffc5e8?)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/panic.go:1120 +0x6c fp=0x7fa54dffc5e0 sp=0x7fa54dffc5a0 pc=0x7fa58c2d9f0c
runtime.throw({0x7fa58dc8b71f?, 0x7fa58fdf75c0?})
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/panic.go:1047 +0x5f fp=0x7fa54dffc610 sp=0x7fa54dffc5e0 pc=0x7fa58c2d9bbf
runtime.gentraceback(0x8?, 0x5?, 0x0?, 0xc000c63860, 0x0, 0x0, 0x7fffffff, 0x7fa54dffcb48, 0x7fa58c2c8a29?, 0x0)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/traceback.go:270 +0x1bb0 fp=0x7fa54dffc968 sp=0x7fa54dffc610 pc=0x7fa58c301d50
runtime.scanstack(0xc000c63860, 0xc000069c38)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgcmark.go:804 +0x1f2 fp=0x7fa54dffcb70 sp=0x7fa54dffc968 pc=0x7fa58c2c0cd2
runtime.markroot.func1()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgcmark.go:239 +0xb5 fp=0x7fa54dffcbc0 sp=0x7fa54dffcb70 pc=0x7fa58c2bfad5
runtime.markroot(0xc000069c38, 0x7e, 0x1)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgcmark.go:213 +0x1a5 fp=0x7fa54dffcc60 sp=0x7fa54dffcbc0 pc=0x7fa58c2bf785
runtime.gcDrain(0xc000069c38, 0x7)
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgcmark.go:1069 +0x39f fp=0x7fa54dffccc0 sp=0x7fa54dffcc60 pc=0x7fa58c2c187f
runtime.gcBgMarkWorker.func2()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/mgc.go:1367 +0x7a fp=0x7fa54dffcd10 sp=0x7fa54dffccc0 pc=0x7fa58c2bdd5a
runtime.systemstack()
	/opt/hostedtoolcache/go/1.20.13/x64/src/runtime/asm_amd64.s:496 +0x46 fp=0x7fa54dffcd18 sp=0x7fa54dffcd10 pc=0x7fa58c30f026

It looks very similar to #8047. Good news is that it's not a new issue, I was probably unlucky to trigger it. Bad news is that the new ibc / cosmos stack does not fix it.

@mhofman
Copy link
Member Author

mhofman commented Feb 8, 2024

I am holding this until we solve #8871

@gibson042
Copy link
Member

I'm seeing a lot of divergence between this PR and my local attempt to recreate it by following the rebase-todo. Did you miss a chunk of picks?

@mhofman
Copy link
Member Author

mhofman commented Feb 8, 2024

I'm seeing a lot of divergence between this PR and my local attempt to recreate it by following the rebase-todo. Did you miss a chunk of picks?

I have merged some other PRs into dev-upgrade-14 since I held up this PR, but haven't updated this branch on top of those yet. Maybe the base if not the same?

gibson042 and others added 24 commits February 18, 2024 15:23
feat(cosmos): Add a "CapData" vstorage RPC endpoint
chore(cosmos): Update doc comments for accuracy
docs(cosmos/x/vstorage): Add REAME link to vstorage/query.proto
docs(cosmos/x/vstorage): Add README sections for CLI and "REST" API interfaces
…-example

docs(vstorage): Add an example external protobuf interface interaction
cf. https://pkg.go.dev/github.com/spf13/cobra#Command
and https://github.com/spf13/cobra/blob/v1.8.0/site/content/user_guide.md#example
Only optional arguments should be wrapped in square brackets,
alternatives should be wrapped in curly braces,
and `Long` strings should generally start similar to `Use`.
feat: agd query vstorage trailing slash for children
Replace panics with friendly error messages.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
style(golang): Align CLI command fields with upstream recommendations
docs(golang): Improve the description of PopulateAction
Copy link
Member

@gibson042 gibson042 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Selected changes look reasonable and the rebase-todo corresponds with final state modulo the edit lines1. LGTM 👍

Footnotes

  1. side note: I wonder if we could eliminate both those and the handful of merge conflicts by means of exec lines to insert novel patches in the stream, but would only want to pursue that if we think this overall pattern will endure.

@mhofman
Copy link
Member Author

mhofman commented Feb 19, 2024

modulo the edit lines1. LGTM 👍

Footnotes

  1. side note: I wonder if we could eliminate both those and the handful of merge conflicts by means of exec lines to insert novel patches in the stream, but would only want to pursue that if we think this overall pattern will endure.

Interesting idea, but hopefully we don't need to do this much longer.

@mhofman mhofman merged commit c8f9e7b into dev-upgrade-14 Feb 19, 2024
63 of 64 checks passed
@mhofman mhofman deleted the mhofman/cherry-pick-ibc-updates branch February 19, 2024 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
force:integration Force integration tests to run on PR proto:expect-breakage
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants