Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic in light client db #444

Closed
colin-axner opened this issue Mar 4, 2021 · 2 comments · Fixed by #447 or #478
Closed

Panic in light client db #444

colin-axner opened this issue Mar 4, 2021 · 2 comments · Fixed by #447 or #478
Assignees
@alexanderbez
Labels
T: Bug 🪲 TYPE: Inconsistencies or issues which will cause an issue or problem for users or implementors.
Milestone
Relayer Productio...

Comments

@colin-axner
Copy link
Contributor

Reported from the cosmos discord channel for IBC

panic: can't open light client database: resource temporarily unavailable

goroutine 411600 [running]:
github.com/cosmos/relayer/relayer.(Chain).MustGetLatestLightHeight(...)
        /home/microtick/relayer/relayer/tm-light-client.go:269
github.com/cosmos/relayer/relayer.(relayMsgRecvPacket).FetchCommitResponse.func1(0x1481000, 0x1)
        /home/microtick/relayer/relayer/relayPackets.go:137 +0x265
github.com/avast/retry-go.Do(0xc00006ee98, 0xc00006ee78, 0x4, 0x4, 0x97, 0x0)
        /home/microtick/go/pkg/mod/github.com/avast/[email protected]+incompatible/retry.go:114 +0x15f
github.com/cosmos/relayer/relayer.(relayMsgRecvPacket).FetchCommitResponse(0xc003114ec0, 0xc000cfdd40, 0xc000cfdc20, 0x0, 0x0)
        /home/microtick/relayer/relayer/relayPackets.go:135 +0x145
github.com/cosmos/relayer/relayer.(NaiveStrategy).sendTxFromEventPackets(0xc000e7f600, 0xc000cfdd40, 0xc000cfdc20, 0xc0054ce880, 0x1, 0x1)
        /home/microtick/relayer/relayer/naive-strategy.go:345 +0x99
github.com/cosmos/relayer/relayer.(*NaiveStrategy).HandleEvents(0xc000e7f600, 0xc000cfdd40, 0xc000cfdc20, 0xc002ffa6c0)
        /home/microtick/relayer/relayer/naive-strategy.go:237 +0x85
created by github.com/cosmos/relayer/relayer.relayerListenLoop
        /home/microtick/relayer/relayer/strategies.go:138 +0xcd1
Disk is not full, and I tried some questionable things like deleting .relayer/light/chain_id.db/LOCK without success
@colin-axner colin-axner added the T: Bug 🪲 TYPE: Inconsistencies or issues which will cause an issue or problem for users or implementors. label Mar 4, 2021
@colin-axner
Copy link
Contributor Author

Try leaving the relayer running to attempt to reproduce

@colin-axner colin-axner mentioned this issue Mar 8, 2021
Merged
@alexanderbez
Copy link
Contributor

Good work @colin-axner but this issue still persists. While I think your attempted approach is sound and makes sense, it doesn't resolve the issue at hand. Being so, I think this just alludes to the fact the current relayer architecture and design is not suited for production-grade environments.

I'll re-open this issue along with an ADR for a proposal that I believe will make the relayer more robust and reliable 👍

@alexanderbez alexanderbez reopened this Mar 25, 2021
@alexanderbez alexanderbez self-assigned this Mar 25, 2021
@alexanderbez alexanderbez mentioned this issue Mar 28, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexanderbez alexanderbez

Labels
T: Bug 🪲 TYPE: Inconsistencies or issues which will cause an issue or problem for users or implementors.
Projects
None yet
Milestone
Relayer Production Readiness
Development

Successfully merging a pull request may close this issue.

add a light client database lock cosmos/relayer
chain: retry on db failure cosmos/relayer
2 participants
@alexanderbez @colin-axner