Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: pruning while acitve reader is present #9570

Closed
1 of 4 tasks
tac0turtle opened this issue Jun 23, 2021 · 14 comments
Closed
1 of 4 tasks

bug: pruning while acitve reader is present #9570

tac0turtle opened this issue Jun 23, 2021 · 14 comments
Assignees
Milestone

Comments

@tac0turtle
Copy link
Member

Summary of Bug

Node tried to prune with an active reader, this should not be possible

Version

0.42.5

Steps to Reproduce

you can reproduce by syncing an osmosis node with pruning set to everything.

logs:

Jun 23 13:27:20 angl-1 osmosisd[75297]: panic: unable to delete version 1500 with 1 active readers
Jun 23 13:27:20 angl-1 osmosisd[75297]: goroutine 150625 [running]:
Jun 23 13:27:20 angl-1 osmosisd[75297]: github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).pruneStores(0xc0000ee280)
Jun 23 13:27:20 angl-1 osmosisd[75297]:         github.com/cosmos/[email protected]/store/rootmulti/store.go:388 +0x26e
Jun 23 13:27:20 angl-1 osmosisd[75297]: github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Commit(0xc0000ee280, 0x5e6, 0x0, 0xc0037bf8c0, 0x71374491c28a2f98)
Jun 23 13:27:20 angl-1 osmosisd[75297]:         github.com/cosmos/[email protected]/store/rootmulti/store.go:362 +0x1c6
Jun 23 13:27:20 angl-1 osmosisd[75297]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc000dfb380, 0x0, 0x0, 0x0, 0x0)
Jun 23 13:27:20 angl-1 osmosisd[75297]:         github.com/cosmos/[email protected]/baseapp/abci.go:293 +0x27c
Jun 23 13:27:20 angl-1 osmosisd[75297]: github.com/tendermint/tendermint/abci/client.(*localClient).CommitSync(0xc000e753e0, 0x0, 0x0, 0x0)
Jun 23 13:27:20 angl-1 osmosisd[75297]:         github.com/tendermint/[email protected]/abci/client/local_client.go:258 +0xab
Jun 23 13:27:20 angl-1 osmosisd[75297]: github.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync(0xc000e0b350, 0x0, 0x0, 0x10930ef)
Jun 23 13:27:20 angl-1 osmosisd[75297]:         github.com/tendermint/[email protected]/proxy/app_conn.go:93 +0x33
Jun 23 13:27:20 angl-1 osmosisd[75297]: github.com/tendermint/tendermint/state.(*BlockExecutor).Commit(0xc000114a80, 0xb, 0x1, 0x0, 0x0, 0xc00f041630, 0x9, 0x1, 0x5e6, 0xc019f77000, ...)
Jun 23 13:27:20 angl-1 osmosisd[75297]:         github.com/tendermint/[email protected]/state/execution.go:228 +0x244
Jun 23 13:27:20 angl-1 osmosisd[75297]: github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc000114a80, 0xb, 0x1, 0x0, 0x0, 0xc00f041630, 0x9, 0x1, 0x5e6, 0xc019f77000, ...)
Jun 23 13:27:20 angl-1 osmosisd[75297]:         github.com/tendermint/[email protected]/state/execution.go:180 +0x725
Jun 23 13:27:20 angl-1 osmosisd[75297]: github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc006e9a540, 0x0)
Jun 23 13:27:20 angl-1 osmosisd[75297]:         github.com/tendermint/[email protected]/blockchain/v0/reactor.go:398 +0x1033
Jun 23 13:27:20 angl-1 osmosisd[75297]: created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
Jun 23 13:27:20 angl-1 osmosisd[75297]:         github.com/tendermint/[email protected]/blockchain/v0/reactor.go:110 +0x8c

cc @ValarDragon @sunnya97
not sure if this coming from one of your modules but should make note in a README.


For Admin Use

  • Not duplicate issue
  • Appropriate labels applied
  • Appropriate contributors tagged
  • Contributor assigned/self-assigned
@tac0turtle
Copy link
Member Author

This is a bug where snapshotting was configured to x and pruning was configured to y. It would be good to error out in the case where snapshotting is not happening on a multiple of pruning. this way you avoid getting to a hieght and erroring out

@clevinson
Copy link
Contributor

Do you think this could be state machine breaking, or is can it be safely back ported to v0.42.x ?

@alexanderbez
Copy link
Contributor

I thought we already added a check for this in BaseApp? Not sure if its in 0.42. I could be wrong here, but I thought we added a check to make sure the snapshotting configuration takes into consideration the pruning configuration.

@alexanderbez
Copy link
Contributor

@tac0turtle
Copy link
Member Author

@amaury1093
Copy link
Contributor

amaury1093 commented Jul 2, 2021

This is actually already in v0.42.x: https://github.com/cosmos/cosmos-sdk/blob/release/v0.42.x/baseapp/baseapp.go#L296-L308

so this issue still needs more investigating

@alexanderbez
Copy link
Contributor

So can we close this?

@amaury1093
Copy link
Contributor

Or at least @marbar3778 could you help us reproduce this? @likhita-809 on our side ran an osmosis node with pruning = "everything" until ~17000 and still can't repro.

@tac0turtle
Copy link
Member Author

Or at least @marbar3778 could you help us reproduce this? @likhita-809 on our side ran an osmosis node with pruning = "everything" until ~17000 and still can't repro.

then it seems there is some non-determinism somewhere. I was able to reproduce it on two nodes. Will try again

@cyberbono3
Copy link
Contributor

cyberbono3 commented Jul 12, 2021

@marbar3778 can you leave any comments on this?

@likhita-809
Copy link
Contributor

@marbar3778 any progress on this ?

@likhita-809
Copy link
Contributor

Or at least @marbar3778 could you help us reproduce this? @likhita-809 on our side ran an osmosis node with pruning = "everything" until ~17000 and still can't repro.

then it seems there is some non-determinism somewhere. I was able to reproduce it on two nodes. Will try again

@marbar3778 did you try this again ?

@tac0turtle
Copy link
Member Author

trying to connect to nodes now

@tac0turtle
Copy link
Member Author

I wasnt able to reproduce, super weird. Will close for now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants