Rewind fix #6
Merged
Rewind fix #6
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Brindrajsinh-Chauhan
force-pushed
the
rewind-fix
branch
from
c24d557 to
5f07d0f
Compare
Brindrajsinh-Chauhan
force-pushed
the
rewind-fix
branch
2 times, most recently
from
8d46965 to
00ea7a2
Compare
Brindrajsinh-Chauhan
force-pushed
the
rewind-fix
branch
from
00ea7a2 to
185355d
Compare
Brindrajsinh-Chauhan
force-pushed
the
rewind-fix
branch
from
2b936a7 to
7b99c03
Compare
Brindrajsinh-Chauhan
force-pushed
the
rewind-fix
branch
from
7b99c03 to
7db6543
Compare
jimthematrix
approved these changes
May 7, 2024
core/blockchain.go
Outdated
| @@ -137,18 +137,22 @@ type CacheConfig struct { | |||
| SnapshotLimit int // Memory allowance (MB) to use for caching snapshot entries in memory | |||
| Preimages bool // Whether to store preimage of trie key to the disk | |||
|
|
|||
| SnapshotNoBuild bool // Whether the background generation is allowed | |||
| SnapshotWait bool // Wait for snapshot construction on startup. TODO(karalabe): This is a dirty hack for testing, nuke it | |||
| AllowForceUpdate bool // Enable to force snapshots based on commit counts | |||
There was a problem hiding this comment.
Suggested change
| AllowForceUpdate bool // Enable to force snapshots based on commit counts | |
| AllowForceUpdate bool // Enable to force root snapshots based on the configured commits threshold |
core/blockchain.go
Outdated
| SnapshotNoBuild bool // Whether the background generation is allowed | ||
| SnapshotWait bool // Wait for snapshot construction on startup. TODO(karalabe): This is a dirty hack for testing, nuke it | ||
| AllowForceUpdate bool // Enable to force snapshots based on commit counts | ||
| CommitThreshold int // Number of commits to force a root snapshot |
There was a problem hiding this comment.
Suggested change
| CommitThreshold int // Number of commits to force a root snapshot | |
| CommitThreshold int // Threshold of commits to force a root snapshot update |
core/blockchain.go
Outdated
| @@ -1368,6 +1374,7 @@ func (bc *BlockChain) writeBlockWithState(block *types.Block, receipts []*types. | |||
|
|
|||
| current := block.NumberU64() | |||
| // Flush limits are not considered for the first TriesInMemory blocks. | |||
| log.Debug("Trie in memory", "current", current, "inMemory", TriesInMemory) | |||
There was a problem hiding this comment.
would this be a large struct to print? if so maybe change to Trace level
core/state/snapshot/difflayer.go
Outdated
| @@ -75,6 +75,9 @@ var ( | |||
| bloomDestructHasherOffset = 0 | |||
| bloomAccountHasherOffset = 0 | |||
| bloomStorageHasherOffset = 0 | |||
|
|
|||
| // Count for number of commits before fore disk root update | |||
There was a problem hiding this comment.
Suggested change
| // Count for number of commits before fore disk root update | |
| // Count for number of commits before forcing disk root update |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the ability to force a diskRoot update after a time threshold. This can be enabled by the following in the config file
AllowForceUpdate- To enable/disable Force update of the stateCommitThreshold- Number of state changes before force updateCurrent Scenario
The snapshot layers maintains a layer of 128 different block roots. Blocks with no transactions have the same root thus do not get updated to the layers. Snapshots beyond that point are flattened into one. Flattened layer is only merged with the snapshot on the disk once the memory of the flattened layers reaches the limit of ~4MB. This could take a lot of time. When a node falls over or has a unclean shutdown and enters a missing head state during start up, it rewinds back to this diskRoot, which could mean a significant roll back based on the activity/size of transactions.
Proposed Scenario
This PR adds a forcing function that is triggered based on commit threshold. After the first 128 layers of snapshot, the flattened layers are merged with the diskRoot when either the memory limit is reached or the commit threshold has been crossed. This allows more frequent update of the diskRoot thus potentially preventing large roll backs
Related eth-boot PR - https://github.com/kaleido-io/photic-eth-boot/pull/112