fix(store/v2/pebble): handle version 0 in keys #22524
Conversation
📝 WalkthroughWalkthroughThe pull request introduces modifications to the Changes
Assessment against linked issues
Possibly related PRs
Suggested labels
Suggested reviewers
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
| const ( | ||
| oneIf64Bit = ^uint(0) >> 63 | ||
| maxUint32OrInt = (1<<31)<<oneIf64Bit - 1 | ||
| maxVarintLen32 = 5 |
There was a problem hiding this comment.
these consts are duplicated from the pebbledb library for size calculation.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (5)
store/v2/storage/pebbledb/batch.go (3)
23-27: Consider adding documentation for platform-specific constantsThe constants are well-designed for handling platform-specific size limits. Consider adding comments explaining the bit manipulation logic for better maintainability.
const ( + // oneIf64Bit is 1 on 64-bit platforms and 0 on 32-bit platforms oneIf64Bit = ^uint(0) >> 63 + // maxUint32OrInt is math.MaxUint32 on 32-bit platforms and math.MaxInt on 64-bit platforms maxUint32OrInt = (1<<31)<<oneIf64Bit - 1 + // maxVarintLen32 is the maximum length of a varint-encoded 32-bit integer maxVarintLen32 = 5 )
29-31: Document size calculation componentsThe size calculation is correct but would benefit from documentation explaining the components of the calculation.
+// keyValueSize returns the total size needed to store a key-value pair, +// including the overhead for varint encoding and metadata func keyValueSize(key, value []byte) int { return len(key) + len(value) + 1 + 2*maxVarintLen32 }
65-78: Improve error message for size limit exceeded caseThe size management logic is well-implemented, but the error message could be more informative by including the actual sizes.
if err := b.batch.Commit(&pebble.WriteOptions{Sync: b.sync}); err != nil { - return fmt.Errorf("max batch size exceed: failed to write PebbleDB batch: %w", err) + return fmt.Errorf("batch size limit exceeded (current: %d, new: %d, max: %d): failed to write PebbleDB batch: %w", + b.size, size, maxUint32OrInt, err) }store/v2/storage/pebbledb/iterator.go (1)
Line range hint
238-244: Fix incorrect variable used in tombstone decoding.There's a bug in the tombstone decoding logic where it uses
vBz(version bytes) instead oftombBz(tombstone bytes).Apply this fix:
var tombstone uint64 if len(tombBz) > 0 { - tombstone, err = decodeUint64Ascending(vBz) + tombstone, err = decodeUint64Ascending(tombBz) if err != nil { panic(fmt.Errorf("failed to decode value tombstone: %w", err)) } }store/v2/storage/pebbledb/db.go (1)
438-444: Consider explicit handling of version 0 comparison.While the implementation correctly handles empty version bytes, consider adding explicit handling for the case where both the target version and key version are 0 to make the logic more robust.
Consider this modification:
var keyVersion uint64 // handle version 0 (no version prefix) if len(vBz) > 0 { keyVersion, err = decodeUint64Ascending(vBz) if err != nil { return nil, fmt.Errorf("failed to decode key version: %w", err) } +} else if version == 0 { + // Special case: both target and key versions are 0 + keyVersion = 0 }
📜 Review details
Configuration used: .coderabbit.yml
Review profile: CHILL
📒 Files selected for processing (3)
store/v2/storage/pebbledb/batch.go(3 hunks)store/v2/storage/pebbledb/db.go(2 hunks)store/v2/storage/pebbledb/iterator.go(1 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
store/v2/storage/pebbledb/batch.go (1)
Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.
store/v2/storage/pebbledb/db.go (1)
Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.
store/v2/storage/pebbledb/iterator.go (1)
Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.
🔇 Additional comments (5)
store/v2/storage/pebbledb/batch.go (3)
20-21: LGTM: Size tracking field addition
The addition of the size field to track cumulative batch size is well-designed and aligns with the PR's objective of handling large data sizes.
48-48: LGTM: Proper size initialization
The size initialization correctly accounts for the version key-value pair using the new helper function.
65-73: Verify handling of large genesis data
The auto-commit mechanism should handle the large genesis data case mentioned in the PR objectives. Let's verify this with a test case.
store/v2/storage/pebbledb/iterator.go (1)
226-232: LGTM: Version 0 handling looks correct.
The changes properly handle the case where a key has no version prefix (version 0) by checking the length of the version bytes before attempting to decode.
Let's verify the version 0 handling with the following script:
store/v2/storage/pebbledb/db.go (1)
241-247: LGTM! Robust handling of version 0 keys.
The implementation correctly handles both versioned and unversioned keys by checking the version byte slice length before decoding. This prevents the "insufficient bytes to decode uint64" error mentioned in issue #22523.
Let's verify the version handling implementation:
* main: build(deps): Bump cosmossdk.io/math from 1.3.0 to 1.4.0 (#22580) fix(server/v2/api/telemetry): enable global metrics (#22571) refactor(server/v2/cometbft): add `codec.Codec` and clean-up APIs (#22566) feat(core/coretesting): make memDB satisfy db.Db interface (#22570) Merge commit from fork fix(server(/v2)): fix fallback genesis path (#22564) fix: only overwrite context chainID when necessary (#22568) docs(client): Update setFeeGranter and setFeePayer comments (#22526) fix(baseapp): populate header info in `NewUncachedContext` (#22557) build(deps): Bump buf.build/gen/go/cometbft/cometbft/protocolbuffers/go from 1.35.1-20240701160653-fedbb9acfd2f.1 to 1.35.2-20240701160653-fedbb9acfd2f.1 in /api (#22551) build(deps): Bump github.com/creachadair/atomicfile from 0.3.5 to 0.3.6 in /tools/confix (#22552) docs: Update reference of Approximation (#22550) fix(server/v2/comebft): wire missing services + fix simulation (#21964) ci: fix permissions for GITHUB_TOKEN on dependabot workflows (#22547) ci: fix permissions for GITHUB_TOKEN in spell check workflow (#22545) build(deps): Bump google.golang.org/protobuf from 1.35.1 to 1.35.2 (#22537) fix(cosmovisor): premature upgrade on restart (#22528) fix(store/v2/pebble): handle version 0 in keys (#22524) refactor(server/v2/telemetry): swap redirects (#22520) docs: Update content in CODE_OF_CONDUCT.md (#22518)
Description
Closes: #22523
Fixes handling of version 0 in keys. Version 0 is supported on the write side, but surprisingly not on read.
Also includes a fix for very large batches; I exceeded the 4gb limit in InitGenesis in load testing.
Author Checklist
All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.
I have...
!in the type prefix if API or client breaking changeCHANGELOG.mdReviewers Checklist
All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.
Please see Pull Request Reviewer section in the contributing guide for more information on how to review a pull request.
I have...
Summary by CodeRabbit