db: refactor compaction iteration interfaces for deferring value retrieval #4197
Comments
jbowens
added a commit
to jbowens/pebble
that referenced
this issue
Dec 12, 2024
During compaction iteration, defer the retrieval of values stored in value blocks until either: a) we need to copy the value before stepping the iterator, or b) the compaction iterator yields the value to the main compaction loop. This refactor ensures that when a KV is elided by a tombstone, we avoid unnecessarily loading its value from the external value block. Additionally, refactoring the compaction iterator interfaces to propagate LazyValues will be used by value separation (cockroachdb#112) in which we will sometimes propagate external value references to output sstables without ever retrieving the value. Close cockroachdb#4197. Informs cockroachdb#112.
jbowens
added a commit
to jbowens/pebble
that referenced
this issue
Dec 12, 2024
During compaction iteration, defer the retrieval of values stored in value blocks until either: a) we need to copy the value before stepping the iterator, or b) the compaction iterator yields the value to the main compaction loop. This refactor ensures that when a KV is elided by a tombstone, we avoid unnecessarily loading its value from the external value block. Additionally, refactoring the compaction iterator interfaces to propagate LazyValues will be used by value separation (cockroachdb#112) in which we will sometimes propagate external value references to output sstables without ever retrieving the value. Close cockroachdb#4197. Informs cockroachdb#112.
jbowens
added a commit
to jbowens/pebble
that referenced
this issue
Dec 13, 2024
During compaction iteration, defer the retrieval of values stored in value blocks until either: a) we need to copy the value before stepping the iterator, or b) the compaction iterator yields the value to the main compaction loop. This refactor ensures that when a KV is elided by a tombstone, we avoid unnecessarily loading its value from the external value block. Additionally, refactoring the compaction iterator interfaces to propagate LazyValues will be used by value separation (cockroachdb#112) in which we will sometimes propagate external value references to output sstables without ever retrieving the value. Close cockroachdb#4197. Informs cockroachdb#112.
jbowens
added a commit
to jbowens/pebble
that referenced
this issue
Dec 13, 2024
During compaction iteration, defer the retrieval of values stored in value blocks until either: a) we need to copy the value before stepping the iterator, or b) the compaction iterator yields the value to the main compaction loop. This refactor ensures that when a KV is elided by a tombstone, we avoid unnecessarily loading its value from the external value block. Additionally, refactoring the compaction iterator interfaces to propagate LazyValues will be used by value separation (cockroachdb#112) in which we will sometimes propagate external value references to output sstables without ever retrieving the value. Close cockroachdb#4197. Informs cockroachdb#112.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
With #112, we will sometimes copy a blob value reference as-is, without ever retrieving the referenced value. Today the compaction loop and iterator greedily load the value. This issue tracks work to refactor the compaction iterator interfaces to propagate LazyValues and references as necessary.
Jira issue: PEBBLE-313
The text was updated successfully, but these errors were encountered: