You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Snapshot is always created and processed in order, taking advantage of that, we can apply delta encodings to some fields:
version
IAVL tree has the nice property that parent node's version is always bigger than or equal to children's version, and in snapshot, nodes are added in post-order, where parent node is put after children nodes, so we can only record the delta value, to be specific parent version - max(left version, right version), smaller value results in smaller encoded size with varint encoding.
height
since IAVL tree has the invariant that parent height = max(left height, right height) + 1, I think we don't need to encode the height field at all in the snapshot.
The version and height compression rely on the fact that nodes are exported in post-order. the key compression only need the leaf nodes are exported in order.
The text was updated successfully, but these errors were encountered:
yihuang
changed the title
snapshot: apply delta encoding
snapshot: apply delta encoding for better compression rate
Feb 23, 2023
Make the IAVL library always support all the legacy versions, so an introduction of a new format can be considered as a non-breaking change, let cosmos-sdk handle the versioning of the snapshot in application.
change the import/export APIs to be format aware, and dealing with raw bytes directly, let IAVL library handle the serialization of ExportNode, it'll be more convenient to introduce new formats.
it's a API breaking change, but not behavior breaking.
add a new api to return the list of supported formats in current version, SDK can check the snapshot format against it, sdk should add a new config option for user to specify which format is used to export a snapshot.
yihuang
changed the title
snapshot: apply delta encoding for better compression rate
export: apply delta encoding for better compression rate
Feb 23, 2023
Snapshot is always created and processed in order, taking advantage of that, we can apply delta encodings to some fields:
IAVL tree has the nice property that parent node's version is always bigger than or equal to children's version, and in snapshot, nodes are added in post-order, where parent node is put after children nodes, so we can only record the delta value, to be specific
parent version - max(left version, right version)
, smaller value results in smaller encoded size with varint encoding.After the export: don't export branch node's key in snapshot #688 improvement, we only have leaf keys in snapshot, they are ordered, so delta encoding can also potentially have some good compression rate there.
since IAVL tree has the invariant that
parent height = max(left height, right height) + 1
, I think we don't need to encode the height field at all in the snapshot.The version and height compression rely on the fact that nodes are exported in post-order. the key compression only need the leaf nodes are exported in order.
The text was updated successfully, but these errors were encountered: