Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

export: apply delta encoding for better compression rate #690

Closed
yihuang opened this issue Feb 22, 2023 · 1 comment · Fixed by #703
Closed

export: apply delta encoding for better compression rate #690

yihuang opened this issue Feb 22, 2023 · 1 comment · Fixed by #703

Comments

@yihuang
Copy link
Collaborator

yihuang commented Feb 22, 2023

Snapshot is always created and processed in order, taking advantage of that, we can apply delta encodings to some fields:

  • version
    IAVL tree has the nice property that parent node's version is always bigger than or equal to children's version, and in snapshot, nodes are added in post-order, where parent node is put after children nodes, so we can only record the delta value, to be specific parent version - max(left version, right version), smaller value results in smaller encoded size with varint encoding.
  • key
    After the export: don't export branch node's key in snapshot #688 improvement, we only have leaf keys in snapshot, they are ordered, so delta encoding can also potentially have some good compression rate there.
  • height
    since IAVL tree has the invariant that parent height = max(left height, right height) + 1, I think we don't need to encode the height field at all in the snapshot.

The version and height compression rely on the fact that nodes are exported in post-order. the key compression only need the leaf nodes are exported in order.

@yihuang yihuang changed the title snapshot: apply delta encoding snapshot: apply delta encoding for better compression rate Feb 23, 2023
@yihuang
Copy link
Collaborator Author

yihuang commented Feb 23, 2023

Some thoughts about snapshot format migration:

  • Make the IAVL library always support all the legacy versions, so an introduction of a new format can be considered as a non-breaking change, let cosmos-sdk handle the versioning of the snapshot in application.
  • change the import/export APIs to be format aware, and dealing with raw bytes directly, let IAVL library handle the serialization of ExportNode, it'll be more convenient to introduce new formats.
    • it's a API breaking change, but not behavior breaking.
  • add a new api to return the list of supported formats in current version, SDK can check the snapshot format against it, sdk should add a new config option for user to specify which format is used to export a snapshot.

@yihuang yihuang changed the title snapshot: apply delta encoding for better compression rate export: apply delta encoding for better compression rate Feb 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant