Make chainHead_unstable_storage more powerful
#37
Conversation
There was a problem hiding this comment.
Amazing!
This should give us parity between what substrate exposed previously wrt iterating over key storages, while having a more compact and robust API.
Hopefully, we'll be able to make a complete transition in the whole ecosystem to the new methods and deprecate the old RPC ones 👍
Co-authored-by: Alexandru Vasile <[email protected]>
Co-authored-by: Alexandru Vasile <[email protected]>
|
I'm actually not completely happy with this design where the client has to call The function achieves all its goals, but the problem is that the client can delay calling What I'm worried about it a very specific situation: if a load balancer redirects a But with the introduction of this |
|
Since it's sitting there, I just wanted to mention that I can't merge this pull request myself as I'm not at Parity anymore. |
|
|
||
| If `type` is `hash` or `descendants-hashes`, then the cryptographic hash of each item is provided rather than the full value. The hashing algorithm used is the one of the chain, which is typically blake2b. This can lead to significantly less bandwidth usage and can be used in order to compare the value of an item with a known hash and querying the full value only if it differs. | ||
|
|
||
| If `type` is `closest-ancestor-merkle-value`, then the so-called trie Merkle value of the `key` is provided. If `key` doesn't exist in the trie, then the Merkle value of the closest ancestor of `key` is provided. Contrary to `hash`, a `closest-ancestor-merkle-value` always exists for every `key`. The Merkle value is similar to a hash of the value and all of its descendants together. |
There was a problem hiding this comment.
If I understand it correctly, I find the naming "hard/confusing". Why not "closest-ancestor-merkle-node" and then also saying that this is the hash of the closest merkle node.
There was a problem hiding this comment.
I kept the same vocabulary as the spec. There's no such thing as a "merkle node". There are "trie nodes" that have a "node value" and a "merkle value" (the merkle value is either the node value or its hash).
"closest ancestor merkle value" means "the merkle value of the closest ancestor".
There was a problem hiding this comment.
I kept the same vocabulary as the spec.
Okay 👍
(the merkle value is either the node value or its hash).
What is the node value? The value attached to a node? Shouldn't we always be interested in the hash as we only use this for tracking changes?
There was a problem hiding this comment.
The node value is (basically) concat(node_type, partial_key, storage_value, merkle_values_of_children).
If the node value is less than 32 bytes then the Merkle value is the same as the node value. If the node value is 32 bytes or more, then it's the hash.
https://spec.polkadot.network/chap-state#defn-merkle-value
There was a problem hiding this comment.
Ahh okay, ty for the explanation 👍
Close #21
Even though they're unrelated, this PR is based on top of #35. I expect #35 to be merged, and didn't want to have conflicts.
This PR enhances
chainHead_unstable_storageby adding back thetypefield that was removed in #17.You can now query not just the value of a storage item, but also the hash of its value, the values of the item and all its descendants, the hashes of item and all its descendants, or the Merkle value of the item or its closest ancestor.
Querying the descendants of the storage item makes it possible to enumerate all the keys under a certain prefix, which is important to know the content of maps.
In order to reproduce something similar to
state_getKeysPaged, in the case when you enumerate descendants the server can now generate awaiting-for-continueevent, after which the client must callchainHead_unstable_storageContinuein order to resume the enumeration. This makes it possible for the client to interrupt the enumeration at any point in time. Rather than letting the client decide of the number of items (like is the case forstate_getKeysPaged), it's now the server which decides, which makes it possible to decide to generate this event based on the size of what was sent rather than the number of items.The
closest-ancestor-merkle-valuequery type makes it possible to know when the content of a map has changed. It can be seen as similar tohash, except thathashis only the hash of the value whileclosest-ancestor-merkle-valueis the hash of the value plus the hash of all the descendants in the trie.Basically, in order to watch a map, a JSON-RPC client would periodically query the
closest-ancestor-merkle-valueof the map, then redownload the list of keys if this Merkle value changes.