Implement trie version 1

[tomaka]

cc #1967

The first version of trie is called version 0. In version 1 of the trie, values that are longer than 32 bytes are hashed. Additionally, the header of each trie element has some additional bit patterns that indicate whether the value is hashed or not.

In order to know whether to use v0 or v1 of the trie, the runtime spec (found in the runtime) has an additional field called state_version (which was already parsed by smoldot since #1971 but ignored) indicating the version.

For this reason, it is no longer appropriate to have a standalone calculate_genesis_block_header function. Calculating the genesis block header requires knowing information from the genesis runtime, and building the genesis runtime is a very expensive operation. Instead, the genesis block header can now be obtained by higher-level code by building the genesis ChainInformation.

Note that technically each individual item of the trie is either v0 or v1. In other words, a trie can be in "hybrid" mode with some items in v0 and some items in v1. This happens if the chain was using v0 in the past and has switched to v1. When a chain switches to v1 (by modifying the state_version field), the updates are done lazily when items are written to the storage.
This means that Merkle proofs can also contain a hybrid of v0 and v1 items.

I have not implemented the full-node-related changes, as they are not straight forward and require some changes to the database. At the moment, the full node will error if you use it on a chain with a v1 trie or try to switch to v1. The places that need an update have been appropriately marked with TODOs.

[mergify]

Automatically approving tomaka's pull requests. This auto-approval will be removed once more maintainers are active.

[github-actions]

twiggy diff report

Difference in .wasm size before and after this pull request.

---

 Delta Bytes │ Item
─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────
       +3314 ┊ smoldot::chain_spec::ChainSpec::as_chain_information::h5af0365bfc8b6662
       +2006 ┊ smoldot::trie::node_value::calculate_merkle_value::h0869a432ff12862a
       +1959 ┊ smoldot::trie::node_value::calculate_merkle_value::h25fe45d9b07708d9
       -1958 ┊ smoldot::chain_spec::ChainSpec::as_chain_information::h1795e00deb6a354f
       +1895 ┊ smoldot::trie::node_value::calculate_merkle_value::h77cef173fc2509c8
       +1829 ┊ smoldot_light_base::Client<TChain,TPlat>::add_chain::h6ec7c3c3c36ad7e2
       +1751 ┊ smoldot::trie::node_value::calculate_merkle_value::he17c2697509c7df5
       +1751 ┊ smoldot::trie::node_value::calculate_merkle_value::hfd20de82dec28c16
       -1481 ┊ smoldot::trie::node_value::calculate_merkle_value::h2306a2433831ba87
       -1468 ┊ smoldot::trie::node_value::calculate_merkle_value::h51b1db0d76779a3e
       -1415 ┊ smoldot::trie::node_value::calculate_merkle_value::h9cfa5ddd866e52c1
       +1397 ┊ smoldot::trie::proof_node_decode::decode::h99d2925f7b6410e9
       -1355 ┊ smoldot::trie::node_value::calculate_merkle_value::h130dcffe60986cae
       -1355 ┊ smoldot::trie::node_value::calculate_merkle_value::hc39de610a4c69c1c
       +1225 ┊ smoldot::trie::node_value::calculate_merkle_value::hb8538d00532e1be4
       -1120 ┊ smoldot::trie::proof_node_decode::decode::h8805cc7ef11002d6
        +851 ┊ <smoldot::chain_spec::FromGenesisStorageError as core::fmt::Display>::fmt::hbcc5fbb75d25d9ee
        -820 ┊ smoldot::trie::node_value::calculate_merkle_value::h4bdce292d75c72f7
        -812 ┊ smoldot::calculate_genesis_block_header::h4ba388fd524cae16
        -802 ┊ <smoldot::chain_spec::FromGenesisStorageError as core::fmt::Display>::fmt::h349cf25b85e2c2f3
       +2738 ┊ ... and 75 more.
       +9567 ┊ Σ [95 Total Rows]

[melekes]

👍

[melekes]

💪 🏋 🥇

[vivekvpandya]

@tomaka once completely switched to V1 , is it possible to go back by setting state_version to 0?

[vivekvpandya]

@tomaka once completely switched to V1 , is it possible to go back by setting state_version to 0?

Any thoughts on this?