Revisit story around unknown digest log items

[tomaka]

Fix #2475

While Substrate supports Aura and Babe, other teams are free to extend Substrate to support other consensus algorithms (such as Nimbus). This PR revisits our strategy regarding these other consensus algorithms.

Before this PR, block headers that contain items that aren't supported by smoldot would simply fail to parse, with some whitelisted exceptions such as Polkadot parachains, Beefy, and Frontier. These exceptions were added for consensus algorithms that don't have any influence on whether a block is valid or not.

After this PR, we now successfully parse block headers even if they contain unrecognized consensus engines. However, the blocks verification code will fail if it is asked to verify a block header with an unrecognized consensus engine.

In order to understand this change, one must note that the parachain code of smoldot doesn't actually verify blocks. It simply follows the blocks that the relay chain considers as the best block for that parachain. If the relay chain says a block is valid, then the parachain code will consider that block as valid as well. It never actually verifies them itself.

This means that the change in this PR now enables parachains with unrecognized consensus engines.

Relay/standalone chains with unrecognized consensus engines, however, still aren't supported. After this PR, they will simply fail to verify blocks rather than fail to parse blocks.

A slightly related change that I've sneaked in this PR is how we handle chains that don't use any consensus engine.
Before this PR, chains that don't use any consensus engine are marked as AllAuthorized and will have see all incoming headers accepted. This can potentially lead to a DoS vector, as anyone can just create millions of fake blocks that are all considered as valid.
After this PR, chains that don't use any consensus engine are marked as Unknown and will now fail all block verifications. Again, this doesn't have any influence on parachains as the parachains code doesn't verify blocks anyway.

[mergify]

Automatically approving tomaka's pull requests. This auto-approval will be removed once more maintainers are active.

[tomaka]

Ok I'm a bit stupid. I've just realized that this breaks everything, since Beefy/Polkadot parachains are now considered as unknown consensus engines and thus blocks validation will fail.

I think that failing all block verifications on unknown consensus engine is overly restrictive.
As commented here, all we need is a guarantee that the number of blocks is bounded, and Aura/Babe are enough for that.
Will adjust the PR.

[github-actions]

twiggy diff report

Difference in .wasm size before and after this pull request.

---

 Delta Bytes │ Item
─────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       +2342 ┊ smoldot::header::DigestItemRef::scale_encoding::h64263ffb55a9341f
       -2297 ┊ smoldot::header::DigestItemRef::scale_encoding::hd67821761cebd7b9
       -2221 ┊ smoldot::header::decode_item::hea8a1eff51e9a4c0
       -2128 ┊ smoldot::executor::host::HostVmPrototype::run_vectored::h2a2659a70b9c38b5
       +2128 ┊ smoldot::executor::host::HostVmPrototype::run_vectored::h6676c0404661779f
       +2068 ┊ <alloc::vec::Vec<T,A> as core::clone::Clone>::clone::hf1781dd9b82e9cc6
       -2049 ┊ <core::iter::adapters::chain::Chain<A,B> as core::iter::traits::iterator::Iterator>::next::h84f3e63451241f00
       +2049 ┊ <core::iter::adapters::chain::Chain<A,B> as core::iter::traits::iterator::Iterator>::next::hb4436cf1cd464c41
       +2029 ┊ smoldot::chain::blocks_tree::verify::<impl smoldot::chain::blocks_tree::NonFinalizedTreeInner<T>>::verify::h45e35104bc5daf24
       -2022 ┊ smoldot::chain::blocks_tree::verify::<impl smoldot::chain::blocks_tree::NonFinalizedTreeInner<T>>::verify::ha26b2797d86bfeea
       +2021 ┊ smoldot::header::decode_item::hd14d3957ab16a95d
       -1924 ┊ <alloc::vec::Vec<T,A> as core::clone::Clone>::clone::hce727235546f2cd3
       -1824 ┊ smoldot::chain::blocks_tree::verify::VerifyContext<T>::apply_success_body::h590e09bd36492ad8
       +1759 ┊ smoldot::chain::blocks_tree::verify::VerifyContext<T>::apply_success_body::hc63155855600f9a4
       -1549 ┊ smoldot::header::decode_partial::h8eb9a1038db2fd19
       +1543 ┊ smoldot::header::decode_partial::hf639e46ddbddc81d
       -1397 ┊ smoldot::chain::blocks_tree::finality::<impl smoldot::chain::blocks_tree::NonFinalizedTreeInner<T>>::set_finalized_block::h0e83beecda71fd3c
       -1396 ┊ smoldot::chain::blocks_tree::finality::<impl smoldot::chain::blocks_tree::NonFinalizedTreeInner<T>>::set_finalized_block::hf16e8c6ef1d0fb78
       +1395 ┊ smoldot::chain::blocks_tree::finality::<impl smoldot::chain::blocks_tree::NonFinalizedTreeInner<T>>::set_finalized_block::hf966ca8a052d6771
       +1394 ┊ smoldot::chain::blocks_tree::finality::<impl smoldot::chain::blocks_tree::NonFinalizedTreeInner<T>>::set_finalized_block::h5f37cef3b9162e69
        -126 ┊ ... and 221 more.
       -1109 ┊ Σ [241 Total Rows]

[tomaka]

Fixed my previous comment by adding an allow_unknown_consensus_engines flag to the configuration of everything relevant.
It's set to true in the light client.

[melekes]

[sea212]

Awesome!