Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite the warp syncing code to use call proofs #2578

Merged
merged 62 commits into from
Aug 5, 2022

Conversation

tomaka
Copy link
Contributor

@tomaka tomaka commented Aug 4, 2022

Fix #2080

cc #933 #2529 #2222

The initial intention behind this PR was to make it use call proofs rather than individual storage gets. See #2080.

However, this turned into a bigger refactoring as I've realized that a bigger refactoring is very much needed and would simplify the code a lot.

The warp syncing code now uses an API similar to all_forks and to optimistic, the two other syncing strategies.
The warp syncing code was the first syncing strategy to be written in smoldot, and at that time we didn't have the experience of the caveats of its approaches.
I don't think there's much to comment, given that the API is now extremely close to what already exists.

Apart from using call proofs, I've opted to keep the exact same behavior as today, even though it is a bit questionable. In particular, when we download fragments from a peer, we then always query the same peer for the storage items. If that peer disconnects, we try download fragments again. As mentioned, I've kept this behavior, but the refactoring will make it much easier to improve compared to before this PR.

Can be reviewed commit by commit if desired. I've taken an incremental approach, and everything still works after almost every commit.

Copy link
Contributor

@mergify mergify bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automatically approving tomaka's pull requests. This auto-approval will be removed once more maintainers are active.

@tomaka
Copy link
Contributor Author

tomaka commented Aug 4, 2022

The biggest change outside of warp_sync.rs is probably the duplication of the RequestDetail struct in two: RequestDetail and DesiredRequest.
DesiredRequest is what the syncing state machine returns, while RequestDetail is what the API user passes as parameter.

The only difference is in RequestDetail::StorageGet vs DesiredRequest::StorageGet. The DesiredRequest one contains an extra state_root_hash field that isn't in the RequestDetail one. That's because this field is redundant with the block_hash field that is already present.

It makes sense for the syncing state machine to provide both a block_hash and a state_root_hash, but it is weird for the API user to pass both, because then we have to deal with situations where the block_hash doesn't correspond to the state_root_hash.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 4, 2022

twiggy diff report

Difference in .wasm size before and after this pull request.


 Delta Bytes │ Item
─────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────
       +5761 ┊ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h348c2f834e7f4df5
       -4821 ┊ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h4a855022f88f540a
       +4555 ┊ smoldot::sync::warp_sync::BuildChainInformation<TSrc,TRq>::build::hb1d3de1a59359d1a
       -4247 ┊ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hfb5acd4fba69ed4c
       -4205 ┊ smoldot_light_base::sync_service::standalone::Task<TPlat>::inject_network_event::h02a25692f02fadb8
       +4205 ┊ smoldot_light_base::sync_service::standalone::Task<TPlat>::inject_network_event::h67917a7887111770
       +3729 ┊ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hb0941db9eb9de92d
       +2728 ┊ smoldot_light_base::sync_service::standalone::Task<TPlat>::start_next_request::hffe8c8c5d20b7229
       -2447 ┊ smoldot::sync::all::AllSync<TRq,TSrc,TBl>::storage_get_response::h9cb8170ffb92ca9b
       -2161 ┊ smoldot_light_base::sync_service::standalone::Task<TPlat>::start_next_request::h88a224083a6643a1
       +2101 ┊ smoldot::sync::all::Shared<TRq>::transition_grandpa_warp_sync_all_forks::h33dd7b0aa1c2d05a
       +1915 ┊ smoldot::sync::warp_sync::InProgressWarpSync<TSrc,TRq>::desired_requests::h79602654acd9ccd0
       +1801 ┊ smoldot::sync::all::AllSync<TRq,TSrc,TBl>::blocks_request_response::h34dd58a32b7fcdc0
       -1788 ┊ smoldot::sync::all::AllSync<TRq,TSrc,TBl>::blocks_request_response::h7f4cb693aa535cb3
       -1646 ┊ smoldot::sync::warp_sync::WarpSync<TSrc>::from_babe_fetch_epoch_query::h7f5b00c20c92eddd
       +1594 ┊ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h0808283f380530d9
       +1434 ┊ smoldot::sync::all::AllSync<TRq,TSrc,TBl>::add_request::h8e8de2d148dd85ad
       -1404 ┊ smoldot::sync::warp_sync::VirtualMachineParamsGet<TSrc>::set_virtual_machine_params::he153bca29a946af7
       -1350 ┊ smoldot::sync::all::AllSync<TRq,TSrc,TBl>::desired_requests::ha0d59c87bc4c53ef
       +1345 ┊ smoldot::trie::proof_verify::trie_node_info::h2f2ac3563e51dd3e
      +20380 ┊ ... and 560 more.
      +52198 ┊ Σ [580 Total Rows]

Copy link
Contributor

@melekes melekes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tomaka tomaka added the automerge Automatically merge pull request as soon as possible label Aug 5, 2022
@mergify mergify bot merged commit 6b26ed6 into paritytech:main Aug 5, 2022
@tomaka tomaka deleted the warp-sync-call-proofs branch August 5, 2022 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automerge Automatically merge pull request as soon as possible
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use a call proof in the Grandpa warp syncing instead of multiple storage proofs
2 participants