Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite the warp syncing code to use call proofs #2578

Merged
merged 62 commits into from
Aug 5, 2022
Merged
Changes from 1 commit
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
60627e9
VirtualMachineParamsGet -> ChainInfoQuery
tomaka Aug 1, 2022
2036e45
Add call proofs requests to AllSync
tomaka Aug 1, 2022
474e521
Start adding a requests system to warp_sync.rs
tomaka Aug 2, 2022
f413a21
warp_sync_source no longer public
tomaka Aug 2, 2022
8323a7f
Start adding call proof requests
tomaka Aug 2, 2022
7e880ec
Provide block_hash in warp_sync::RequestDetail
tomaka Aug 2, 2022
4efb51c
Fix implementation in sync_service
tomaka Aug 2, 2022
5c197e3
Send the call proofs
tomaka Aug 2, 2022
13abcd4
Store the call proof responses
tomaka Aug 3, 2022
9fc643e
Use the call proofs
tomaka Aug 3, 2022
dd479e9
Remove StorageGet and NextKey variants from warp_sync
tomaka Aug 3, 2022
07a70df
Add RequestDetail::WarpSyncRequest
tomaka Aug 3, 2022
1ecb2b9
Inline PostVerificationState
tomaka Aug 3, 2022
46ec8b2
Add a Phase enum to ChainInfoQuery
tomaka Aug 3, 2022
cbe5707
Do the warp sync as part of ChainInfoQuery
tomaka Aug 3, 2022
d632883
Remove GrandpaWarpSync and Verifier variants
tomaka Aug 3, 2022
c3a157b
Remove ChainInfoQuery::current_source
tomaka Aug 3, 2022
1ca81c4
Remove source_id field, instead get source on the fly
tomaka Aug 3, 2022
30a546f
Remove WaitingForSources struct
tomaka Aug 3, 2022
77cb3ee
Merge InProgressWarpSync and ChainInfoQuery into one
tomaka Aug 3, 2022
ff59fd9
Remove warp_sync_header()
tomaka Aug 3, 2022
656e5a4
Have a run() function that drives the CPU process
tomaka Aug 4, 2022
5470748
Properly transition requests
tomaka Aug 4, 2022
ff1a01f
Update inject_error function
tomaka Aug 4, 2022
dc91084
as_chain_information is in fact correct
tomaka Aug 4, 2022
f41a52a
Add warp_sync::Config::requests_capacity
tomaka Aug 4, 2022
5d86d54
Update remove_source
tomaka Aug 4, 2022
dfd2b75
Store the raw downloaded runtime code and compile later
tomaka Aug 4, 2022
1b0c670
Store the downloaded fragments and verify them in run()
tomaka Aug 4, 2022
71319c6
Small tweak to run()
tomaka Aug 4, 2022
97c742d
handle_response_ok no longer takes ownership
tomaka Aug 4, 2022
3f816aa
Tweaks to downloaded_proof
tomaka Aug 4, 2022
5a0316c
Tweak desired_requests so that rustfmt is happy
tomaka Aug 4, 2022
fd437dd
Small tweaks to desired_requests
tomaka Aug 4, 2022
8b47c71
Add a TRq user data for requests
tomaka Aug 4, 2022
991f734
Remove functions a bit
tomaka Aug 4, 2022
7471b55
Split RequestDetail from DesiredRequest, to remove state_trie_root field
tomaka Aug 4, 2022
324a9f5
Use Cow for the function name and parameters
tomaka Aug 4, 2022
1d654b5
Split PreVerification phase in two
tomaka Aug 4, 2022
c459589
Immediately create a Verifier after download
tomaka Aug 4, 2022
424d9d1
Extract body of run() to different sub-structs
tomaka Aug 4, 2022
53612d7
run() -> process_one() and tweaks
tomaka Aug 4, 2022
3b424d0
BuildChainInformation::verify() -> build()
tomaka Aug 4, 2022
e1eb68f
Fix one todo in VerifyWarpSyncFragment::verify
tomaka Aug 4, 2022
b66e95e
Restore verifying fragments one by one
tomaka Aug 4, 2022
781d8e4
Only verify one fragment at a time in warp_sync
tomaka Aug 4, 2022
c7f18fe
CHANGELOG
tomaka Aug 4, 2022
04010db
Remove all remaining todo!()s from warp_sync.rs
tomaka Aug 4, 2022
d1dafbc
Fix most warnings
tomaka Aug 4, 2022
9c3016a
Properly check block hash on warp sync request success
tomaka Aug 4, 2022
7d27437
Try again sources if all sources banned
tomaka Aug 4, 2022
b7f881f
Tweak usage
tomaka Aug 4, 2022
0b54af2
Properly remove obsolete requests when removing source
tomaka Aug 4, 2022
79b0f7e
Improve docs somewhat
tomaka Aug 4, 2022
3eaaf67
Move process_one to the bottom
tomaka Aug 4, 2022
48d12d5
PR number
tomaka Aug 4, 2022
93c18c5
Rustfmt
tomaka Aug 4, 2022
c5c959f
Fix doclink
tomaka Aug 4, 2022
e8f20f4
Fix spellcheck
tomaka Aug 4, 2022
53f64ba
Properly change phase if multiple sets of fragments to download
tomaka Aug 4, 2022
39a0273
No longer ignore responses from banned sources, to prevent infinite loop
tomaka Aug 5, 2022
31f2c3f
Merge branch 'main' into warp-sync-call-proofs
mergify[bot] Aug 5, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Properly transition requests
tomaka committed Aug 4, 2022

Verified

This commit was signed with the committer’s verified signature.
tomaka Pierre Krieger
commit 5470748759b32c7f4b689a3d203f4d9f7c0a904e
68 changes: 58 additions & 10 deletions src/sync/all.rs
Original file line number Diff line number Diff line change
@@ -2499,6 +2499,59 @@ impl<TRq> Shared<TRq> {
.iter()
.all(|(_, s)| matches!(s, SourceMapping::GrandpaWarpSync(_))));

// TODO: O(n2)
for (source_id, inner_id, detail) in grandpa.in_progress_requests {
let (_, self_request) = self
.requests
.iter_mut()
.find(|(_, rq)| match rq {
RequestMapping::WarpSync(id, _) if *id == inner_id => true,
_ => false,
})
.unwrap();

// TODO: DRY
let detail = match detail {
warp_sync::RequestDetail::WarpSyncRequest { block_hash } => {
RequestDetail::GrandpaWarpSync {
sync_start_block_hash: block_hash,
}
}
warp_sync::RequestDetail::RuntimeParametersGet { block_hash } => {
RequestDetail::StorageGet {
block_hash,
state_trie_root: [0; 32], // TODO: wrong /!\ needs more refactoring to remove the field entirely
keys: vec![b":code".to_vec(), b":heappages".to_vec()],
}
}
warp_sync::RequestDetail::RuntimeCallMerkleProof {
block_hash,
function_name,
parameter_vectored,
} => RequestDetail::RuntimeCallMerkleProof {
block_hash,
function_name,
parameter_vectored,
},
};

let (source_id, _) = self
.sources
.iter()
.find(|(_, s)| match s {
SourceMapping::GrandpaWarpSync(s) if *s == source_id => true,
_ => false,
})
.unwrap();

match mem::replace(self_request, RequestMapping::Poisoned) {
RequestMapping::WarpSync(_, user_data) => {
*self_request = RequestMapping::Inline(SourceId(source_id), detail, user_data)
}
_ => unreachable!(),
};
}

for source in grandpa.sources {
let source_user_data = AllForksSourceExtra {
user_data: source.user_data,
@@ -2521,20 +2574,14 @@ impl<TRq> Shared<TRq> {
self.sources[source.outer_source_id.0] = SourceMapping::AllForks(updated_source_id);
}

// TODO: convert requests
/*for (_, request) in self.requests.iter_mut() {
match request {
RequestMapping::WarpSync(_, user_data) => {
*request = RequestMapping::Inline(_, _, user_data)
}
_ => {}
}
}*/

debug_assert!(self
.sources
.iter()
.all(|(_, s)| matches!(s, SourceMapping::AllForks(_))));
debug_assert!(self
.requests
.iter()
.all(|(_, s)| matches!(s, RequestMapping::AllForks(..) | RequestMapping::Inline(..))));

(
all_forks,
@@ -2551,6 +2598,7 @@ enum RequestMapping<TRq> {
AllForks(all_forks::RequestId),
Optimistic(optimistic::RequestId),
WarpSync(warp_sync::RequestId, TRq), // TODO: move TRq to warp sync state machine
Poisoned,
}

#[derive(Debug, Clone, PartialEq, Eq)]
10 changes: 10 additions & 0 deletions src/sync/warp_sync.rs
Original file line number Diff line number Diff line change
@@ -163,6 +163,9 @@ pub struct Success<TSrc> {

/// The list of sources that were added to the state machine.
pub sources: Vec<TSrc>,

/// The list of requests that were added to the state machine.
pub in_progress_requests: Vec<(SourceId, RequestId, RequestDetail)>,
}

/// The warp sync state machine.
@@ -617,6 +620,11 @@ impl<TSrc> InProgressWarpSync<TSrc> {
.drain()
.map(|source| source.user_data)
.collect(),
in_progress_requests: self
.in_progress_requests
.iter()
.map(|(id, (src_id, detail))| (*src_id, RequestId(id), detail.clone()))
.collect(),
}),
None,
);
@@ -803,6 +811,7 @@ struct Source<TSrc> {
already_tried: bool,
}

#[derive(Debug, Clone)]
pub enum DesiredRequest {
WarpSyncRequest {
block_hash: [u8; 32],
@@ -818,6 +827,7 @@ pub enum DesiredRequest {
},
}

#[derive(Debug, Clone)]
pub enum RequestDetail {
WarpSyncRequest {
block_hash: [u8; 32],