Elastic scaling: use an assumed `CoreIndex` in `candidate-backing` #3229

sandreim · 2024-02-06T16:19:50Z

First step in implementing #3144

Summary of changes

switch statement Table candidate mapping from ParaId to CoreIndex
introduce experimental InjectCoreIndex node feature.
determine and assume a CoreIndex for a candidate based on statement validator index. If the signature is valid it means validator controls the validator that index and we can easily map it to a validator group/core.
introduce a temporary provisioner fix until we fully enable elastic scaling in the subystem. The fix ensures we don't fetch the same backable candidate when calling get_backable_candidate for each core.

TODO:

fix backing tests
fix statement table tests
add new test

Signed-off-by: Andrei Sandu <[email protected]>

…reim/backing_multiple_cores_per_para

Signed-off-by: Andrei Sandu <[email protected]>

alindima

Looks good! Nice work!

Cargo.lock

polkadot/node/core/backing/src/lib.rs

polkadot/node/core/provisioner/src/lib.rs

polkadot/primitives/src/v6/mod.rs

polkadot/primitives/src/vstaging/mod.rs

Signed-off-by: Andrei Sandu <[email protected]>

polkadot/statement-table/src/generic.rs

polkadot/node/core/backing/src/lib.rs

polkadot/node/core/backing/src/tests/mod.rs

Signed-off-by: Andrei Sandu <[email protected]>

eskimor · 2024-02-14T21:16:37Z

polkadot/node/core/backing/src/lib.rs

+	/// The validator groups at this relay parent.
+	validator_groups: Vec<Vec<ValidatorIndex>>,
+	/// The associated group rotation information.
+	group_rotation_info: GroupRotationInfo,


Those two are really per session. Storing them per relay parent is rather unnecessary. Seems fine for an interim solution though.

eskimor · 2024-02-14T21:17:56Z

polkadot/node/core/backing/src/lib.rs

+	if inject_core_index {
+		let core_index_to_inject: BitVec<u8, BitOrderLsb0> =
+			BitVec::from_vec(vec![core_index.0 as u8]);
+		validator_indices.extend(core_index_to_inject);


Shouldn't we be using some nice BackedCandidate function to be introduced in the other PR for that?

that's a good idea, will do that in the follow-up PR

polkadot/node/core/backing/src/lib.rs

…ltiple_cores_per_para

Signed-off-by: alindima <[email protected]>

…ltiple_cores_per_para

polkadot/node/core/backing/src/lib.rs

eskimor · 2024-02-21T17:08:24Z

polkadot/node/core/backing/src/lib.rs

-	/// The validator groups at this relay parent.
-	validator_groups: Vec<Vec<ValidatorIndex>>,
+	/// The validator index -> group mapping at this relay parent.
+	validator_to_group: IndexedVec<ValidatorIndex, Option<GroupIndex>>,


Is it really necessary to copy this to the relay parent state? Can't we just keep the session index here and then fetch a reference from state directly where we need it?

this being an LruMap, it makes it a bit harder to reason about the existence of the session index in the map.

And we can't store a mutable reference here and just populate the map when the session is not present because we'd need to hand out multiple mutable references.

Realistically, backing can only happen on the current session, right? So the LruMap doesn't need a capacity larger than 1 (so then, it doesn't even need to be a LruMap). Am I right?

with current async backing code, we cannot start backing a candidate in the next session in advance. But we may want to keep this possibility, making the LruMap neccessary.

What I can do is wrap it in an Arc, because they don't need to modify it. Which would achieve the same

done. I'll merge it once CI passes again

…rent cores (#3231) Fixes #3144 Builds on top of #3229 ### Summary Some preparations for Runtime to support elastic scaling, guarded by config node features bit `FeatureIndex::ElasticScalingMVP`. This PR introduces a per-candidate `CoreIndex` but does it in a hacky way to avoid changing `CandidateCommitments`, `CandidateReceipts` primitives and networking protocols. #### Including `CoreIndex` in `BackedCandidate` If the `ElasticScalingMVP` feature bit is enabled then `BackedCandidate::validator_indices` is extended by 8 bits. The value stored in these bits represents the assumed core index for the candidate. It is temporary solution which works by creating a mapping from `BackedCandidate` to `CoreIndex` by assuming the `CoreIndex` can be discovered by checking in which validator group the validator that signed the statement is. TODO: - [x] fix tests - [x] add new tests - [x] Bump runtime API for Kusama, so we have that node features thing! -> polkadot-fellows/runtimes#194 --------- Signed-off-by: Andrei Sandu <[email protected]> Signed-off-by: alindima <[email protected]> Co-authored-by: alindima <[email protected]>

…3229) First step in implementing #3144 - switch statement `Table` candidate mapping from `ParaId` to `CoreIndex` - introduce experimental `InjectCoreIndex` node feature. - determine and assume a `CoreIndex` for a candidate based on statement validator index. If the signature is valid it means validator controls the validator that index and we can easily map it to a validator group/core. - introduce a temporary provisioner fix until we fully enable elastic scaling in the subystem. The fix ensures we don't fetch the same backable candidate when calling `get_backable_candidate` for each core. TODO: - [x] fix backing tests - [x] fix statement table tests - [x] add new test --------- Signed-off-by: Andrei Sandu <[email protected]> Signed-off-by: alindima <[email protected]> Co-authored-by: alindima <[email protected]>

…rent cores (#3231) Fixes #3144 Builds on top of #3229 Some preparations for Runtime to support elastic scaling, guarded by config node features bit `FeatureIndex::ElasticScalingMVP`. This PR introduces a per-candidate `CoreIndex` but does it in a hacky way to avoid changing `CandidateCommitments`, `CandidateReceipts` primitives and networking protocols. If the `ElasticScalingMVP` feature bit is enabled then `BackedCandidate::validator_indices` is extended by 8 bits. The value stored in these bits represents the assumed core index for the candidate. It is temporary solution which works by creating a mapping from `BackedCandidate` to `CoreIndex` by assuming the `CoreIndex` can be discovered by checking in which validator group the validator that signed the statement is. TODO: - [x] fix tests - [x] add new tests - [x] Bump runtime API for Kusama, so we have that node features thing! -> polkadot-fellows/runtimes#194 --------- Signed-off-by: Andrei Sandu <[email protected]> Signed-off-by: alindima <[email protected]> Co-authored-by: alindima <[email protected]>

…3229) First step in implementing #3144 ### Summary of changes - switch statement `Table` candidate mapping from `ParaId` to `CoreIndex` - introduce experimental `InjectCoreIndex` node feature. - determine and assume a `CoreIndex` for a candidate based on statement validator index. If the signature is valid it means validator controls the validator that index and we can easily map it to a validator group/core. - introduce a temporary provisioner fix until we fully enable elastic scaling in the subystem. The fix ensures we don't fetch the same backable candidate when calling `get_backable_candidate` for each core. TODO: - [x] fix backing tests - [x] fix statement table tests - [x] add new test --------- Signed-off-by: Andrei Sandu <[email protected]> Signed-off-by: alindima <[email protected]> Co-authored-by: alindima <[email protected]>

…rent cores (#3231) Fixes #3144 Builds on top of #3229 ### Summary Some preparations for Runtime to support elastic scaling, guarded by config node features bit `FeatureIndex::ElasticScalingMVP`. This PR introduces a per-candidate `CoreIndex` but does it in a hacky way to avoid changing `CandidateCommitments`, `CandidateReceipts` primitives and networking protocols. #### Including `CoreIndex` in `BackedCandidate` If the `ElasticScalingMVP` feature bit is enabled then `BackedCandidate::validator_indices` is extended by 8 bits. The value stored in these bits represents the assumed core index for the candidate. It is temporary solution which works by creating a mapping from `BackedCandidate` to `CoreIndex` by assuming the `CoreIndex` can be discovered by checking in which validator group the validator that signed the statement is. TODO: - [x] fix tests - [x] add new tests - [x] Bump runtime API for Kusama, so we have that node features thing! -> polkadot-fellows/runtimes#194 --------- Signed-off-by: Andrei Sandu <[email protected]> Signed-off-by: alindima <[email protected]> Co-authored-by: alindima <[email protected]>

sandreim added 7 commits February 1, 2024 22:29

Switch statement table from ParaId to CoreIndex

cab6ab9

Signed-off-by: Andrei Sandu <[email protected]>

cargo lock

d2df658

Signed-off-by: Andrei Sandu <[email protected]>

add experimental feature

4a8b8d5

Signed-off-by: Andrei Sandu <[email protected]>

inject core_index from statements

9244632

Signed-off-by: Andrei Sandu <[email protected]>

temporary provisioner fix

22e017b

Signed-off-by: Andrei Sandu <[email protected]>

Merge branch 'master' of github.com:paritytech/polkadot-sdk into sand…

574b06a

…reim/backing_multiple_cores_per_para

cargo lock

fbb7351

Signed-off-by: Andrei Sandu <[email protected]>

sandreim mentioned this pull request Feb 6, 2024

Runtime: allow backing multiple candidates of same parachain on different cores #3231

Merged

3 tasks

It was damn hard to fix these tests

6c72918

Signed-off-by: Andrei Sandu <[email protected]>

sandreim added R0-silent Changes should not be mentioned in any release notes T0-node This PR/Issue is related to the topic “node”. T8-polkadot This PR/Issue is related to/affects the Polkadot network. labels Feb 12, 2024

sandreim marked this pull request as ready for review February 12, 2024 12:39

sandreim added 5 commits February 12, 2024 14:47

These tests were easy to fix

fc5c109

Signed-off-by: Andrei Sandu <[email protected]>

Fix comment

33351b4

Signed-off-by: Andrei Sandu <[email protected]>

clippy was angry

0d994bf

Signed-off-by: Andrei Sandu <[email protected]>

A bit refactor and add a test

534c019

Signed-off-by: Andrei Sandu <[email protected]>

taplo happy

10d86dd

Signed-off-by: Andrei Sandu <[email protected]>

alindima reviewed Feb 14, 2024

View reviewed changes

review feedback

222609c

Signed-off-by: Andrei Sandu <[email protected]>

alindima approved these changes Feb 14, 2024

View reviewed changes

remove log

a02e896

Signed-off-by: Andrei Sandu <[email protected]>

eskimor reviewed Feb 14, 2024

View reviewed changes

polkadot/statement-table/src/generic.rs Show resolved Hide resolved

polkadot/node/core/backing/src/lib.rs Outdated Show resolved Hide resolved

polkadot/node/core/backing/src/tests/mod.rs Show resolved Hide resolved

more feedback

dd34850

Signed-off-by: Andrei Sandu <[email protected]>

eskimor reviewed Feb 14, 2024

View reviewed changes

alindima added 5 commits February 19, 2024 11:19

Merge remote-tracking branch 'origin/master' into sandreim/backing_mu…

838a846

…ltiple_cores_per_para

use next up on available instead of occupied core index

ad98f18

ElasticScalingCoreIndex -> ElasticScalingMVP

606d7c4

small nits and typos

c793b89

Signed-off-by: alindima <[email protected]>

Merge remote-tracking branch 'origin/master' into sandreim/backing_mu…

8398bb1

…ltiple_cores_per_para

alindima force-pushed the sandreim/backing_multiple_cores_per_para branch from 7bd84bd to 8398bb1 Compare February 20, 2024 15:59

Merge remote-tracking branch 'origin/master' into sandreim/backing_mu…

9f70276

…ltiple_cores_per_para

eskimor reviewed Feb 21, 2024

View reviewed changes

polkadot/node/core/backing/src/lib.rs Show resolved Hide resolved

polkadot/node/core/backing/src/lib.rs Outdated Show resolved Hide resolved

alindima mentioned this pull request Feb 21, 2024

backing: improve session buffering for runtime information #3421

Closed

3 tasks

cache Validator->Group mapping

9c3dd5c

eskimor reviewed Feb 21, 2024

View reviewed changes

eskimor approved these changes Feb 21, 2024

View reviewed changes

use Arc to avoid cloning

af1cd82

alindima added this pull request to the merge queue Feb 22, 2024

Merged via the queue into master with commit 60e537b Feb 22, 2024
129 of 130 checks passed

alindima deleted the sandreim/backing_multiple_cores_per_para branch February 22, 2024 08:47

This was referenced Feb 26, 2024

Fix multiple cores 1.8 #3477

Merged

Fix multiple cores assignments 1.7 #3469

Merged

sandreim mentioned this pull request Feb 28, 2024

Testnet HostConfiguration migration (v11 to v12) #3503

Closed

patriciobcs self-assigned this Apr 9, 2024

CanonicalJP unassigned patriciobcs Apr 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elastic scaling: use an assumed `CoreIndex` in `candidate-backing` #3229

Elastic scaling: use an assumed `CoreIndex` in `candidate-backing` #3229

sandreim commented Feb 6, 2024 •

edited by sw10pa

Loading

alindima left a comment

eskimor Feb 14, 2024

eskimor Feb 14, 2024

alindima Feb 19, 2024

eskimor Feb 21, 2024

alindima Feb 22, 2024

alindima Feb 22, 2024

alindima Feb 22, 2024

Elastic scaling: use an assumed CoreIndex in candidate-backing #3229

Elastic scaling: use an assumed CoreIndex in candidate-backing #3229

Conversation

sandreim commented Feb 6, 2024 • edited by sw10pa Loading

Summary of changes

alindima left a comment

Choose a reason for hiding this comment

eskimor Feb 14, 2024

Choose a reason for hiding this comment

eskimor Feb 14, 2024

Choose a reason for hiding this comment

alindima Feb 19, 2024

Choose a reason for hiding this comment

eskimor Feb 21, 2024

Choose a reason for hiding this comment

alindima Feb 22, 2024

Choose a reason for hiding this comment

alindima Feb 22, 2024

Choose a reason for hiding this comment

alindima Feb 22, 2024

Choose a reason for hiding this comment

Elastic scaling: use an assumed `CoreIndex` in `candidate-backing` #3229

Elastic scaling: use an assumed `CoreIndex` in `candidate-backing` #3229

sandreim commented Feb 6, 2024 •

edited by sw10pa

Loading