Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

omni-node: add metadata checks for runtime/parachain compatibility #6450

Merged
merged 71 commits into from
Dec 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
9cd1a7c
(wip) cumulus/omninode-lib: parachain system check
iulianbarbu Oct 5, 2024
0053b5a
polish a bit the pallet existance check
iulianbarbu Oct 23, 2024
f0055ea
break line length
iulianbarbu Oct 24, 2024
8b99816
adjust message length
iulianbarbu Oct 24, 2024
350985d
wip: testing why metadata decoding doesn't work
iulianbarbu Oct 29, 2024
40141d1
fix runtime metadata decoding
iulianbarbu Nov 11, 2024
0631349
remove left over chain spec
iulianbarbu Nov 12, 2024
00757ef
merge with master
iulianbarbu Nov 12, 2024
5cbcb26
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Nov 14, 2024
149defd
made small adjustments with comments
iulianbarbu Nov 14, 2024
1f8128a
check frame-metadata storage block number
iulianbarbu Nov 15, 2024
bbe6e42
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Nov 15, 2024
f869f32
added tests
iulianbarbu Nov 17, 2024
36861c0
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Nov 17, 2024
6740b97
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Nov 18, 2024
fe0887c
add docs for runtime conventions
iulianbarbu Nov 18, 2024
337ac48
add prdoc & remove leftover dir
iulianbarbu Nov 18, 2024
2d40797
remove decode feature
iulianbarbu Nov 18, 2024
5849e12
fix some weird Cargo.toml changes
iulianbarbu Nov 18, 2024
3dc55ad
fix compilation
iulianbarbu Nov 19, 2024
5583014
some more fixes
iulianbarbu Nov 19, 2024
8945e9d
fix linting
iulianbarbu Nov 19, 2024
cc0b3d1
fix typo
iulianbarbu Nov 19, 2024
c30917d
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Nov 19, 2024
18aa6c7
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Nov 27, 2024
993d264
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Nov 27, 2024
6e747c1
udpate Cargo.lock
iulianbarbu Nov 27, 2024
f3a779f
revert litep2p version
iulianbarbu Nov 27, 2024
5f969cd
fix Cargo.lock
iulianbarbu Nov 27, 2024
08576d5
Merge branch 'master' of github.com:paritytech/polkadot-sdk into ib-o…
iulianbarbu Nov 28, 2024
3f2cef3
revert Cargo.toml
iulianbarbu Nov 28, 2024
ab564ba
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Nov 28, 2024
58790cc
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Nov 29, 2024
ba0096a
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Nov 29, 2024
c5a6d09
revert to master Cargo.lock
iulianbarbu Nov 29, 2024
b657f81
corrected doc link
iulianbarbu Nov 29, 2024
fcfa988
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Nov 30, 2024
8df94e5
update prdoc
iulianbarbu Nov 30, 2024
8c2c693
check metadata earlier
iulianbarbu Dec 4, 2024
ea95f47
update benchmarking-cli
iulianbarbu Dec 4, 2024
ccd8187
fixed scrambled code
iulianbarbu Dec 4, 2024
a5c71ef
style fixes
iulianbarbu Dec 4, 2024
55d84d1
add sp-io dev dep
iulianbarbu Dec 5, 2024
4523b1c
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Dec 5, 2024
384bc82
updated Cargo.toml
iulianbarbu Dec 5, 2024
ce713f9
fix umbrella
iulianbarbu Dec 5, 2024
7e97c7a
update cargo.lock
iulianbarbu Dec 5, 2024
3581f46
more style fixes
iulianbarbu Dec 5, 2024
0c5f785
redo import
iulianbarbu Dec 5, 2024
11c39dd
taplo format
iulianbarbu Dec 5, 2024
e064430
umbrella fixes
iulianbarbu Dec 5, 2024
c858bfe
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Dec 5, 2024
b425b6c
more umbrella Cargo.toml fmt
iulianbarbu Dec 5, 2024
08e63bd
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Dec 5, 2024
ec9b04b
Update cumulus/polkadot-omni-node/lib/src/common/runtime.rs
iulianbarbu Dec 5, 2024
0ed4f59
Update cumulus/polkadot-omni-node/lib/src/common/runtime.rs
iulianbarbu Dec 5, 2024
9279e6e
add new crate to prdoc
iulianbarbu Dec 6, 2024
cf29ed2
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Dec 6, 2024
0ba6ecb
fixes from seb & michal reviews
iulianbarbu Dec 10, 2024
5e1ef1e
update prdoc
iulianbarbu Dec 10, 2024
1aa4d3d
rephrased public do
iulianbarbu Dec 10, 2024
c6e9f8c
remove redundant dependency
iulianbarbu Dec 11, 2024
0261c3b
avoid exposing APIs around metadata inspection for now
iulianbarbu Dec 11, 2024
1cb5361
Update docs/sdk/src/reference_docs/omni_node.rs
iulianbarbu Dec 11, 2024
4d469da
Update docs/sdk/src/reference_docs/omni_node.rs
iulianbarbu Dec 11, 2024
6354748
this builds fine too
iulianbarbu Dec 11, 2024
16a43e5
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Dec 11, 2024
4fd9df5
updated prdoc
iulianbarbu Dec 11, 2024
2a7b404
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Dec 12, 2024
6ae6615
Merge branch 'master' into ib-omni-node-ps-check
iulianbarbu Dec 12, 2024
7deb995
change back formatting in Cargo.toml
iulianbarbu Dec 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,7 @@ members = [
"substrate/client/rpc-api",
"substrate/client/rpc-servers",
"substrate/client/rpc-spec-v2",
"substrate/client/runtime-utilities",
"substrate/client/service",
"substrate/client/service/test",
"substrate/client/state-db",
Expand Down Expand Up @@ -1183,6 +1184,7 @@ sc-rpc-api = { path = "substrate/client/rpc-api", default-features = false }
sc-rpc-server = { path = "substrate/client/rpc-servers", default-features = false }
sc-rpc-spec-v2 = { path = "substrate/client/rpc-spec-v2", default-features = false }
sc-runtime-test = { path = "substrate/client/executor/runtime-test" }
sc-runtime-utilities = { path = "substrate/client/runtime-utilities", default-features = true }
sc-service = { path = "substrate/client/service", default-features = false }
sc-service-test = { path = "substrate/client/service/test" }
sc-state-db = { path = "substrate/client/state-db", default-features = false }
Expand Down Expand Up @@ -1316,6 +1318,7 @@ substrate-test-runtime-transaction-pool = { path = "substrate/test-utils/runtime
substrate-test-utils = { path = "substrate/test-utils" }
substrate-wasm-builder = { path = "substrate/utils/wasm-builder", default-features = false }
subxt = { version = "0.38", default-features = false }
subxt-metadata = { version = "0.38.0", default-features = false }
subxt-signer = { version = "0.38" }
syn = { version = "2.0.87" }
sysinfo = { version = "0.30" }
Expand Down
16 changes: 10 additions & 6 deletions cumulus/polkadot-omni-node/lib/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,13 @@ docify = { workspace = true }
# Local
jsonrpsee = { features = ["server"], workspace = true }
parachains-common = { workspace = true, default-features = true }
scale-info = { workspace = true }
subxt-metadata = { workspace = true, default-features = true }

# Substrate
frame-benchmarking = { optional = true, workspace = true, default-features = true }
frame-benchmarking-cli = { workspace = true, default-features = true }
sp-crypto-hashing = { workspace = true }
sp-runtime = { workspace = true }
sp-core = { workspace = true, default-features = true }
sp-session = { workspace = true, default-features = true }
Expand All @@ -57,12 +60,16 @@ sc-rpc = { workspace = true, default-features = true }
sp-version = { workspace = true, default-features = true }
sp-weights = { workspace = true, default-features = true }
sc-tracing = { workspace = true, default-features = true }
sc-runtime-utilities = { workspace = true, default-features = true }
sp-storage = { workspace = true, default-features = true }
frame-system-rpc-runtime-api = { workspace = true, default-features = true }
pallet-transaction-payment = { workspace = true, default-features = true }
pallet-transaction-payment-rpc-runtime-api = { workspace = true, default-features = true }
sp-inherents = { workspace = true, default-features = true }
sp-api = { workspace = true, default-features = true }
sp-consensus-aura = { workspace = true, default-features = true }
sp-io = { workspace = true, default-features = true }
sp-wasm-interface = { workspace = true, default-features = true }
sc-consensus-manual-seal = { workspace = true, default-features = true }
sc-sysinfo = { workspace = true, default-features = true }
prometheus-endpoint = { workspace = true, default-features = true }
Expand Down Expand Up @@ -93,15 +100,12 @@ assert_cmd = { workspace = true }
nix = { features = ["signal"], workspace = true }
tokio = { version = "1.32.0", features = ["macros", "parking_lot", "time"] }
wait-timeout = { workspace = true }
cumulus-test-runtime = { workspace = true }

[features]
default = []
rococo-native = [
"polkadot-cli/rococo-native",
]
westend-native = [
"polkadot-cli/westend-native",
]
rococo-native = ["polkadot-cli/rococo-native"]
westend-native = ["polkadot-cli/westend-native"]
runtime-benchmarks = [
"cumulus-primitives-core/runtime-benchmarks",
"frame-benchmarking-cli/runtime-benchmarks",
Expand Down
147 changes: 144 additions & 3 deletions cumulus/polkadot-omni-node/lib/src/common/runtime.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,19 @@

//! Runtime parameters.

use codec::Decode;
use cumulus_client_service::ParachainHostFunctions;
use sc_chain_spec::ChainSpec;
use sc_executor::WasmExecutor;
use sc_runtime_utilities::fetch_latest_metadata_from_code_blob;
use scale_info::{form::PortableForm, TypeDef, TypeDefPrimitive};
use std::fmt::Display;
use subxt_metadata::{Metadata, StorageEntryType};

/// Expected parachain system pallet runtime type name.
pub const DEFAULT_PARACHAIN_SYSTEM_PALLET_NAME: &str = "ParachainSystem";
/// Expected frame system pallet runtime type name.
pub const DEFAULT_FRAME_SYSTEM_PALLET_NAME: &str = "System";

/// The Aura ID used by the Aura consensus
#[derive(PartialEq)]
Expand All @@ -35,14 +47,42 @@ pub enum Consensus {
}

/// The choice of block number for the parachain omni-node.
#[derive(PartialEq)]
#[derive(PartialEq, Debug)]
pub enum BlockNumber {
/// u32
U32,
/// u64
U64,
}

impl Display for BlockNumber {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
BlockNumber::U32 => write!(f, "u32"),
BlockNumber::U64 => write!(f, "u64"),
}
}
}

impl Into<TypeDefPrimitive> for BlockNumber {
fn into(self) -> TypeDefPrimitive {
match self {
BlockNumber::U32 => TypeDefPrimitive::U32,
BlockNumber::U64 => TypeDefPrimitive::U64,
}
}
}

impl BlockNumber {
fn from_type_def(type_def: &TypeDef<PortableForm>) -> Option<BlockNumber> {
match type_def {
TypeDef::Primitive(TypeDefPrimitive::U32) => Some(BlockNumber::U32),
TypeDef::Primitive(TypeDefPrimitive::U64) => Some(BlockNumber::U64),
_ => None,
}
}
}

/// Helper enum listing the supported Runtime types
#[derive(PartialEq)]
pub enum Runtime {
Expand All @@ -62,7 +102,108 @@ pub trait RuntimeResolver {
pub struct DefaultRuntimeResolver;

impl RuntimeResolver for DefaultRuntimeResolver {
fn runtime(&self, _chain_spec: &dyn ChainSpec) -> sc_cli::Result<Runtime> {
Ok(Runtime::Omni(BlockNumber::U32, Consensus::Aura(AuraConsensusId::Sr25519)))
fn runtime(&self, chain_spec: &dyn ChainSpec) -> sc_cli::Result<Runtime> {
let metadata_inspector = MetadataInspector::new(chain_spec)?;
let block_number = match metadata_inspector.block_number() {
Some(inner) => inner,
None => {
log::warn!(
r#"⚠️ There isn't a runtime type named `System`, corresponding to the `frame-system`
pallet (https://docs.rs/frame-system/latest/frame_system/). Please check Omni Node docs for runtime conventions:
https://paritytech.github.io/polkadot-sdk/master/polkadot_sdk_docs/reference_docs/omni_node/index.html#runtime-conventions.
Note: We'll assume a block number size of `u32`."#
);
BlockNumber::U32
},
};

if !metadata_inspector.pallet_exists(DEFAULT_PARACHAIN_SYSTEM_PALLET_NAME) {
log::warn!(
r#"⚠️ The parachain system pallet (https://docs.rs/crate/cumulus-pallet-parachain-system/latest) is
missing from the runtime’s metadata. Please check Omni Node docs for runtime conventions:
https://paritytech.github.io/polkadot-sdk/master/polkadot_sdk_docs/reference_docs/omni_node/index.html#runtime-conventions."#
);
}

Ok(Runtime::Omni(block_number, Consensus::Aura(AuraConsensusId::Sr25519)))
}
}

struct MetadataInspector(Metadata);

impl MetadataInspector {
fn new(chain_spec: &dyn ChainSpec) -> Result<MetadataInspector, sc_cli::Error> {
MetadataInspector::fetch_metadata(chain_spec).map(MetadataInspector)
}

fn pallet_exists(&self, name: &str) -> bool {
self.0.pallet_by_name(name).is_some()
}

fn block_number(&self) -> Option<BlockNumber> {
let pallet_metadata = self.0.pallet_by_name(DEFAULT_FRAME_SYSTEM_PALLET_NAME);
pallet_metadata
.and_then(|inner| inner.storage())
.and_then(|inner| inner.entry_by_name("Number"))
.and_then(|number_ty| match number_ty.entry_type() {
StorageEntryType::Plain(ty_id) => Some(ty_id),
_ => None,
})
.and_then(|ty_id| self.0.types().resolve(*ty_id))
.and_then(|portable_type| BlockNumber::from_type_def(&portable_type.type_def))
}

fn fetch_metadata(chain_spec: &dyn ChainSpec) -> Result<Metadata, sc_cli::Error> {
let mut storage = chain_spec.build_storage()?;
let code_bytes = storage
.top
.remove(sp_storage::well_known_keys::CODE)
.ok_or("chain spec genesis does not contain code")?;
let opaque_metadata = fetch_latest_metadata_from_code_blob(
&WasmExecutor::<ParachainHostFunctions>::builder()
.with_allow_missing_host_functions(true)
.build(),
sp_runtime::Cow::Borrowed(code_bytes.as_slice()),
)
.map_err(|err| err.to_string())?;

Metadata::decode(&mut (*opaque_metadata).as_slice()).map_err(Into::into)
}
}

#[cfg(test)]
mod tests {
use crate::runtime::{
BlockNumber, MetadataInspector, DEFAULT_FRAME_SYSTEM_PALLET_NAME,
DEFAULT_PARACHAIN_SYSTEM_PALLET_NAME,
};
use codec::Decode;
use cumulus_client_service::ParachainHostFunctions;
use sc_executor::WasmExecutor;
use sc_runtime_utilities::fetch_latest_metadata_from_code_blob;

fn cumulus_test_runtime_metadata() -> subxt_metadata::Metadata {
let opaque_metadata = fetch_latest_metadata_from_code_blob(
&WasmExecutor::<ParachainHostFunctions>::builder()
.with_allow_missing_host_functions(true)
.build(),
sp_runtime::Cow::Borrowed(cumulus_test_runtime::WASM_BINARY.unwrap()),
)
.unwrap();

subxt_metadata::Metadata::decode(&mut (*opaque_metadata).as_slice()).unwrap()
}

#[test]
fn test_pallet_exists() {
let metadata_inspector = MetadataInspector(cumulus_test_runtime_metadata());
assert!(metadata_inspector.pallet_exists(DEFAULT_PARACHAIN_SYSTEM_PALLET_NAME));
assert!(metadata_inspector.pallet_exists(DEFAULT_FRAME_SYSTEM_PALLET_NAME));
}
iulianbarbu marked this conversation as resolved.
Show resolved Hide resolved

#[test]
fn test_runtime_block_number() {
let metadata_inspector = MetadataInspector(cumulus_test_runtime_metadata());
assert_eq!(metadata_inspector.block_number().unwrap(), BlockNumber::U32);
}
}
Loading
Loading