Bridges - Add improved congestion control mechanism #6231

bkontur · 2024-10-25T08:51:14Z

Closes: #5551
Closes: #5550

Context

Before permissionless lanes, bridges only supported hard-coded, static lanes. The congestion mechanism was based on sending Transact(report_bridge_status(is_congested)) from pallet-xcm-bridge-hub to pallet-xcm-bridge-hub-router. Depending on is_congested, we adjusted the fee factor to increase or decrease fees. This congestion mechanism relied on monitoring XCMP queues, which could cause issues like suspending the entire XCMP queue rather than just the affected bridge.

Additionally, we are progressing with deploying bridge message pallets/routing directly on AssetHub, where we don’t interact with XCMP to perform ExportXcm locally.

Description

This PR re-introduces and improves congestion for bridges:

Enhanced Bridge Congestion Mechanism: The bridge queue mechanism has been restructured to operate independently of XCMP, with a refined protocol for congestion detection and suspension management.
Bridge-Specific Channel Suspension: pallet-xcm-bridge-hub and pallet-xcm-bridge-hub-router now use BridgeId to identify specific bridges, enabling selective suspension and resumption of individual bridge channels.
Dynamic Congestion Detection: pallet-xcm-bridge-hub now includes callbacks for fn suspend_bridge and fn resume_bridge based on congestion status:
- For sibling chains, the router sends xcm::Transact(report_bridge_status(bridge_id, is_congested)) using the stored callback information.
- For local chain deployments, the router manages state directly.
New Stop Threshold: A stop_threshold limit in pallet-xcm-bridge-hub enables or disables ExportXcm::validate, providing a fallback mechanism when the router does not adhere to the suspend signal.
Flexible Message Routing: pallet-xcm-bridge-hub-router has been refactored to support message routing for both sibling chains (ExportMessage) and local deployment (ExportXcm).

These updates improve modularity, allow more granular bridge congestion handling, and support diverse deployment scenarios.

bkontur · 2024-10-25T11:25:35Z

bot fmt
/cmd prdoc --audience runtime_dev --bump patch

prdoc/pr_6231.prdoc

bkontur · 2024-10-26T20:30:05Z

bot fmt

bkontur · 2024-10-28T15:04:20Z

bot fmt

bkontur · 2024-11-05T12:40:39Z

bot fmt

bkontur · 2024-11-07T15:57:44Z

/cmd bench --runtime asset-hub-westend asset-hub-rococo --pallet pallet_xcm_bridge_hub_router

bkontur · 2024-11-07T16:18:32Z

bot bench cumulus-assets --runtime=asset-hub-westend --pallet=pallet_xcm_bridge_hub_router
bot bench cumulus-assets --runtime=asset-hub-rococo --pallet=pallet_xcm_bridge_hub_router

bkontur · 2024-11-07T16:51:47Z

bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-assets --runtime=asset-hub-westend --pallet=pallet_xcm_bridge_hub_router
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-assets --runtime=asset-hub-rococo --pallet=pallet_xcm_bridge_hub_router

bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --runtime=bridge-hub-rococo --pallet=pallet_bridge_messages
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --runtime=bridge-hub-westend --pallet=pallet_bridge_messages

bkontur · 2024-11-07T21:51:26Z

bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --runtime=bridge-hub-rococo --pallet=pallet_bridge_messages
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --runtime=bridge-hub-westend --pallet=pallet_bridge_messages
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --subcommand=xcm --runtime=bridge-hub-rococo --pallet=pallet_xcm_benchmarks::generic
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --subcommand=xcm --runtime=bridge-hub-westend --pallet=pallet_xcm_benchmarks::generic

bkontur · 2024-11-08T23:55:04Z

bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --runtime=bridge-hub-rococo --pallet=pallet_bridge_messages
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --runtime=bridge-hub-westend --pallet=pallet_bridge_messages
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --runtime=bridge-hub-rococo --pallet=pallet_xcm_bridge_hub
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --runtime=bridge-hub-westend --pallet=pallet_xcm_bridge_hub

bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --subcommand=xcm --runtime=bridge-hub-rococo --pallet=pallet_xcm_benchmarks::generic
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --subcommand=xcm --runtime=bridge-hub-westend --pallet=pallet_xcm_benchmarks::generic

bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-assets --runtime=asset-hub-westend --pallet=pallet_xcm_bridge_hub_router
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-assets --runtime=asset-hub-rococo --pallet=pallet_xcm_bridge_hub_router

github-actions · 2024-11-21T11:54:37Z

Command "fmt" has finished ✅ See logs here

…(check Xcmp, check UMP, ..)

bkontur · 2024-11-27T12:28:35Z

/cmd fmt

github-actions · 2024-11-27T12:30:20Z

Command "fmt" has started 🚀 See logs here

github-actions · 2024-11-27T12:30:45Z

Command "fmt" has finished ✅ See logs here

acatangiu

All of the XCM transport (pallet-messages layer) will be deployed to Asset Hub instead of Bridge Hub, so would it be easier to not go through all of the effort of upgrading Bridge Hub with all this only to deprecate it on the next iteration?

What if instead of changing and renaming xcm-bridge-hub to support permissionless lanes, we just create a new pallet xcm-bridge which is what you basically have in this PR, and we revert xcm-bridge-hub to the same code actually deployed today on Bridge Hub?

That way we keep legacy code with legacy lane on BH without any migrations or risk, and all of this new code goes straight to AH, then at some point we switch the AH exporter from legacy to new?

I know this is late and maybe we should've had this idea earlier, so I'm leaving it up to you to decide which way is easiest to do.

acatangiu · 2024-11-29T13:46:30Z

bridges/modules/relayers/src/extension/mod.rs

@@ -1127,7 +1126,6 @@ mod tests {
 		Option<PreDispatchData<ThisChainAccountId, BridgedChainBlockNumber, TestLaneIdType>>,
 		TransactionValidityError,
 	> {
-		sp_tracing::try_init_simple();


nit: why remove this? it's useful in debugging, no?

acatangiu · 2024-11-29T13:56:09Z

bridges/modules/xcm-bridge-hub-router/src/impls.rs

+		// Here, we have the actual result fees covering bridge fees, so now we need to check/apply
+		// the congestion and dynamic_fees features (if possible).


Suggested change

// Here, we have the actual result fees covering bridge fees, so now we need to check/apply

// the congestion and dynamic_fees features (if possible).

// `fees` is populated with base bridge fees, now let's apply congestion/dynamic fees if required.

acatangiu · 2024-11-29T14:01:25Z

bridges/modules/xcm-bridge-hub-router/src/impls.rs

+/// Adapter implementation for [`SendXcm`] that allows adding a message size fee and/or dynamic fees
+/// based on the `BridgeId` resolved by the `T::BridgeIdResolver` resolver, if and only if `E`
+/// supports routing. This adapter can be used, for example, as a wrapper over
+/// [`xcm_builder::UnpaidLocalExporter`], enabling it to compute message and/or dynamic fees using a
+/// fee factor.


Naming of these is confusing. When/how is UnpaidLocalExporter used when there are bridge fees to be paid?

Maybe rename UnpaidLocalExporter to just LocalExporter or smth?.

Or maybe this one can completely integrate UnpaidLocalExporter instead of being a wrapper over it?

acatangiu · 2024-11-29T14:14:37Z

bridges/modules/xcm-bridge-hub-router/src/impls.rs

+				bridged_dest.clone()
+			}
+		} else {
+			// if `bridged_dest` does not contain `GlobalConsensus`, let's prepend one


is there a case when ensure_is_remote(UniversalLocation::get(), dest.clone()) returns Ok, but dest does not contain GlobalConsensus?
should we even support such a case?

acatangiu · 2024-11-29T14:31:46Z

bridges/modules/xcm-bridge-hub-router/src/lib.rs

+//!
+//! ### On the Same Chain as the Message Exporter
+//! In this setup, the router directly calls an `ExportXcm` implementation. In this case,
+//! `ViaLocalBridgeHubExporter` can be used as a wrapper with `T::ToBridgeHubSender`.


We need to rename T::ToBridgeHubSender since it's not fit for the local case, where Bridge Hub is not involved.

acatangiu · 2024-11-29T15:41:42Z

bridges/primitives/xcm-bridge-hub/src/lib.rs

+	/// `bool` - `true` means that we keep accepting messages to the bridge.
+	Suspended(bool),


I would add another enum variant instead of overcharging Suspended:

Suggested change

/// `bool` - `true` means that we keep accepting messages to the bridge.

Suspended(bool),

/// We keep accepting messages to the bridge to allow any inflight messages to be processed.

SoftSuspended,

/// Bridge is suspended and new messages are now being actively dropped.

HardSuspended,

acatangiu · 2024-11-29T15:42:49Z

bridges/modules/xcm-bridge-hub/src/exporter.rs

@@ -151,6 +144,20 @@ where
 			message,
 		)?;

+		// Here, we know that the message is relevant to this pallet instance, so let's check for
+		// congestion defense.
+		if bridge.state == BridgeState::Suspended(false) {


This would be much more readable

Suggested change

if bridge.state == BridgeState::Suspended(false) {

if bridge.state == BridgeState::HardSuspended {

acatangiu · 2024-11-29T15:44:32Z

bridges/modules/xcm-bridge-hub/src/exporter.rs

+			BridgeState::Suspended(true) => {
+				if enqueued_messages > T::CongestionLimits::get().outbound_lane_stop_threshold {
+					// If its suspended and reached `outbound_lane_stop_threshold`, we stop
+					// accepting new messages (a.k.a. start dropping).
+					Bridges::<T, I>::mutate_extant(bridge_id, |bridge| {
+						bridge.state = BridgeState::Suspended(false);
+					});
+					return


Suggested change

BridgeState::Suspended(true) => {

if enqueued_messages > T::CongestionLimits::get().outbound_lane_stop_threshold {

// If its suspended and reached `outbound_lane_stop_threshold`, we stop

// accepting new messages (a.k.a. start dropping).

Bridges::<T, I>::mutate_extant(bridge_id, |bridge| {

bridge.state = BridgeState::Suspended(false);

});

return

BridgeState::SoftSuspended => {

if enqueued_messages > T::CongestionLimits::get().outbound_lane_stop_threshold {

// If its suspended and reached `outbound_lane_stop_threshold`, we stop

// accepting new messages (a.k.a. start dropping).

Bridges::<T, I>::mutate_extant(bridge_id, |bridge| {

bridge.state = BridgeState::HardSuspended;

});

return

acatangiu · 2024-11-29T15:46:25Z

bridges/modules/xcm-bridge-hub/src/exporter.rs

+				// We cannot accept new messages to the suspended bridge, hoping that it'll be
+				// actually resumed soon


Suggested change

// We cannot accept new messages to the suspended bridge, hoping that it'll be

// actually resumed soon

// We cannot accept new messages and start dropping messages, until the outbound lane goes below the drop limit.

acatangiu · 2024-11-29T16:02:34Z

prdoc/pr_6231.prdoc

@@ -0,0 +1,45 @@
+title: Bridges - revert-back and improve congestion


Suggested change

title: Bridges - revert-back and improve congestion

title: Bridges - Add improved congestion control mechanism

Closes: #5551 ## Description With [permissionless lanes PR#4949](#4949), the congestion mechanism based on sending `Transact(report_bridge_status(is_congested))` from `pallet-xcm-bridge-hub` to `pallet-xcm-bridge-hub-router` was replaced with a congestion mechanism that relied on monitoring XCMP queues. However, this approach could cause issues, such as suspending the entire XCMP queue instead of isolating the affected bridge. This PR reverts back to using `report_bridge_status` as before. ## TODO - [x] benchmarks - [x] prdoc ## Follow-up #6231 --------- Co-authored-by: GitHub Action <[email protected]> Co-authored-by: command-bot <> Co-authored-by: Adrian Catangiu <[email protected]>

Closes: #5551 ## Description With [permissionless lanes PR#4949](#4949), the congestion mechanism based on sending `Transact(report_bridge_status(is_congested))` from `pallet-xcm-bridge-hub` to `pallet-xcm-bridge-hub-router` was replaced with a congestion mechanism that relied on monitoring XCMP queues. However, this approach could cause issues, such as suspending the entire XCMP queue instead of isolating the affected bridge. This PR reverts back to using `report_bridge_status` as before. ## TODO - [x] benchmarks - [x] prdoc ## Follow-up #6231 --------- Co-authored-by: GitHub Action <[email protected]> Co-authored-by: command-bot <> Co-authored-by: Adrian Catangiu <[email protected]> (cherry picked from commit 8f4b99c) # Conflicts: # Cargo.lock # cumulus/parachains/runtimes/assets/asset-hub-rococo/tests/tests.rs # cumulus/parachains/runtimes/assets/asset-hub-westend/src/lib.rs

Closes: #5551 ## Description With [permissionless lanes PR#4949](#4949), the congestion mechanism based on sending `Transact(report_bridge_status(is_congested))` from `pallet-xcm-bridge-hub` to `pallet-xcm-bridge-hub-router` was replaced with a congestion mechanism that relied on monitoring XCMP queues. However, this approach could cause issues, such as suspending the entire XCMP queue instead of isolating the affected bridge. This PR reverts back to using `report_bridge_status` as before. ## TODO - [x] benchmarks - [x] prdoc ## Follow-up #6231 --------- Co-authored-by: GitHub Action <[email protected]> Co-authored-by: command-bot <> Co-authored-by: Adrian Catangiu <[email protected]>

yrong · 2024-12-18T17:57:41Z

bridges/modules/xcm-bridge-hub-router/src/lib.rs

+			for (bridge_id, previous_value, bridge_state) in bridges_to_update.into_iter() {
+				let new_value = bridge_state.delivery_fee_factor;
+				log::info!(
+					target: LOG_TARGET,
+					"Bridge channel with id {:?} is uncongested. Decreasing fee factor from {} to {}!",
+					bridge_id,
+					previous_value,
+					new_value,
+				);
+				Bridges::<T, I>::insert(&bridge_id, bridge_state);
+				Self::deposit_event(Event::DeliveryFeeFactorDecreased {
+					previous_value,
+					new_value,
+					bridge_id,
+				});
 			}


Just curious why not guard the weight usage with meter in second loop, what if there is a lot of bridges to update/remove here while the weight is used up in first loop?

yrong · 2024-12-18T18:19:22Z

cumulus/parachains/runtimes/assets/asset-hub-westend/src/lib.rs

+	// For congestion - allow only calls from BH.
+	type BridgeHubOrigin =
+		AsEnsureOriginWithArg<EnsureXcm<Equals<xcm_config::bridging::SiblingBridgeHub>>>;


Seems param bridge_id is not used when check the origin here, would be simpler to just use EnsureXcm<Equals<xcm_config::bridging::SiblingBridgeHub>>>?

bkontur · 2024-12-22T21:40:49Z

/cmd fmt

github-actions · 2024-12-22T21:42:46Z

Command "fmt" has started 🚀 See logs here

github-actions · 2024-12-22T21:43:09Z

Command "fmt" has finished ✅ See logs here

paritytech-workflow-stopper · 2024-12-22T21:57:44Z

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/12457915814
Failed job name: cargo-clippy

bkontur added the T15-bridges This PR/Issue is related to bridges. label Oct 25, 2024

bkontur self-assigned this Oct 25, 2024

bkontur force-pushed the bko-bridges-congestion branch 2 times, most recently from 659be89 to b48b8a5 Compare October 25, 2024 21:14

bkontur commented Oct 26, 2024

View reviewed changes

prdoc/pr_6231.prdoc Outdated Show resolved Hide resolved

bkontur force-pushed the bko-bridges-congestion branch from a663bc2 to cbc6ae7 Compare October 26, 2024 20:29

bkontur force-pushed the bko-bridges-congestion branch from c78f9bc to 501a5c0 Compare October 28, 2024 15:01

bkontur force-pushed the bko-bridges-congestion branch 7 times, most recently from c78e707 to 152389a Compare November 5, 2024 12:33

bkontur force-pushed the bko-bridges-congestion branch 3 times, most recently from edd9c5c to 38f1bb3 Compare November 7, 2024 13:33

bkontur force-pushed the bko-bridges-congestion branch from f06433a to d329dec Compare November 7, 2024 16:44

bkontur force-pushed the bko-bridges-congestion branch from 21baa7f to c00ff6b Compare November 8, 2024 17:44

bkontur added the A4-needs-backport Pull request must be backported to all maintained releases. label Nov 11, 2024

Nits

ad0931c

bkontur added 5 commits November 21, 2024 13:14

Merge branch 'master' into bko-bridges-congestion

4c1c315

Merge remote-tracking branch 'origin/master' into bko-bridges-congestion

85dbd00

Merge remote-tracking branch 'origin/master' into bko-bridges-congestion

9ad289b

Tuple implementation for ChannelStatusProvider to cover more cases …

c93d722

…(check Xcmp, check UMP, ..)

Merge remote-tracking branch 'origin/master' into bko-bridges-congestion

8ccb773

Update from bkontur running command 'fmt'

981e3df

Merge branch 'master' into bko-bridges-congestion

91567e0

acatangiu reviewed Nov 29, 2024

View reviewed changes

bkontur mentioned this pull request Dec 7, 2024

Bridges - revert-back congestion mechanism #6781

Merged

2 tasks

acatangiu changed the title ~~Bridges - revert-back and improve congestion~~ Bridges - add new congestion control protocol for dedicated lanes Dec 9, 2024

yrong reviewed Dec 18, 2024

View reviewed changes

bkontur changed the title ~~Bridges - add new congestion control protocol for dedicated lanes~~ Bridges - Add improved congestion control mechanism Dec 19, 2024

bkontur added 2 commits December 20, 2024 22:32

Merge remote-tracking branch 'origin/master' into bko-bridges-congestion

18f31da

Merge remote-tracking branch 'origin/master' into bko-bridges-congestion

16ae079

paritytech-review-bot bot requested a review from a team December 20, 2024 21:34

bkontur added 2 commits December 22, 2024 22:38

Merge branch 'master' into bko-bridges-congestion

4569510

zepter/taplo

8e681fb

Update from bkontur running command 'fmt'

69e85cc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bridges - Add improved congestion control mechanism #6231

Bridges - Add improved congestion control mechanism #6231

bkontur commented Oct 25, 2024 •

edited

Loading

bkontur commented Oct 25, 2024

bkontur commented Oct 26, 2024

bkontur commented Oct 28, 2024

bkontur commented Nov 5, 2024

bkontur commented Nov 7, 2024

bkontur commented Nov 7, 2024

bkontur commented Nov 7, 2024

bkontur commented Nov 7, 2024

bkontur commented Nov 8, 2024

github-actions bot commented Nov 21, 2024

bkontur commented Nov 27, 2024

github-actions bot commented Nov 27, 2024

github-actions bot commented Nov 27, 2024

acatangiu left a comment

acatangiu Nov 29, 2024

acatangiu Nov 29, 2024

acatangiu Nov 29, 2024

acatangiu Nov 29, 2024

acatangiu Nov 29, 2024

acatangiu Nov 29, 2024

acatangiu Nov 29, 2024

acatangiu Nov 29, 2024

acatangiu Nov 29, 2024

acatangiu Nov 29, 2024

yrong Dec 18, 2024

yrong Dec 18, 2024

bkontur commented Dec 22, 2024

github-actions bot commented Dec 22, 2024

github-actions bot commented Dec 22, 2024

paritytech-workflow-stopper bot commented Dec 22, 2024

		// Here, we have the actual result fees covering bridge fees, so now we need to check/apply
		// the congestion and dynamic_fees features (if possible).

	// Here, we have the actual result fees covering bridge fees, so now we need to check/apply
	// the congestion and dynamic_fees features (if possible).
	// `fees` is populated with base bridge fees, now let's apply congestion/dynamic fees if required.

		/// `bool` - `true` means that we keep accepting messages to the bridge.
		Suspended(bool),

-	/// `bool` - `true` means that we keep accepting messages to the bridge.
-	Suspended(bool),
+	/// We keep accepting messages to the bridge to allow any inflight messages to be processed.
+	SoftSuspended,
+	/// Bridge is suspended and new messages are now being actively dropped.
+	HardSuspended,

	if bridge.state == BridgeState::Suspended(false) {
	if bridge.state == BridgeState::HardSuspended {

		// We cannot accept new messages to the suspended bridge, hoping that it'll be
		// actually resumed soon

	// We cannot accept new messages to the suspended bridge, hoping that it'll be
	// actually resumed soon
	// We cannot accept new messages and start dropping messages, until the outbound lane goes below the drop limit.

		@@ -0,0 +1,45 @@
		title: Bridges - revert-back and improve congestion

	title: Bridges - revert-back and improve congestion
	title: Bridges - Add improved congestion control mechanism

Bridges - Add improved congestion control mechanism #6231

Are you sure you want to change the base?

Bridges - Add improved congestion control mechanism #6231

Conversation

bkontur commented Oct 25, 2024 • edited Loading

Context

Description

bkontur commented Oct 25, 2024

bkontur commented Oct 26, 2024

bkontur commented Oct 28, 2024

bkontur commented Nov 5, 2024

bkontur commented Nov 7, 2024

bkontur commented Nov 7, 2024

bkontur commented Nov 7, 2024

bkontur commented Nov 7, 2024

bkontur commented Nov 8, 2024

github-actions bot commented Nov 21, 2024

bkontur commented Nov 27, 2024

github-actions bot commented Nov 27, 2024

github-actions bot commented Nov 27, 2024

acatangiu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bkontur commented Dec 22, 2024

github-actions bot commented Dec 22, 2024

github-actions bot commented Dec 22, 2024

paritytech-workflow-stopper bot commented Dec 22, 2024

bkontur commented Oct 25, 2024 •

edited

Loading