Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Advisory issue about potential Message Queue Bug #1375

Closed
ggwpez opened this issue Oct 25, 2024 · 4 comments
Closed

Advisory issue about potential Message Queue Bug #1375

ggwpez opened this issue Oct 25, 2024 · 4 comments

Comments

@ggwpez
Copy link

ggwpez commented Oct 25, 2024

Hey I saw that you have the Idle processing in the MQ pallet enabled, this was recently discovered to mark messages incorrectly as overweight.

(Feel free to close at any time, this issue is just meant as an advisory)

Issue

The bug was accidentially introduced by #3844 and fixed in #6205. It is awaiting backports for versions 1.10 up to stable2409.

Impact

The MQ pallet can - under specific circumstances - mark a message as permanently overweight and put it into an overweight queue instead of processing it. The message can still be executed manually, but it is not automatic (like it should). Someone wrote a bot for this: unofficial link.

I only saw this happen once recently, so impact should hopefully be limited:

Screenshot 2024-10-25 at 18 24 02

Work Around

A. Disable the Idle processing with type IdleMaxServiceWeight = (). This means that incoming messages will only be process in on_initialize - potentially increasing the latency of incoming messages by one block.

B. Another way is to manually call message_queue::execute_overweight on all messages that got stuck. Eg with the bot linked above.

@ermalkaleci
Copy link
Contributor

@ggwpez I tried message_queue::execute_overweight using Chopsticks and got error messageQueue.InsufficientWeight, so it seems like those 2 enqueued overweight are not related to IdleMaxServiceWeight

@ggwpez
Copy link
Author

ggwpez commented Oct 28, 2024

Okay, i also just tried it - indeed it seems to be properly overweight and not due to the bug.
I see that you have the normal and idle service weight configured to the same value; AFAIK this mitigates the issue.
Going to close now.

@ggwpez ggwpez closed this as completed Oct 28, 2024
@ermalkaleci
Copy link
Contributor

The only scenario when bug can happen is when on_idle remaining weight is less than xcm required weight, so basically a real full block

@ermalkaleci
Copy link
Contributor

Both solutions will require runtime upgrade so better to apply the patch, in the meantime if this happens we will trigger it manually

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants