Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

XCM delivery fee breaks XCM compatibility #3070

Closed
xlc opened this issue Jan 26, 2024 · 40 comments
Closed

XCM delivery fee breaks XCM compatibility #3070

xlc opened this issue Jan 26, 2024 · 40 comments

Comments

@xlc
Copy link
Contributor

xlc commented Jan 26, 2024

A failed transfer: https://karura.subscan.io/xcm_message/kusama-024c52268887c3fb756eeea47242a7d2d831b6ba

Since the new Kusama runtime upgrade, which includes #1234, KSM transfer between parachains stops working.

This is due to the XCM execution on Kusama unable to charge deliver fee from holding register and therefore refuse to send DMP to dest chain.

This also highlights a main issue with the current deliver fee mechanism. It is infeasible to query and calculate the delivery fee pure from onchain runtime. It is a tx version breaking change to ask user to specify an extra delivery fee parameter. It is not possible to enable jit_withdraw mode in this particular case. It just doesn't work.

@Polkadot-Forum
Copy link

This issue has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/polkadot-digest-26-jan-2024/5848/1

@albertov19
Copy link

Similar problem here -> https://kusama.subscan.io/extrinsic/21573927-7

@acatangiu
Copy link
Contributor

@xlc do you have a chopstix test or similar to use to test out ideas for quick fixes?

I believe something like this 371d5b0 could get us running again until we come up with a comprehensive solution for:

This also highlights a main issue with the current deliver fee mechanism. It is infeasible to query and calculate the delivery fee pure from onchain runtime.

@acatangiu
Copy link
Contributor

Also, does no one have these scenarios running on testnets?

#1234 is live on Rococo relay and System Chains since 3-4 months now - not cool getting burned directly on Kusama 😢

@xlc
Copy link
Contributor Author

xlc commented Jan 26, 2024

To reproduce this with Chopsticks

To run the networks
KARURA_BLOCK_NUMBER=6136909 KUSAMA_BLOCK_NUMBER=21594465 BIFROST_BLOCK_NUMBER=5895485 npx @acala-network/chopsticks@latest xcm --relaychain kusama --parachain karura --parachain bifrost

Open Karura and send the extrinsic:
https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A8000#/rpc

submit author.submitExtrinsic with data:
0x95028400842745b99b8042d28a7c677d9469332bfc24aa5266c7ec57c43c7af125a0c16c016ed23bfa0338e2c971e6e9c1e375b56c6791de705fee18e43d2f0d7c1e06f864546e9b62179b9691e364cb9aa39773f1b8b090980cbd66a2b4e551b70cb0c28da4008981003600008200743ba40b000000000000000000000003010200451f0100842745b99b8042d28a7c677d9469332bfc24aa5266c7ec57c43c7af125a0c16c00

Submit it and wait for few minutes to produce blocks. Check terminal logs for output.

Check the new block produced on Kusama: https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A8002#/explorer

If the issue is fixed, check the block on Bifrost: https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A8001#/explorer

To test the fix or print XCM logs, we need to do wasm override for Kusama:
copy the kusama config: https://github.com/AcalaNetwork/chopsticks/blob/master/configs/kusama.yml
specify wasm-override point to the new Kusama runtime
repeat from step 1 but use --relaychain path/to/kuama.yml

@xlc
Copy link
Contributor Author

xlc commented Jan 26, 2024

I can create something similar to https://github.com/xlc/chopsticks-ksm-xcm tomorrow. but it should be trivial enough to modify it to test for this bug

@itsbirdo
Copy link
Member

Linked issue: #3075

@adaOctopus
Copy link

Same issue for me. Bridging KSM to KAR seems fine in the explorer, but KSM funds never arrived to the KAR wallet.
https://kusama.subscan.io/extrinsic/21602766-2
Any ideas when this is fixed and if funds have been lost?

@albertov19
Copy link

FYI - I've created two scripts to calculate the worst-case scenario for Kusama -> Parachain KSM transfers and from AssetHub -> Parachain Asset Transfers:

Kusama -> https://github.com/albertov19/xcmTools/blob/main/calculateKusamaDeliveryFees.ts
AssetHub -> https://github.com/albertov19/xcmTools/blob/main/calculateAHXcmDeliveryFees.ts

These scripts assume the worst-case scenario by applying U128::MAX to the variable input of the elements that result in the "longest" XCM possible. This check should be done in the origin chain (either Kusama or AssetHub) by dApps, and add this value to the tx fee of paymentInfo endpoint.

@rageruun
Copy link

Tokens stuck in parachain bridge when transmitted from the Kusama network to the Moonriver (xcKSM).
image

@acatangiu
Copy link
Contributor

update:

fix is merged to patch-release branch: #3085 and released to crates.io: https://crates.io/crates/staging-xcm-executor/4.0.1

another fix is also being backported #3113 and 4.0.2 needs to be published to crates.io

next, we'll have to include it in Polkadot & Kusama runtimes and do a patch release there

@rageruun
Copy link

rageruun commented Jan 30, 2024 via email

@Hawkarhf92
Copy link

Please someone help, I sent some KSM 5 days ago but still not received, from kusama network to Sora mainnet, but xcm transfer not created, here is my extrinsic hash, (0x78540d5b0b3831468c7c4a70ef5a81bccb3c7cb96e44ad81e39c96e73a9ebcff ) please help ,

@albertov19
Copy link

FYI - For people that sent XCM transfers to Moonriver -> https://forum.moonbeam.network/t/kusama-assethub-rt-1001000-xcm-failed-transactions/1542

@acatangiu
Copy link
Contributor

should be fixed in runtimes patch release https://github.com/polkadot-fellows/runtimes/releases/tag/v1.1.2

@adaOctopus
Copy link

@acatangiu do you know funds bridged from Kusama to Karura will be unlocked and available again to us? It has been already a week. There are dozens of people by the way asking this same question in the discord server. Any inputs from your end in terms of ETAs will be much appreciated. Super thanks !!!

@acatangiu
Copy link
Contributor

@acatangiu do you know funds bridged from Kusama to Karura will be unlocked and available again to us? It has been already a week. There are dozens of people by the way asking this same question in the discord server. Any inputs from your end in terms of ETAs will be much appreciated. Super thanks !!!

Once Kusama upgrade is enacted, the root cause issue will be fixed - aka it should not happen again (further transfers should work well).
Regarding already locked funds of past transfers, these can be unlocked by the specific parachains involved at any time.

In your case, Karura needs to manually unlock the funds, but bear in mind that this is probably a somewhat manual process so it might take a bit of time on their side.

@adaOctopus
Copy link

@acatangiu thanks for the reply. Appreciate it. But how can we enact this manual process? is there someone from Karura we need to talk to? And what does "a bit of time" mean? I know Karura probably is not your team to know more stuff about it, but any guidance would help.

@acatangiu
Copy link
Contributor

Please move this discussion to a more relevant place.

I know Karura probably is not your team to know more stuff about it, but any guidance would help.

On their homepage I can see links at the bottom to their telegram, twitter, discord, etc, try one of those.

@jonathanpdunne
Copy link

I'm aware of at least one case where there are locked funds of past transfers stuck on Kusama, where the recovery process is likely a referenda via OpenGov.

Would it appropriate for someone familiar with Substrate to take a look at the history of the chain, and propose a referenda which retrieves all stuck KSM that can be attributed to this deliveryFee error?

@xlc
Copy link
Contributor Author

xlc commented Feb 8, 2024

There are two different cases. For KSM transfer between parachains, it is the dest chain need referenda to credit the missing fund. For Karura, we plan to make it happen after Kusama have fixed the issue and then try to identify all the failed transfer to Karura and then create a referendum to credit the fund.

For KSM from Kusama to parachain that goes missing, it can only be returned by Kusama referendum.

@albertov19
Copy link

albertov19 commented Feb 8, 2024

@xlc for KSM from Kusama to para chains, it can also be returned through a referendum in the parachain itself.

We indexed all transactions in Kusama that transferred KSM to Moonriver's Sovereign account but the XCM Attempted failed. Then we proposed a referendum that sends an XCM message to Kusama doing [WithdrawAsset, ClearOrigin, BuyExecution, DepossitAsset(1), DepossitAsset(2)....,DepositAsset(Wild(AllCounted)....] where 1,2,3.. are the addresses of the affected users that we got from the indexer.

https://forum.moonbeam.network/t/kusama-assethub-rt-1001000-xcm-failed-transactions/1542/2?u=albertov19

The only KSMs that need to be returned through a Kusama referenda are those that were teleported from Kusama to AssetHub, as they are locked in a system account in Kusama (some XCM related account)

@sebastianmontero
Copy link

Has this issue been fixed in rococo?

@acatangiu
Copy link
Contributor

Has this issue been fixed in rococo?

yes, this is fixed on Rococo, Westend, Kusama and Polkadot.

@Pescador1551
Copy link

@albertov19 @acatangiu : I lost 76 KSM funds in January during an XCM teleport from Kusama to Assethub due to the bug. Do I understand correctly that I can only recover this by initiating a referendum on Kusama? Or is there another way to regain control over my funds? Thanks in advance

@albertov19
Copy link

Hey @Pescador1551 to my understanding, yes.

Unsure if the fellowship is doing anything to help recover these funds. Or the wallet team or dApp that you used to transfer the funds

@acatangiu
Copy link
Contributor

@albertov19 @acatangiu : I lost 76 KSM funds in January during an XCM teleport from Kusama to Assethub due to the bug. Do I understand correctly that I can only recover this by initiating a referendum on Kusama? Or is there another way to regain control over my funds? Thanks in advance

Hi @Pescador1551, you should be able to recover them yourself, by using new claim_assets() extrinsic. This will be available in the upcoming runtimes v1.2.0 release happening next week (plus a couple of weeks for gov to upgrade).

Alternatively, you can go the referendum route to unlock the funds which may or may not be faster in the end.

@Pescador1551
Copy link

Hi @acatangiu, I would like to reclaim my assets by using the function you mentioned in your previous message ('new claim_asset() extrinsic'). However, I'm not an expert in this field and honestly don't know where to start. I kindly ask if you have any information on how I can execute this action. I highly appreciate it. Thank you in advance!

@franciscoaguirre
Copy link
Contributor

Hello @Pescador1551, I made a small guide you can follow here: https://hackmd.io/@franciscoaguirre/Byahn6wZR.
Here if you have any other questions 😄

@Pescador1551
Copy link

Hello @franciscoaguirre, thank you so much for this explanation. I've made progress, but before I proceed with the action, I'd like to verify with you if I've set up all the variables correctly so I don't end up losing my assets after all. It was actually about a transfer from the native KSM chain to the Statemine chain where I lost my assets. It was this transaction. I have added a screenshot of the action I want to perform. I have two questions, how much KSM should be available in my account to perform the action? As far as you can see, the variables are defined correctly (XCM version: V3, interior: Here (does that mean it will be settled on my own account?).

Screenshot_KSM

BTW: If this is not the right place to continue this conversation, we can certainly move to another platform. Do you have any suggestions where we could potentially continue?

@franciscoaguirre
Copy link
Contributor

Hi @Pescador1551, thank you for clarifying with me first.

Two things:

  • The assets should be exactly the assets that were trapped, so both the version and the id and amount must be correct. So you need to click on "add item" and put the assets that were trapped.
  • Don't use (parents: 0, interior: Here), that's the chain origin! You should use your own account, so (parents: 0, interior: X1(AccountId32 { })). You can just click on your account's avatar to copy it and then paste it there.

I'll update the guide with this. Thank you for letting me know!

I don't have any problem continuing the conversation here. It's useful for anyone who might have this issue.

@Pescador1551
Copy link

Thank you for your prompt response and your updated guide. I have set up the action again and would like to ask if you find what I entered now logical, given the extrinsic (which I already mentioned in my previous message)?
Screenshot 2024-04-26 205939
For the ID, I have now entered my Kusama address, is that what you meant? I see in the example that it should be a 0x address.
Screenshot 2024-04-26 205710
Thank you in advance!

@Pescador1551
Copy link

Hi @franciscoaguirre, I forgot to tag you in my previous message so you missed it probably! My bad, sorry for that!

@franciscoaguirre
Copy link
Contributor

@Pescador1551 that looks okay now, but taking a look at your transaction, I don't see the AssetsTrapped event.
It might be that we have to make a governance proposal to get them back.
We are investigating to see if there are more affected accounts, then we'll create proposals to get all the funds back.

@Pescador1551
Copy link

Hi @franciscoaguirre, should I attempt to carry out the action, or am I risking the assets disappearing from sight? Additionally, I lack experience in creating a governance proposal if that becomes necessary. I hope you can assist me.

@Pescador1551
Copy link

Hi @franciscoaguirre, I found a proposal on Kusama that addresses the same issue, but from an individual address. You can find it here: https://kusama.subsquare.io/referenda/356. Could this help with a new proposal for addresses that have encountered the same issue?

@franciscoaguirre
Copy link
Contributor

@Pescador1551 We will assist you and everyone affected by this bug. Thanks for sharing that proposal.

I'll let you know when the proposal is created

@Pescador1551
Copy link

Hi @franciscoaguirre, that would be very nice. Hopefully the proposal will be approved and I can support your work a little bit with a small portion of lost funds. Thank you in advance. I really appreciate your help.

@Pescador1551
Copy link

Hi @franciscoaguirre, I am curious if there have been any developments regarding the drafting of a proposal for the lost funds. Is there anything I can contribute to this?

@Pescador1551
Copy link

Hi @franciscoaguirre, I hope this message reaches you well. Could you provide an update on the proposal being drafted to recover the lost funds? Thank you in advance!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests