state_getStorage keeps failing with "No node available for storage query" in v2.0.26 (smoldot-light-js)

[jsamol]

Summary

I use Smoldot to interact with a parachain. After updating Smoldot (smoldot-light-js) from version 2.0.23 to 2.0.26, calls to state_getStorage initially succeed but eventually start failing with the error message "No node available for storage query" and never recover.

Environment

I initially observed the issue on Android, where I run my custom bindings based on wasm-node, but I was also able to reproduce it with Node.js and the JavaScript smoldot package:

smoldot: 2.0.26
node.js: 20.14.0
os: macOS 14.5

Note: I realized that while I was debugging and writing the issue for 2.0.26, a new version 2.0.27 was released. I briefly checked it, but the issue persists.

Steps to reproduce

1. Download the latest relay and parachain specs, for example Kusama and Moonriver:

Kusama

$ echo '{"id":53,"jsonrpc":"2.0","method":"sync_state_genSyncSpec","params":[true]}' \
   | websocat -n1 -B 99999999 wss://ksm-rpc.stakeworld.io \
   | jq .result > kusama.json

Moonriver

$ curl https://raw.githubusercontent.com/moonbeam-foundation/moonbeam/master/specs/moonriver/parachain-embedded-specs.json \
   --output moonriver.json

# may require adjusting the `relayChain` value

2. Periodically query the parachain storage:

const smoldot = require('smoldot')
const fs = require('fs')

async function main() {
    const client = smoldot.start()

    const relayChainSpec = fs.readFileSync('kusama.json', 'utf-8')
    const relayChain = await client.addChain({ chainSpec: relayChainSpec })

    const parachainSpec = fs.readFileSync('moonriver.json', 'utf-8')
    const parachain = await client.addChain({ chainSpec: parachainSpec, potentialRelayChains: [relayChain] })

    while (true) {
        const request = {
            jsonrpc: '2.0',
            id: Math.floor(Math.random() * 100000),
            method: 'state_getStorage',
            params: ['0xf0c365c3cf59d671eb72da0e7a4113c49f1f0515f462cdcf84e0f1d6045dfcbb']
        }
    
        console.log('request', JSON.stringify(request, null, 2))
        parachain.sendJsonRpc(JSON.stringify(request))
        
        const response = await parachain.nextJsonRpcResponse()
        console.log('response', JSON.stringify(JSON.parse(response), null, 2))

        await new Promise(resolve => setTimeout(resolve, 60 * 1000)) // delay 1 minute
    }
}

main()

The first attempt should complete successfully, but eventually, you will start observing errors in responses:

{
  "jsonrpc": "2.0",
  "id": <id>,
  "error": {
    "code": -32000,
    "message": "No node available for storage query"
  }
}

Please let me know if you need more details to debug this issue.

[tomaka]

If you could provide the logs, that would be very useful 🙏
Pass { maxLogLevel: 4 } to start() (example), and upload what's being printed.

[jsamol]

Sure! I let it run for a couple of minutes: 1716984807.log

[tomaka]

The network query to retrieve the state trie root seems to work, but not the storage query. Suspecting this.

[jsamol]

Also, I'm not sure how relevant it is and why it was prompted to my console rather than piped to the log file, but I'm also seeing a lot of these:

[smoldot] The task named `sync-service-kusama` has occupied the CPU for an unreasonable amount of time (190ms).

[tomaka]

So the problem indirectly comes from this filter. It filters out all peers.
If we look at the logs, all the block announces coming from the parachain peers have is_best=false. We track in smoldot which peer has which best block, but this value is never updated because of this.

I think I'm going to rework the way the parachain sync service tracks parachain peers. We can assume that all parachain peers know about the parachain block stored on the relay chain.
The objective of the current code (that tracks precisely which peer has which block) is to filter out malfunctioning parachain nodes, however there are many different ways a parachain node can malfunction. All the possible malfunctions are also covered by the fact that we ban peers if they fail to answer the requests that we think they should be able to answer. So let's get rid of this precise tracking.

[tomaka]

Something annoying is that the system_peers JSON-RPC function requires returning the best block of each peer. I don't really know how to deal with this. I guess I'll just return the block stored on the relay chain, as that should be the best block, and it will be more correct than right now anyway.

EDIT: we can't do that, as we don't know the height of the block.