[RPC] mismatch block hash on the private network RPC node

[zjg555543]

Versions：

• Private network
• Erigon: xlayer dev which base on v2.60.0
• Contract: ForkID.13

Reproduction

• Stress txs for a period of time, RPC node can't sync with Seq node.

[INFO] [11-28|15:24:41.816] [8/15 IntermediateHashes] Trie root      hash=0x09631bb355b41de6b74463f5f5e5d46287b57d30811a7708d9eddea04515e594
[EROR] [11-28|15:24:41.822] Staged Sync                              err="[8/15 IntermediateHashes] Wrong trie root of block 70969: 09631bb355b41de6b74463f5f5e5d46287b57d30811a7708d9eddea04515e594, expected (from header): 7ddb51014445bb4bd0ee7137bf5193734684ecd6e92d6d0f6299cc6d458c489f. Block hash: b8c4b7afdc5a627eeee12ff76c851dcf9a88662af53ed0ccca7972b3e469f408, trace: [stageloop.go:105 panic.go:920 stage_interhashes.go:174 stages.go:347 sync.go:513 sync.go:383 stageloop.go:154 stageloop.go:73 asm_amd64.s:1650]"

Deployment

[mandrigin]

sequencer logs needed too

[mandrigin]

also, did you use forkid.13's features there? what kind of load there was? would be very nice to have some script to reproduce

[zjg555543]

We did not use the features of fork13.
They are common native token transfer txs.
The miss match issue only appear once today, I deleted the rpc's data, and resync again, and it works ok and sync the latest block.

I hava replaced with this original tag v2.60.0-RC4 and test it anagin.

[mandrigin]

also we need to make sure that both seq and rpc are on the same version, especially around non-stable (RC) versions

[zjg555543]

Yes, they use the same verson v2.60.0-RC4

[zjg555543]

The mismatched issue appeared again after stressing test about 3hours.
Sequencer and RPC use this tag.v2.61.0-RC1

Stress CMD

polycli loadtest --rpc-url https://testrpc.xlayer.tech/terigon --private-key xxxx --verbosity 600 --requests 20000 -c 1000 --rate-limit 1000 --mode erc20 --legacy

Error logs

2024-12-10T10:00:38.403846186+08:00 [EROR] [12-10|10:00:38.403] Staged Sync                              err="[8/16 IntermediateHashes] Wrong trie root of block 272886: 92dec2d9a1ced0d75ad28bde610322031ee114c4a265c8b1783312008c240a2d, expected (from header): 4a9ef0cb376939008c725e9d55d3d967f88d2ec3a04d2034e3cd5dcfa7cefbb6. Block hash: fa11a4643e740129c1b7cc8d57a4c2b02c8def52efb0d56797cac46da4702c94, trace: [stageloop.go:105 panic.go:920 stage_interhashes.go:168 stages.go:348 sync.go:513 sync.go:383 stageloop.go:154 stageloop.go:73 asm_amd64.s:1650]"
Pod: