evm tx skips a block before confirmation

[shunjizhan]

Problem

When sending tx at block X, it is supposed to be mined at block X + 1, assuming no congestion. However, there seems to be a shared issue among substrate evms: such tx is usuallly mined at block X + 2, which basically mysteriously skipped a block.

This causes a terrible UX. For 12s block time, expected confimation time should be 12/2 = 6s on average, which is great. However, with the current issue, expected confirmation time increases to 12/2 + 12 = 18s. This is super slow compared to these alternative evm networks, which usually confirms in less than 5s, sometimes even 2s.

18s confirmation time could be a huge blocker for mass adoption. Users will escape.

Reproduce Script

https://github.com/shunjizhan/tx-confirmation-test

[shunjizhan]

moonbeam-foundation/moonbeam#2286
AstarNetwork/Astar#930