State diffing at head can leave gaps

[dboreham]

If some fatal error occurs in state diffing inside geth (such as this), after re-start a gap in the set of state diff data in the database can result.

geth shouldn't leave gaps even when it exits due to a fatal error / crash and is re-started.

[i-norden]

Related:

• https://github.com/cerc-io/issue_tracking/issues/11
• https://github.com/vulcanize/ops/issues/201
• Journal blocks that are pending statediffing when the node is shutdown #131
• Small gap during statediffing node redeployments #292
• See https://github.com/cerc-io/go-ethereum/issues?q=is%3Aissue+known+is%3Aclosed
• Feature/known gaps #217
• Update known gaps condition to allow single block gaps. #223

[telackey]

Gap detection and filling has been added to https://git.vdb.to/cerc-io/chain-chunker .

It is useful when there are known to be gaps (eg, after an incident) and could be used to fill smaller gaps, such as those created by restarting the containers automatically (eg, by cron job).

In short, in makes it quite easy to fix gaps, however, it does nothing to prevent gaps, which is more of the the issue here.

Statediffing for head is triggered by the event feed for the blockchain. These events are triggered after the block is processed by geth. When geth is restarted, it will start syncing from the last block it successfully processed. However, statediffing (since it is event based) will start from the next block that geth processes. This allows for small gaps to open, since any statediffs which were in-flight at the time geth was restarted will not be triggered again.

The likely fix is to add a check (probably on restart, but perhaps periodic) which looks for any gap between the current block_number and max(block_number) from eth.header_cids and triggers statediffing accordingly.

[telackey]

I'm going to close this one as a duplicate of #292