state sync: reverse sync implementation

[cmwaters]

This PR implements ADR-068 allowing state synced nodes to fetch prior light blocks which can be needed in verifying any future evidence that may arise.

How it works

Backfill (the new method within the state sync reactor) runs directly after a node has verified and applied application state. I will provide a quick overview of how all the new pieces work so as to give you context before looking into the code. There are 2 new pieces:

• A dispatcher which is tied with the state sync reactor and allows for concurrent processes to request light blocks.
• A blockQueue which is initiated with a start height and an end height and allows for concurrent workers to use the dispatcher to get a light block and add it to the queue. The queue is then responsible for serializing the incoming blocks and giving them to another process to verify them.

The backfill method is then simply:

• Work out the start and end height based on the current height and evidence params
• Start n workers to request blocks and add them to the queue
• Have a main thread pick blocks up of the queue and verify them based on the check: header[height].Hash() == header[height + 1].LastBlockID.Hash
• Store the light blocks

FIXME's

There are various incongruities that have arisen from these changes. The main one to consider is that prior we considered the Base and Height of the blockchain based from the first and last BlockMeta. Adding headers and commits to the BlockStore means that the Base might not be the height of the earliest block rather the height of the earliest light block. A client may run /status to work out the earliest height and then call /block at that height only to find that it's not there. I have left this behavior as is however we might want to explore other solutions.

The other minor one is in the BlockStore. The BlockStore saves headers by saving BlockMeta which includes the header as well as other information. This other information is BlockSize and NumTxs which from the light block we don't know hence I've filled them with -1 this is hacky and it means that the BlockMeta isn't really complete.

[codecov]

Codecov Report

Merging #6463 (d2965b6) into master (a855f96) will increase coverage by 0.13%.
The diff coverage is 64.02%.

❗ Current head d2965b6 differs from pull request most recent head 9d804cb. Consider uploading reports for the commit 9d804cb to get more accurate results

@@            Coverage Diff             @@
##           master    #6463      +/-   ##
==========================================
+ Coverage   60.96%   61.10%   +0.13%     
==========================================
  Files         293      295       +2     
  Lines       27435    27870     +435     
==========================================
+ Hits        16727    17029     +302     
- Misses       9011     9124     +113     
- Partials     1697     1717      +20     

Impacted Files	Coverage Δ
internal/statesync/stateprovider.go	0.00% <0.00%> (ø)
internal/test/factory/block.go	83.72% <ø> (ø)
proto/tendermint/statesync/message.go	57.62% <0.00%> (-14.72%)	⬇️
rpc/core/blocks.go	23.36% <0.00%> (ø)
store/store.go	51.87% <0.00%> (-4.22%)	⬇️
state/store.go	56.72% <16.66%> (-0.79%)	⬇️
internal/consensus/metrics.go	14.59% <50.00%> (ø)
node/node.go	51.57% <50.00%> (+0.63%)	⬆️
internal/statesync/reactor.go	58.17% <54.94%> (+1.65%)	⬆️
internal/statesync/dispatcher.go	79.85% <79.85%> (ø)
... and 24 more