State sync is broken in v12.2.1+

[faddat]

Context

I've never been able to state sync osmosis. I've contributed state sync scripts, some with pebbledb, some without, to > a dozen other cosmos chains.

There's been progress lately, namely that errors are reduced to those described here:

• chore: update state sync scripts #3507
• tendermint wrong Block.Header.AppHash bug #1090

My initial guess was that it was related to the use of upstream iavl, but that's proven incorrect. I've purchased a server, that will allow me to investigate this more thoroughly.

[faddat]

So I just spent ... pretty much way too long making state sync work.

this was going to be a writeup of problems, but I just got it working.

I don't know why it frequently doesn't work, but my node just reached the tip.

In order to make it work, I set a server to make a snapshot every 100 blocks. This actually takes longer than every 100 blocks takes to pass by, but basically I:

used the Notional snapshot, which is pruned, but not heavily, and is pebble flavored:

https://snapshot.notional.ventures/

I applied #3522 to Osmosis v12.3.0, and compiled on that server with the pebble code (mainly for speed reasons -- this took over 8 hours to get rolling) and then from my desktop, I began a state sync where that server was my only peer.

Previously, when I tried to state sync, I got block.Header.AppHash errors.

Now, it works. I'm pushing my working state sync branch here, and it is possible that we can finally make it work in CI.

If code changes made state sync work, these are the code changes that worked:

#3522