Problem: restore-app-db don't support goleveldb backend

[GAtom22]

Describe the bug
Getting error when trying to migrate from goleveldb to versionDB.

Error message:

Error: failed to initialize database: Corruption: VersionBuilder: Cannot add table file #2 to level 0 since it is already in the LSM tree on level 0  The file /home/tom/.tmp-evmosd/data/blockstore.db/MANIFEST-000004 may be corrupted.

To Reproduce
Steps to reproduce the behavior:

1. Run a local node with goleveldb for a while
2. Stop the node
3. Perform the migration procedure mentioned in the migration docs:

evmosd changeset dump data --home ~/.tmp-evmosd 

evmosd changeset verify data                         
37e3a66ee7facb4137e72f27027e409730b282388cb6d8053c8c0162320bc84e
{"version":"23","storeInfos":[{"name":"acc","commitId": ...

evmosd changeset build-versiondb-sst ./data ./sst

evmosd changeset ingest-versiondb-sst ~/.tmp-evmosd/data/versiondb sst/*.sst --move-files --maximum-version 23

evmosd changeset verify data --save-snapshot snapshot                                                         
37e3a66ee7facb4137e72f27027e409730b282388cb6d8053c8c0162320bc84e
{"version":"23","storeInfos":[{"name":"acc","commitId":{"version":"23","hash": ...

evmosd changeset restore-app-db snapshot application.db
version: 23, app hash: E1F14F2F27588F9B66701CE2D55B300EDA94E7440EED5D84ED2674299F232183

mv application.db ~/.tmp-evmosd/data/

# updated the db_backend to 'rocksdb'
evmosd start --home ~/.tmp-evmosd --chain-id evmos_9000-1
11:49AM INF Unlocking keyring module=server
11:49AM INF starting ABCI with Tendermint module=server
11:49AM INF starting node with ABCI Tendermint in-process module=server
11:49AM ERR failed init node error="failed to initialize database: Corruption: VersionBuilder: Cannot add table file #2 to level 0 since it is already in the LSM tree on level 0  The file /home/tom/.tmp-evmosd/data/blockstore.db/MANIFEST-000004 may be corrupted." module=server

Question: Am I missing a step before starting the node again?

Expected behavior
Migration should be successful and the node should start from the corresponding height

Additional context

VersionDB commit used:

github.com/crypto-org-chain/cronos/versiondb v0.0.0-20231027074119-c05c9c61c90e

Noted also that the app hash is different when restoring the app db.

Here's the block info (app hash: 8D7912F77B21A378F3FE3C8CF8728E3F1F2C43CA0256343B2B0E68897F9F85B4):

{"header":{"version":{"block":11},"chain_id":"evmos_9000-1","height":23,"time":"2024-01-24T14:43:46.61733572Z","last_block_id":{"hash":"FCD1541A9B46FBF653012CD9EFBD43B30C5F0996679FB98843B93A690A171D20","parts":{"total":1,"hash":"CA3F64938BE4A9BDCB0F713F4E58B2129CFB5635E84AA79F2B9B6F5C47060DD4"}},"last_commit_hash":"110AD94CFBF7000E10D1A80222B2A66B8D3ABA9CA99915F1754837945EE34B7F","data_hash":"E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855","validators_hash":"6D12A2512820728F1DCD760867D0367A2BAD473C8E18E0582343F115C27E8530","next_validators_hash":"6D12A2512820728F1DCD760867D0367A2BAD473C8E18E0582343F115C27E8530","consensus_hash":"7D0B88F8835DB6E92EC4E959CCD9324052C68794BC42567CFA45FB176ED6679A","app_hash":"8D7912F77B21A378F3FE3C8CF8728E3F1F2C43CA0256343B2B0E68897F9F85B4","last_results_hash":"E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855","evidence_hash":"E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855","proposer_address":"0494AD1D12D5BA6328ABB30A2D0AFDAB316364BD"},"data":{"txs":[]},"evidence":{"evidence":[]},"last_commit":{"height":22,"round":0,"block_id":{"hash":"FCD1541A9B46FBF653012CD9EFBD43B30C5F0996679FB98843B93A690A171D20","parts":{"total":1,"hash":"CA3F64938BE4A9BDCB0F713F4E58B2129CFB5635E84AA79F2B9B6F5C47060DD4"}},"signatures":[{"block_id_flag":2,"validator_address":"0494AD1D12D5BA6328ABB30A2D0AFDAB316364BD","timestamp":"2024-01-24T14:43:46.61733572Z","signature":"ycaR6+VvoCph+pxYgiSqZAUvMdymu8HDBtS4I9CYD7rSUk0G5nS/2Q59wKuT1xOSYaHMWKIMgSzxkV8qYLlICA=="}]}}

And this is the output from the restore-app-db command:

evmosd changeset restore-app-db snapshot application.db
version: 23, app hash: E1F14F2F27588F9B66701CE2D55B300EDA94E7440EED5D84ED2674299F232183

[yihuang]

sorry for the delay, but the restore-app-db command only supports rocksdb backend ;D
it's briefly mentioned here: https://github.com/crypto-org-chain/cronos/tree/main/versiondb#restore-iavl-tree

Because it uses some rocksdb specific feature to be faster, but it's trivial to support goleveldb, just insert the key-values in the normal way.

[yihuang]

about the app hash mismatch, I guess it's because block 23's header includes app hash of block 22?