ADR-040 Implementation

[clevinson]

Epic for tracking SMT Migration and other work related to ADR040.

Architecture

• support state-sync ADR-040 Implementation: Support state sync #10194

• rootstore / multistore design (wip)
  • docs: adr-40: reduce multistore and make it atomic #9355
  • test the design
• docs(adr): update ADR-40 by adding low level access to SC #9451
• migration design @robert-zaremba

store/v2 Implementation

For MVP:

• feat: ADR 040: New DB interface #9573
• feat: ADR-040: Implement RocksDB backend #9851
• feat: ADR-040: Implement BadgerDB backend #9848
• feat: ADR-040: Implement KV Store with decoupled storage and SMT #9892
• feat: ADR-040: Add RootStore implementation #10430
• Implement KVStore Cache / session
• feat: ADR-040: ICS-23 proofs for SMT store #10015
• ADR-040 Implementation: Incorporate v2.MultiStore into BaseApp #10804
  • for MVP, integrate store/v2 into baseapp. The support for optional store/v1 or store/v2 we will do later (see Next)
  • feat!: ADR-040: Refactor App to use v2.MultiStore #10174
• ADR-040 Implementation: Migrate storage command/script for store v2 #10803

Next:

• State sync: ADR-040 Implementation: Support state sync #10194
  • feat: ADR-040: add state sync for v2 store #10794
  • Optimization State sync: use native DB export - stream - import #10886
• Interblock multistore cache
• base app wiring: create two functions to wire app with store/v1 or store/v2
  • abstract app setup to be able to be able to choose
• low level interface implementation
• key prefix compression
• Overall performance Analyze and improve store/v2 performance #11328

Quality Assurance

Orijtech + SDK QA Team

• Orijtech audit of SMT library @cuonglm @odeke-em
• Storage benchmark (best to compare it with the current one)
• store: write extensive tests that exorcize store/* #10298
• store: write benchmarks /store and /store/v2 with different load scenarios #10297

Documentation and spec update

• Review and update documentation
• Review and update specs

Child Issues

Child issues:

• New DB interface: ADR-040 Implementation: New DB interface #10094
• Hook state store and SMT into the SDK: ADR-040 Implementation: Hook state store and SMT into the SDK #10117
• Review and refactor MultiStore: ADR-040 Implementation: Review and refactor MultiStore  #10192
• Refactor BaseApp to use new v2 MultiStore: ADR-040 Implementation: Incorporate v2.MultiStore into BaseApp #10804
• Support state sync: ADR-040 Implementation: Support state sync #10194
• Storage migration: ADR-040 Implementation: Migrate storage command/script for store v2 #10803
• Store performance: Analyze and improve store/v2 performance #11328

[i-norden]

Child issues:

• New DB interface: ADR-040 Implementation: New DB interface #10094
• Hook state store and SMT into the SDK: ADR-040 Implementation: Hook state store and SMT into the SDK #10117
• Review and refactor MultiStore: ADR-040 Implementation: Review and refactor MultiStore  #10192
• Support state sync: ADR-040 Implementation: Support state sync #10194

[i-norden]

Hi @clevinson, can you add a section to link to the child issues in the above comment from your top-level comment?

[clevinson]

@i-norden done!

[robert-zaremba]

I've updated the descriptions and added more subtasks. @clevinson , can we somehow enable @i-norden to edit this task?

[roysc]

The ICS-23 proof implementation should be added as a "store/v2" component: #10015

[roysc]

I created an issue for store migration implementation: #10517 - we can outline design there too.

[roysc]

"Base app wiring" should be covered by #10174

[roysc]

I've created an issue to track end-to-end testing of the store: #10518. This should happen after #10174

[roysc]

Rough estimates for remaining work:

• state sync - 20h
• base app wiring - 5h
• low level access - depends on design, maybe 10h
• migration command - 20h
• store/v1 backwards compatibility - 20h
• key prefix compression - 5h

[robert-zaremba]

Thanks @roysc . Could you translate it to the effective weeks of work?

[adu-web3]

Hi @roysc , is it possible or convenient for me to help with some of the above remaining works like testing?

[roysc]

I think we are at a point where benchmarks would be helpful (#10297). It would also be good to have integration tests covering the IBC UpgradeClient procedure with a change in the proof spec if you are familiar with that.

[adu-web3]

I think we are at a point where benchmarks would be helpful (#10297). It would also be good to have integration tests covering the IBC UpgradeClient procedure with a change in the proof spec if you are familiar with that.

I think I could try to start with benchmarks by doing some investigation.

[Terra854]

Just want to check, is ADR-040 still being worked on? It would be great to get it up asap cause it's getting harder and longer to sync a node with the ever increasing size of the databases

[adu-crypto]

I'd like to pick up the issue Interblock multistore cache and work on it.

[roysc]

The implementation is still in progress, but is waiting on integrated network tests being done internally. Once we have some results from that, we will start advancing the open PRs and pick up the remaining tasks.

@adu-crypto no problem, I think the v1 implementation can mostly be adapted.

[windycrypto]

hi @roysc thanks for the great job, do we have an ETA we can experience this new implementation? sorry not a push just looking forward to it :)

[tac0turtle]

adr40 is on pause till we better understand store/v1 and why it was designed this way then adr40 design will be evaluated