Use sqlite to store seeds / proofs / etc

[ethanfrey]

(Moved from tendermint/light-client#33)

Replace attempt at adding two indexes using the file system (seeds by height and hash), with a standard system. Seems nicer to use sqlite than leveldb:

for the interactive client and easy for inspection
for the tooling including migration frameworks
for powerful query possibilities
performance is not much of an issue as long as db is not too big (<<1 GB ???) maybe more if the binary data is stored separately, and that limit is more about meta-data. should not be major issue for clients.

Info on sqlite

Max db size seems to be several gb:
https://softwareengineering.stackexchange.com/questions/332069/what-is-a-realistic-real-world-maximum-size-for-a-sqlite-database

Seems they support decently powerful secondary indexes
https://sqlite.org/lang_createindex.html

Performance storing large blobs :
https://sqlite.org/intern-v-extern-blob.html

[zramsay]

still relevant?

[ebuchman]

Nice idea but more of a long term optimization than a priority.

[jackzampolin]

I think we are currently using sqlite for the client seeds. Going to close this issue.

[ethanfrey]

@jackzampolin Can you please back up "I think" with a link?

From what I understand, all the code that dealt with seed/hashes/etc is moved into tendermint/tendermint/lite. In particular, the DbProvider is still based upon tmlibs.dbm, which is basically leveldb.

If sqlite is now used under dbm interface, please provide a link, I'd love to see that to inspect the data.

Can you either re-open the issue or close it with a "won't fix" label? I see it as "post-post-launch", or maybe never, but I don't think it was done.

[jackzampolin]

@ethanfrey Looking through the code you are correct! It is a leveldb store.

I'll change the reason for closing to proposal not accepted. Thank you for correcting me here. If you would like to implement sqlite under the dbm interface please start another issue for that on the appropriate tmlibs repo.

[ethanfrey]

Actually the idea was not to stick with the dbm interface for the client-side storage. We use it for tendermint and cosmos-sdk uses it. But no need to force that on a client. We could consider more user-friendly options.

That being said, I no longer have much interest in a pure golang client. It is great for cli and development, but I realize many people build apps with javascript. So, let's target that... or wasm....

[jackzampolin]

Long term plan here is to publish spec for our wallet derivations and then allow users to do whatever they want here.