Valnodes forceprune

[moniya12]

Closes: #XXX

Description

Add forceprune to CLI (Ability to compact/prune via CLI)

Please review cmd/osmosisd/cmd/forceprune.go

Forceprune provides options to prune blockstore.db and state.db. By default, it would prune blockstore.db and state.db up to 14 days (unbounding period). In state.db while it keeps validator and consensus information up to 14 days it prunes abci responses (keeping only last 1000 blocks abci response).

Forceprune does not touch application.db or indexer.

ABCI Responses are stored in index db and so redundant especially if one is running pruned nodes. As a result we are removing ABCI data from state.db aggressively by default. One can override height for blockstore.db and state.db by using -f option and for abci response by using -m option.

Example osmosisd forceprune -f 188000 -m 1000.

---

For contributor use:

• Targeted PR against correct branch (see CONTRIBUTING.md)
• Updated relevant documentation (docs/) or specification (x/<module>/spec/)
• Added a relevant changelog entry to the Unreleased section in CHANGELOG.md
• Re-reviewed Files changed in the Github PR explorer

[ValarDragon]

Thanks for PR'ing this! It will be really helpful for node operators!

@czarcas7ic tested it on nodes I believe. The PR lgtm once it passes linting. (Most of these should get fixed if you run make format. Any remaining ones will come up in make lint)

[codecov-commenter]

Codecov Report

Merging #1262 (8caa057) into main (68f2661) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1262   +/-   ##
=======================================
  Coverage   19.81%   19.81%           
=======================================
  Files         210      210           
  Lines       27884    27884           
=======================================
  Hits         5525     5525           
  Misses      21333    21333           
  Partials     1026     1026           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 68f2661...8caa057. Read the comment docs.

[p0mvn]

Thank you for this change. I left some comments, please take a look

[tac0turtle]

This is similar to how i setup cosmprund, while it works perfectly fine its super slow, interested to hear your speeds on 100GB+.

My next design was going to create a new db and migrate the blocks over, while more complex you can do it in a fraction of the time.

[faddat]

moniya12#4

That'll fix ci for this.

I am actually going to test this code right now, here's the scenario:

I have a machine at hetzner where I have been struggling to make a rocksdb image for distribution.

When the state gets too big it slows to the pace of a camel.

I am going to combine this with some of my other recent work and see where it gets us.

[faddat]

@marbar3778 and @moniya12

I'm trying cosmosprund on rocksdb v7 now, I'll make a draft PR into there. Right now, it does not seem that this supports rocks.

[p0mvn]

@moniya12 Thank you for addressing my comments and questions. I took another look and, mostly, had minor style suggestions.

Currently, my only concern is not having defer db_bs.Close(), please let me know what you think

[czarcas7ic]

This is similar to how i setup cosmprund, while it works perfectly fine its super slow, interested to hear your speeds on 100GB+.

My next design was going to create a new db and migrate the blocks over, while more complex you can do it in a fraction of the time.

I did this on a 106GB image and it completed in ~15 minutes

[ValarDragon]

defer db_bs.Close() is added in the logic! I suggest we merge this in, and then do a second follow-up PR, to make the db openings / closings match the desired idioms a bit more

Thanks for adding this @moniya12 / Valnodes team!