LevelDB compation leads to unresponsive ABCI queries

[Zygimantass]

Summary of Bug

LevelDB compactions lead to unresponsive ABCI queries (staking, balances, etc.). When profiling, looks like a encoding/decoding and write/read syscalls take up most of the compaction time, and the compaction might be a blocking process.

Here you'll find the call graph with times attached (if a raw profile is needed, we can provide it.)

pprof001.pdf

Version

0.42.10

Steps to Reproduce

We used the following script to see when the requests fail. These timeouts coincide with LevelDB compaction and even when it finishes, the request times out. If you run a concurrent request after the compaction finishes (even with the timed out still hanging), it works. On timeout, the script fails, so it's easy to notice when the request is hanging.

#!/bin/bash

set -euo pipefail

for i in $(seq 1 500); do
  time curl -X POST -d '{"jsonrpc":"2.0","id":1435136462736,"method":"abci_query","params":{"path":"/cosmos.staking.v1beta1.Query/Delegation","data":"0a2d636f736d6f73313437353435716b686a686a72773930616a68677a756b66786761633773797468656e7a3873671234636f736d6f7376616c6f70657231686a63743671376e707373707367336467767a6b33736466383973706d6c7066646e366d3964","prove":false}}' http://localhost:26657/ --silent --fail
done

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[tac0turtle]

this is being worked on from multiple fronts, there is currently a refactor going on in iavl to reduce compaction and with store v2 it would be reduced further. Closing this issue in favour of #10987