kvserver: investigate raft entry cache sizing

[tbg]

The raft entry cache is sized to 16mb per store. This seems very small:

cockroach/pkg/kv/kvserver/store.go

Lines 103 to 105 in e005c0f

	// defaultRaftEntryCacheSize is the default size in bytes for a
	// store's Raft log entry cache.
	defaultRaftEntryCacheSize = 1 << 24 // 16M

The raft.entrycache.{accesses,hits} metrics can be used to arrive at a cache hit rate. This is something we should be paying attention to when optimizing write performance.

x-ref #98576 where this was noticed.

Jira issue: CRDB-25431

[blathers-crl]

cc @cockroachdb/replication

[tbg]

Supplementary screenshot from an 8tb restore

We're reading from the raft storage quite a bit, probably we're usually lucky and pebble serves most of it from the block cache, as I don't see anywhere near that read bandwidth used by the disks.

[erikgrinaker]

I'd also expect this to still be in OS disk caches.

[nvanbenschoten]

In a write-heavy workload running kv0 with 2kb writes (repro steps), we see a gradual drop off in throughput. This is due in part to an increase in raft scheduler latency, which is a consequence of a reduction in the raft entry cache hit rate as the workload runs. Interestingly, we are able to see this clearly because some of the nodes have higher cache entry hit rates than others, and these perform much better.

This demonstrates that for write-heavy workloads, a 16mb raft cache entry cache is insufficient. We should explore dynamically sizing the cache based on available memory capacity in the system.

In the meantime, #105799 makes it configurable through an environment variable.

---

qps and p99 workload latency

p99 raft scheduler latency: notice node 2 and 3, compared to the other nodes

raft entry cache accesses and hits

same data, tabular form

	Accesses	Hits	Hit Rate
n1	314468	308334	98.1%
n2	276748	260645	94.2%
n3	271915	255306	93.9%
n4	325052	320766	98.7%
n5	326403	321934	98.6%

cpu profile from n3 and n4: notice n3's time reading the raft log from pebble

[erikgrinaker]

Interesting, thanks for these results.

There are other actors here too -- we've seen previously (#78412) that performance tanks when recent Raft log entries are flushed from the memtable into L0 before application and truncation, and the Pebble block cache and OS caches also come into play.

I suspect the memtable flushing primarily affects performance by increasing compaction disk IO, competing with foreground writes. However, it'd be interesting to see if it would be sufficient/better to try to have the log entries stick around in the memtable for as long as possible, instead of relying on a separate entry cache, since that would solve two problems at once.

[erikgrinaker]

That said, let's see if we can do something tactical for 23.2 -- 16 MB is miniscule.

[jbowens]

the log entries stick around in the memtable for as long as possible, instead of relying on a separate entry cache, since that would solve two problems at once.

I'm unsure how much keeping the entries in the memtable will help with these reads, because any read other than a point DB.Get lookup still needs to read and merge across all levels of the LSM. Even if the entries themselves are dropped in flushes, the tombstones dilute the keyspace making it less likely for reads of adjacent raft logs to hit the same sstable blocks.

[erikgrinaker]

Good point, I guess that answers that.

[erikgrinaker]

I sampled CentMon hit rates:

quantile(x, raft_entrycache_hits / raft_entrycache_accesses)

• 10th percentile: 100%
• 5th percentile: 99.8%
• 1st percentile: 99.4%
• Minimum: 90.3%

[erikgrinaker]

Some quick benchmarks, recording hit rates for all 3 nodes:

• kv0/enc=false/nodes=3/size=4kb: 99.3% 99.4% 99.4%
• kv0/enc=false/nodes=3/cpu=32/size=4kb: 99.4% 99.4% 99.4%
• kv0/enc=false/nodes=3/size=64kb: 86.1% 87.3% 87.3%
• kv0/enc=false/nodes=3/cpu=32/size=64kb: 87.1% 87.7% 87.4%

Will run some experiments with the small-node variants.

[erikgrinaker]

Average hit rate on single 10-minute runs of kv0/enc=false/nodes=3/size=64kb with varying COCKROACH_RAFT_ENTRY_CACHE_SIZE:

• 16 MB: 86.9%
• 64 MB: 96.3%
• 128 MB: 98.0%
• 256 MB: 98.5%
• 512 MB: 98.5%
• 1 GB: 98.5%

In the >=256 MB cases, the steady-state hit rate was 100% -- the misses all happened at the start of the workload. The 128 MB case was almost able to keep up, with a 99.6% hit rate if we ignore the first minute.

I don't think we should optimize for the 64 KB case, since it seems like a bit of an outlier. I'll do some 30-minute runs at 128 MB cache size with 1 KB, 4 KB, 16 KB, and 64 KB writes. If 128 MB seems reasonable then I think we can simply change the default without any scaling heuristics, at least as a trivial first step.

Here's the 256 MB chart:

[erikgrinaker]

3 runs of kv0/enc=false/nodes=3/size= on GCE for 30 minutes with COCKROACH_RAFT_ENTRY_CACHE_SIZE=128MB. Cache metrics discarded for first 2 minutes. Median of 3 runs.

Size	Hit Rate	Throughput	Nightly
1 kb	100%	12442 w/s
4 kb	100%	6522 w/s	6498 w/s
16 kb	100%	1692 w/s
64 kb	99.88%	417 w/s	418 w/s

While the cache hit rate is significantly improved compared to #98666 (comment), it unfortunately doesn't seem to have a significant impact on throughput for this specific workload.

The results in #98666 (comment) were on AWS though, which I believe tends to have lower IOPS capacity. I'll rerun that experiment with a 128 MB entry cache.

[erikgrinaker]

I re-ran the experiments from #98666 (comment), using 16 MB and 128 MB entry cache sizes respectively. I saw similar cluster behavior as the original results, including the throughput degradation and node bimodality.

The 128 MB entry cache basically eliminated cache misses. Furthermore, these are 256 GB RAM nodes, so if we had used the linear 1/256 memory scaling from #107424 the entry cache would have been 1 GB, which almost certainly would have ensured a 100% hit rate.

Per-node hit rates:

• 16 MB: 98.84% 98.84% 98.42% 95.53% 94.92%
• 128 MB: 99.98% 99.98% 99.97% 99.98% 98.85%

Unfortunately, as with other results, this did not translate to a significant throughput improvement. In fact, the 16 MB cluster had higher throughput (40127 w/s vs. 38865 w/s) which is most likely down to VM variation.

Furthermore, the 128 MB cluster also showed Raft scheduler bimodality that did not correlate strongly with entry cache misses:

The CPU profile from n3 (one with high scheduler latencies) confirms that we're not reading entries from disk:

n1 with low Raft scheduler latency appears very similar:

For the purposes of the entry cache, I think we can say that the linear 1/256 scaling proposed in #107424 appears sufficient to avoid cache misses, but the impact on throughput is unclear.