[Bug]: Fix RocksMQ Slow on consume

[xiaofan-luan]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:
- Deployment mode(standalone or cluster):
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Rocks MQ consume fairly slow on consume, see the log "rocksmq consume too slowly xxx"

Expected Behavior

rocksmq should consume fairly fast because most of the time it consume only from the end of the queue, it should be always on memory table or l0 files.

however, since the value of store is very large, if we hit on disk it might be fairly slow.

Before we use prefix_extractor, which rely on the prefix is fixed length, and brings much trouble.

Thus we changed to SetIterator Upper Bound, it works fine but sometimes we see loss of data or duplicated data.

Then we changed to Iter.ValidForPrefix(), the correctness works but we saw this performance issue.

Steps To Reproduce

keep writing data and consume on rocksmq, after few hours you will see this issue.

Anything else?

SetIterator Upper Bound issue is due to go gc the upper bound array.

Iter.ValidForPrefix is slow because it always fetch one more row from rocksdb, and that row is cold and occur some extra ios

[xiaofan-luan]

see also #14077