[BanyanDB-Server] Add Memory Usage Control for BanyanDB Queries

[hanahmily]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

BanyanDB queries can use significant memory, particularly when processing large datasets. This proposal introduces a mechanism to control the maximum memory consumption during query execution. By utilizing the statistical information available for each block, the system can summarize the total memory usage of ongoing queries and enforce limits to prevent out-of-memory (OOM) errors.

A configurable parameter, such as max_query_memory, should be introduced to set the maximum allowable memory usage. If the estimated memory consumption exceeds this limit, the query execution should be gracefully terminated or paused.

It is important to provide detailed error messages when a query is terminated due to memory constraints. Additionally, memory usage metrics should be made available, and memory usage details should be logged for debugging and optimization purposes.

Optionally, a backpressure mechanism could be implemented to slow down or queue queries when memory usage approaches the set limit.

Please note that series index queries should not be affected by this limitation.

Use case

No response

Related issues

No response

Are you willing to submit a pull request to implement this on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct