Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support different memtable implementation #11

Closed
al8n opened this issue Sep 30, 2024 · 4 comments
Closed

Support different memtable implementation #11

al8n opened this issue Sep 30, 2024 · 4 comments
Assignees
Labels
enhancement New feature or request

Comments

@al8n
Copy link
Owner

al8n commented Sep 30, 2024

While crossbeam_skiplist::SkipMap and std::collections::BTreeMap are solid choices for a general-purpose ordered map, they may not be ideal for a database environment. This is primarily due to its reliance on dynamic memory allocations for each entry, leading to non-contiguous memory layouts. In a database scenario, where performance and efficient memory usage are critical, this can become a bottleneck.

The lack of contiguous memory means that pointer chasing occurs frequently during lookups, increasing cache misses and search times. This added overhead can significantly impact performance when performing large-scale or frequent queries.

In contrast, skl::SkipMap provides a more suitable approach for database environments because it stores entries in a continuous memory buffer, reducing the need for frequent allocations. This leads to better cache locality and faster search operations, making skl::SkipMap a more performant choice in cases where minimizing memory overhead and maximizing search efficiency are crucial, such as in database systems.

Hence, in 0.5, the sidecar of the write-ahead log will be changed to skl::SkipMap.

@al8n al8n added the enhancement New feature or request label Sep 30, 2024
@al8n al8n self-assigned this Sep 30, 2024
@al8n
Copy link
Owner Author

al8n commented Oct 1, 2024

The benchmark of skl shows that crossbeam_skiplist is faster than skl, need to figure out what happens in skl implementation

@al8n
Copy link
Owner Author

al8n commented Oct 1, 2024

Find the problem, see al8n/skl#41

@al8n
Copy link
Owner Author

al8n commented Oct 25, 2024

In #22, which supports using different memtables, either based on crossbeam-skiplist or skl. However, the performance regression problem on skl has not been solved.

@al8n al8n changed the title Use skl::SkipMap as the sidecar lookup map instead of crossbeam_skiplist::SkipMap and std::collections::BTreeMap Support different memtable implementation Nov 2, 2024
@al8n
Copy link
Owner Author

al8n commented Nov 2, 2024

Finished in f49130a.

@al8n al8n closed this as completed Nov 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant