Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: insert api for HNSW #2395

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

feat: insert api for HNSW #2395

wants to merge 13 commits into from

Conversation

jacketsj
Copy link
Contributor

@jacketsj jacketsj commented May 25, 2024

We didn't previously have a way of inserting into HNSW post-build.
There's an important aspect to how this works without affecting performance:

  • Always keep the nodes stored in a flat structure
  • To do that, we need the number of nodes to be "fixed".
  • Add an RwLock to the nodes vec itself, and double its size whenever we overflow with a global lock.
    This will result in a brief slow-down whenever the size overflows, so latency of inserts (and also queries) will very occasionally spike briefly, but with smaller and smaller frequency as the graph size grows.

Copy link

ACTION NEEDED

Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

@jacketsj jacketsj changed the title [chore] Insert api for HNSW chore: Insert api for HNSW May 25, 2024
@github-actions github-actions bot added the chore label May 25, 2024
@jacketsj jacketsj changed the title chore: Insert api for HNSW feat: Insert api for HNSW May 25, 2024
@github-actions github-actions bot added the enhancement New feature or request label May 25, 2024
@jacketsj jacketsj changed the title feat: Insert api for HNSW feat: insert api for HNSW May 25, 2024
@jacketsj jacketsj marked this pull request as ready for review May 25, 2024 00:57
@codecov-commenter
Copy link

codecov-commenter commented May 25, 2024

Codecov Report

Attention: Patch coverage is 97.95918% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 79.73%. Comparing base (72b6715) to head (02cf8a4).

Files Patch % Lines
rust/lance-index/src/vector/hnsw/builder.rs 97.95% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2395      +/-   ##
==========================================
+ Coverage   79.70%   79.73%   +0.02%     
==========================================
  Files         200      200              
  Lines       54937    55015      +78     
  Branches    54937    55015      +78     
==========================================
+ Hits        43788    43864      +76     
- Misses       8609     8612       +3     
+ Partials     2540     2539       -1     
Flag Coverage Δ
unittests 79.73% <97.95%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jacketsj jacketsj requested a review from chebbyChefNEQ May 25, 2024 07:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants