Support sparsevec in weighted vector search function #328

dqii · 2024-08-16T06:55:49Z

No description provided.

github-actions · 2024-08-16T07:04:25Z

Benchmarks

metric	old	new	pct change
recall (after create)	0.952	0.952	-
recall (after insert)	0.000	0.000	-
select tps	26694.845	26981.450	+1.07%
select bulk(100) tps	36.678	35.175	-4.10%
select latency (ms)	0.705 ± 1.491𝜎	0.684 ± 1.477𝜎	-2.98%
select bulk(100) latency (ms)	855.498 ± 112.767𝜎	878.168 ± 117.118𝜎	+2.65%
create latency (ms)	391018.657	407151.075	+4.13%
insert tps	441.719	440.408	-0.30%
insert bulk(100) tps	4.753	4.560	-4.05%
insert latency (ms)	71.822 ± 16.181𝜎	72.047 ± 14.955𝜎	+0.31%
insert bulk(100) latency (ms)	6629.397 ± 96.700𝜎	6844.803 ± 257.844𝜎	+3.25%
disk usage (bytes)	8192008192.000	8192008192.000	-

codecov · 2024-08-18T09:14:16Z

Codecov Report

All modified and coverable lines are covered by tests ✅

📢 Thoughts on this report? Let us know!

greptile-apps

PR Summary

This pull request adds support for sparsevec in weighted vector search functions, focusing on enhancing the functionality and flexibility of vector operations in the Lantern project. Here are the key points:

Introduced new overloads for weighted_vector_search functions to handle combinations of vector and sparsevec types in sql/lantern.sql and sql/updates/0.3.1--0.3.2.sql.
Added a new small_world_sparsevec.sql file with sample data for testing sparse vector functionality alongside dense vectors.
Updated test/sql/weighted_search.sql to include conditional tests for sparsevec, ensuring compatibility with systems that may not have the type available.
Modified docker/Dockerfile.dev to upgrade pgvector to version 0.6.1 and simplify the development environment setup process.
Enhanced input validation, error handling, and vector type casting in the weighted vector search functions to improve robustness and flexibility.

_{6 file(s) reviewed, 24 comment(s)}
_{Edit PR Review Bot Settings}

greptile-apps · 2024-08-18T09:35:45Z

docker/Dockerfile.dev

@@ -31,7 +31,7 @@ RUN gem install pg -- --with-pg-include=/usr/local/pgsql/include/ --with-pg-lib=
 # hack to make sure postgres user has write access to externally mounted volumes
 RUN mkdir /lantern_shared && chown postgres:postgres /lantern_shared

-RUN cd /root/postgresql-15.5/contrib && make install -j
+RUN cd /root/postgresql-15.5/contrib && make install


info: Removed '-j' flag from make install. This might slow down the build process.

greptile-apps · 2024-08-18T09:35:46Z

docker/Dockerfile.dev

@@ -55,7 +55,7 @@ COPY . .
 RUN sudo rm -rf build \
    && mkdir build \
    && cd build \
-    && cmake -DCMAKE_BUILD_TYPE=Debug .. \
+    && cmake .. \


info: Removed '-DCMAKE_BUILD_TYPE=Debug'. This changes the build type from Debug to default (usually Release). Ensure this is intentional and won't affect development workflows.

greptile-apps · 2024-08-18T09:35:46Z

docker/Dockerfile.dev

@@ -65,7 +65,7 @@ RUN git clone https://github.com/lanterndata/benchmark build/benchmark \
    && pip install -r external/requirements.txt

 # Install perf
-RUN sudo apt update && sudo apt install -y linux-tools-common linux-tools-generic linux-tools-`uname -r`
+RUN sudo apt update && sudo apt install -y linux-tools-common linux-tools-generic


info: Removed 'linux-tools-uname -r'. This might cause issues if specific kernel version tools are needed.

greptile-apps · 2024-08-18T09:35:47Z

docker/Dockerfile.dev

@@ -1,5 +1,5 @@
 ARG VERSION=15
-ARG PGVECTOR_VERSION=0.5.1
+ARG PGVECTOR_VERSION=0.6.1


info: Upgraded pgvector from 0.5.1 to 0.6.1. Verify compatibility with the rest of the system.

greptile-apps · 2024-08-18T09:36:31Z

sql/lantern.sql

@@ -667,6 +667,7 @@ CREATE OR REPLACE FUNCTION _lantern_internal.maybe_setup_weighted_vector_search(
 $weighted_vector_search$
 DECLARE
  pgvector_exists boolean;


info: Added check for pgvector sparsevec type existence

greptile-apps · 2024-08-18T09:39:00Z

test/sql/utils/small_world_sparsevec.sql

+    ('000', TRUE,  '[0,0,0]', '{}/3'),
+    ('001', TRUE,  '[0,0,1]', '{3:1}/3'),
+    ('010', FALSE, '[0,1,0]' , '{2:1}/3'),
+    ('011', TRUE,  '[0,1,1]', '{2:1,3:1}/3'),
+    ('100', FALSE, '[1,0,0]', '{1:1}/3'),
+    ('101', FALSE, '[1,0,1]', '{1:1,3:1}/3'),
+    ('110', FALSE, '[1,1,0]', '{1:1,2:1}/3'),
+    ('111', TRUE,  '[1,1,1]', '{1:1,2:1,3:1}/3');


info: Inserted test data covering all 3-bit combinations, with corresponding sparse vectors

greptile-apps · 2024-08-18T09:39:48Z

test/sql/weighted_search.sql

+-- Check if the type 'sparsevec' exists and store the result in a variable
+SELECT EXISTS (
+    SELECT 1
+    FROM pg_type
+    WHERE typname = 'sparsevec'
+) AS exists_sparsevec \gset


info: Added check for 'sparsevec' type existence

greptile-apps · 2024-08-18T09:39:49Z

test/sql/weighted_search.sql

+\if :exists_sparsevec
+    \echo 'The sparsevec type exists. Running commands...'
+    \ir utils/small_world_sparsevec.sql
+    SELECT '{1:0.4,2:0.3,3:0.2}/3' AS s3 \gset
+    SELECT '[-0.5,-0.1,-0.3]' AS v3 \gset
+    SELECT
+      id,
+      0.9 * (s <-> :'s3'::sparsevec) + 0.1 * (v <-> :'v3'::vector) as dist
+    FROM lantern.weighted_vector_search(CAST(NULL as "small_world"), exact => false, ef => 5,
+      w1=> 0.9, col1=>'s'::text, vec1=>:'s3'::sparsevec,
+      w2=> 0.1, col2=>'v'::text, vec2=>:'v3'::vector
+    );
+\else
+    \echo 'The sparsevec type does not exist. Skipping commands...'
+\endif


info: Introduced conditional execution for sparsevec tests

greptile-apps · 2024-08-18T09:39:50Z

test/sql/weighted_search.sql

+    SELECT
+      id,
+      0.9 * (s <-> :'s3'::sparsevec) + 0.1 * (v <-> :'v3'::vector) as dist
+    FROM lantern.weighted_vector_search(CAST(NULL as "small_world"), exact => false, ef => 5,
+      w1=> 0.9, col1=>'s'::text, vec1=>:'s3'::sparsevec,
+      w2=> 0.1, col2=>'v'::text, vec2=>:'v3'::vector
+    );


info: New test case for weighted vector search with sparsevec and vector types

Should we do something like this?

round(cast(0.9 * (s <-> :'s3'::sparsevec) + 0.1 * (v <-> :'v3'::vector) as numeric), 2) as dist

On arm processors the high precision floats seems to slightly differ from x86 ones which makes tests to fail

greptile-apps · 2024-08-18T09:39:51Z

test/sql/weighted_search.sql

+    SELECT '{1:0.4,2:0.3,3:0.2}/3' AS s3 \gset
+    SELECT '[-0.5,-0.1,-0.3]' AS v3 \gset


style: Consider using more descriptive variable names than 's3' and 'v3'

Ngalstyan4 · 2024-08-18T21:10:31Z

test/sql/weighted_search.sql

@@ -121,6 +120,30 @@ SELECT count(*)
  w3=> 0.52, col3=>'v_real'::text, vec3=>:'v444'::vector
 );

+-- Check if the type 'sparsevec' exists and store the result in a variable
+SELECT EXISTS (


could we follow the current pattern and not do this in the test itself?

Maybe, see how we filter out pgvector or lantern extras tests when relevant extensions are not installed and do something similar here

Ngalstyan4 · 2024-08-28T02:45:17Z

docker/Dockerfile.dev

@@ -31,7 +31,7 @@ RUN gem install pg -- --with-pg-include=/usr/local/pgsql/include/ --with-pg-lib=
 # hack to make sure postgres user has write access to externally mounted volumes
 RUN mkdir /lantern_shared && chown postgres:postgres /lantern_shared

-RUN cd /root/postgresql-15.5/contrib && make install -j
+RUN cd /root/postgresql-15.5/contrib && make install


Why change this?

440.5 /usr/bin/install -c -m 755 postgres_fdw.so '/usr/local/pgsql/lib/postgres_fdw.so' 440.5 /usr/bin/install -c -m 644 ./postgres_fdw.control '/usr/local/pgsql/share/extension/' 440.6 /usr/bin/install -c -m 644 ./postgres_fdw--1.0.sql ./postgres_fdw--1.0--1.1.sql '/usr/local/pgsql/share/extension/' 440.6 make[1]: Leaving directory '/root/postgresql-15.5/contrib/postgres_fdw' ------ Dockerfile.dev:34 -------------------- 32 | RUN mkdir /lantern_shared && chown postgres:postgres /lantern_shared 33 | 34 | >>> RUN cd /root/postgresql-15.5/contrib && make install -j 35 | 36 | # allow non-root users to install in the container to make it easier to run update-tests -------------------- ERROR: failed to solve: process "/bin/sh -c cd /root/postgresql-15.5/contrib && make install -j" did not complete successfully: exit code: 2 View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/ojyc3i59rx9nve3a7qeqpdid7 diqi@Dis-Laptop lantern %

The error I get with -j

Ngalstyan4 · 2024-08-28T02:46:33Z

docker/Dockerfile.dev

@@ -55,7 +55,7 @@ COPY . .
 RUN sudo rm -rf build \
    && mkdir build \
    && cd build \
-    && cmake -DCMAKE_BUILD_TYPE=Debug .. \
+    && cmake .. \


why change this?

The dockerfile is meant for development and I often use GDB from inside it, so having a build with symbols is often good.

Though, I usually attach a folder from host and rebuild the DB for debugging so not a big deal but would still like to know why the change.

The build did not succeed for me without it

dqii force-pushed the di/weighted-vector-search branch 9 times, most recently from 863cdc9 to 5e89295 Compare August 18, 2024 02:09

Support sparsevec in weighted vector search function

b689c2d

dqii force-pushed the di/weighted-vector-search branch from 5e89295 to b689c2d Compare August 18, 2024 09:01

dqii marked this pull request as ready for review August 18, 2024 09:35

dqii requested review from Ngalstyan4 and var77 August 18, 2024 09:35

greptile-apps bot reviewed Aug 18, 2024

View reviewed changes

Ngalstyan4 reviewed Aug 18, 2024

View reviewed changes

var77 and others added 5 commits August 20, 2024 17:34

use pgvector v0.7.3-lanterncloud in CI and development dockerfile

ec924eb

remove unnecessary assert for external_socket_fd

decf208

Change version to 0.3.3, update tests given pgvector 0.7+

c661fea

Fix test difference due to CPU architecture inconsistency

4a20f39

Simplify code

2eff4f1

Ngalstyan4 reviewed Aug 28, 2024

View reviewed changes

dqii force-pushed the di/weighted-vector-search branch from 4c52324 to 6d6c72d Compare September 3, 2024 01:57

Add loop test

17bc1f5

dqii force-pushed the di/weighted-vector-search branch from 6d6c72d to 17bc1f5 Compare September 3, 2024 03:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support sparsevec in weighted vector search function #328

Support sparsevec in weighted vector search function #328

dqii commented Aug 16, 2024

github-actions bot commented Aug 16, 2024 •

edited

Loading

codecov bot commented Aug 18, 2024

greptile-apps bot left a comment

greptile-apps bot Aug 18, 2024

greptile-apps bot Aug 18, 2024

greptile-apps bot Aug 18, 2024

greptile-apps bot Aug 18, 2024

greptile-apps bot Aug 18, 2024

greptile-apps bot Aug 18, 2024

greptile-apps bot Aug 18, 2024

greptile-apps bot Aug 18, 2024

greptile-apps bot Aug 18, 2024

var77 Aug 21, 2024

dqii Aug 25, 2024

greptile-apps bot Aug 18, 2024

Ngalstyan4 Aug 18, 2024

Ngalstyan4 Aug 28, 2024

dqii Aug 31, 2024 •

edited

Loading

Ngalstyan4 Aug 28, 2024

dqii Aug 28, 2024

		SELECT '{1:0.4,2:0.3,3:0.2}/3' AS s3 \gset
		SELECT '[-0.5,-0.1,-0.3]' AS v3 \gset

Support sparsevec in weighted vector search function #328

Are you sure you want to change the base?

Support sparsevec in weighted vector search function #328

Conversation

dqii commented Aug 16, 2024

github-actions bot commented Aug 16, 2024 • edited Loading

Benchmarks

codecov bot commented Aug 18, 2024

Codecov Report

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot Aug 18, 2024

Choose a reason for hiding this comment

greptile-apps bot Aug 18, 2024

Choose a reason for hiding this comment

greptile-apps bot Aug 18, 2024

Choose a reason for hiding this comment

greptile-apps bot Aug 18, 2024

Choose a reason for hiding this comment

greptile-apps bot Aug 18, 2024

Choose a reason for hiding this comment

greptile-apps bot Aug 18, 2024

Choose a reason for hiding this comment

greptile-apps bot Aug 18, 2024

Choose a reason for hiding this comment

greptile-apps bot Aug 18, 2024

Choose a reason for hiding this comment

greptile-apps bot Aug 18, 2024

Choose a reason for hiding this comment

var77 Aug 21, 2024

Choose a reason for hiding this comment

dqii Aug 25, 2024

Choose a reason for hiding this comment

greptile-apps bot Aug 18, 2024

Choose a reason for hiding this comment

Ngalstyan4 Aug 18, 2024

Choose a reason for hiding this comment

Ngalstyan4 Aug 28, 2024

Choose a reason for hiding this comment

dqii Aug 31, 2024 • edited Loading

Choose a reason for hiding this comment

Ngalstyan4 Aug 28, 2024

Choose a reason for hiding this comment

dqii Aug 28, 2024

Choose a reason for hiding this comment

github-actions bot commented Aug 16, 2024 •

edited

Loading

dqii Aug 31, 2024 •

edited

Loading