Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warning issued if limit is larger than num rows #3086

Open
wjones127 opened this issue Nov 4, 2024 · 2 comments · May be fixed by #3248
Open

Warning issued if limit is larger than num rows #3086

wjones127 opened this issue Nov 4, 2024 · 2 comments · May be fixed by #3248
Labels
bug Something isn't working good first issue Good for newcomers

Comments

@wjones127
Copy link
Contributor

When we do head(), we should check the number of rows in the table before performing take, and just use the lower value. That way we avoid this unnecessary warning.

https://github.com/lancedb/lance/blob/ceaf49c9d24eab6cf2dd61712849cb4f3bd14d69/rust/lance/src/dataset/scanner.rs#L1189C43-L1189C44

We actually seem to do this for offset, just not limit.

import lance
import pyarrow as pa

data = pa.table({"x": [1, 2, 3]})
ds = lance.write_dataset(data, "./repro_limit")
ds.to_table(limit=10)
[2024-11-04T21:03:31Z WARN  lance::io::exec::scan] Ran out of fragments before we were done scanning for range: 0..10
pyarrow.Table
x: int64
----
x: [[1,2,3]]

Same with head:

ds.head(10)
[2024-11-04T21:03:42Z WARN  lance::io::exec::scan] Ran out of fragments before we were done scanning for range: 0..10
pyarrow.Table
x: int64
----
x: [[1,2,3]]
@wjones127 wjones127 added bug Something isn't working good first issue Good for newcomers labels Nov 4, 2024
@wjones127
Copy link
Contributor Author

This came from #2747

@takaebato
Copy link

takaebato commented Dec 13, 2024

I'll work on this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants