Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rewrite query #1415

Merged
merged 5 commits into from
Dec 31, 2021
Merged

Rewrite query #1415

merged 5 commits into from
Dec 31, 2021
+284 −422

Conversation

aliubimov
Copy link
Contributor

No description provided.

@aliubimov aliubimov added the trigger-test Label trigger to run tests on PRs label Dec 27, 2021
@activeloop-bot activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Dec 27, 2021
@aliubimov aliubimov changed the title Compile query Rewrite query Dec 28, 2021
@aliubimov aliubimov added the trigger-test Label trigger to run tests on PRs label Dec 28, 2021
@activeloop-bot activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Dec 28, 2021
aliubimov
aliubimov commented Dec 28, 2021
View reviewed changes
hub/core/query/query.py
return r


def _get_np(dataset: Dataset, block: IOBlock) -> NP_ACCESS:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lazy loading of tensor.numpy()

@aliubimov aliubimov added the trigger-test Label trigger to run tests on PRs label Dec 28, 2021
@activeloop-bot activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Dec 28, 2021
aliubimov
aliubimov commented Dec 28, 2021
View reviewed changes
hub/core/query/filter.py Outdated
index_map: List[int]

if num_workers > 0:
index_map = query_inplace(dataset, query, progressbar)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A stub for now.

@aliubimov aliubimov added the trigger-test Label trigger to run tests on PRs label Dec 28, 2021
@activeloop-bot activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Dec 28, 2021
@aliubimov aliubimov added the trigger-test Label trigger to run tests on PRs label Dec 28, 2021
@activeloop-bot activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Dec 28, 2021
@codecov
Copy link

codecov bot commented Dec 28, 2021 •
edited
Loading

Codecov Report

Merging #1415 (b92c5c7) into main (08c1e15) will decrease coverage by 0.26%.
The diff coverage is 74.46%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1415      +/-   ##
==========================================
- Coverage   92.33%   92.06%   -0.27%     
==========================================
  Files         175      175              
  Lines       14111    14043      -68     
==========================================
- Hits        13029    12929     -100     
- Misses       1082     1114      +32
Flag Coverage Δ
unittests 92.06% <74.46%> (-0.27%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
hub/core/query/filter.py 58.06% <63.63%> (-16.94%) ⬇️
hub/core/compute/process.py 91.30% <71.42%> (-8.70%) ⬇️
hub/core/query/query.py 76.55% <73.84%> (-10.26%) ⬇️
hub/core/dataset/dataset.py 93.78% <100.00%> (ø)
hub/core/query/__init__.py 100.00% <100.00%> (ø)
hub/core/query/test/test_query.py 100.00% <100.00%> (ø)
hub/core/tests/test_compute.py 100.00% <100.00%> (ø)
hub/core/dataset/hub_cloud_dataset.py 88.99% <0.00%> (-2.76%) ⬇️
hub/core/lock.py 85.24% <0.00%> (+1.63%) ⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 08c1e15...b92c5c7. Read the comment docs.

@aliubimov aliubimov added the trigger-test Label trigger to run tests on PRs label Dec 28, 2021
@activeloop-bot activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Dec 28, 2021
@aliubimov aliubimov added the trigger-test Label trigger to run tests on PRs label Dec 28, 2021
@activeloop-bot activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Dec 28, 2021
@aliubimov aliubimov added the trigger-test Label trigger to run tests on PRs label Dec 29, 2021
@activeloop-bot activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Dec 29, 2021
@aliubimov aliubimov added the trigger-test Label trigger to run tests on PRs label Dec 29, 2021
@activeloop-bot activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Dec 29, 2021
@aliubimov aliubimov added the trigger-test Label trigger to run tests on PRs label Dec 29, 2021
@activeloop-bot activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Dec 29, 2021
@aliubimov aliubimov added the trigger-test Label trigger to run tests on PRs label Dec 29, 2021
@activeloop-bot activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Dec 29, 2021
@aliubimov aliubimov added the trigger-test Label trigger to run tests on PRs label Dec 29, 2021
@activeloop-bot activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Dec 29, 2021
@aliubimov aliubimov added the trigger-test Label trigger to run tests on PRs label Dec 29, 2021
@aliubimov aliubimov mentioned this pull request Dec 29, 2021
Closed
7 tasks
@activeloop-bot activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Dec 29, 2021
@aliubimov aliubimov added the trigger-test Label trigger to run tests on PRs label Dec 30, 2021
@activeloop-bot activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Dec 30, 2021
@aliubimov aliubimov added the trigger-test Label trigger to run tests on PRs label Dec 30, 2021
@activeloop-bot activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Dec 30, 2021
@aliubimov aliubimov added the trigger-test Label trigger to run tests on PRs label Dec 30, 2021
@activeloop-bot activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Dec 30, 2021
@aliubimov aliubimov added the trigger-test Label trigger to run tests on PRs label Dec 30, 2021
@activeloop-bot activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Dec 30, 2021
davidbuniat
davidbuniat requested changes Dec 30, 2021
View reviewed changes
Copy link
Member

@davidbuniat davidbuniat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we drop support for tensor_functions, groups, json etc. since unit tests are removed?

@aliubimov
Copy link
Contributor Author

aliubimov commented Dec 30, 2021 •
edited
Loading

Did we drop support for tensor_functions, groups, json etc. since unit tests are removed?

Tests was removed as they are not compatible with never version of query and have to be rewritten.
Tensor functions covered by query_test, groups have it own set of unit tests. Json have to be added.

davidbuniat
davidbuniat approved these changes Dec 30, 2021
View reviewed changes
hub/core/query/filter.py
finally:
bar.close()
else:
return DatasetQuery(dataset, query).execute()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

restrict the scope of the query and also make sure .compile is called beforehand not on every eval

@aliubimov aliubimov merged commit 6d4693d into activeloopai:main Dec 31, 2021
@aliubimov aliubimov deleted the query/compile branch December 31, 2021 00:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AbhinavTuli AbhinavTuli AbhinavTuli approved these changes

@davidbuniat davidbuniat davidbuniat approved these changes

@jraman jraman Awaiting requested review from jraman

@farizrahman4u farizrahman4u Awaiting requested review from farizrahman4u

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@aliubimov @davidbuniat @AbhinavTuli @activeloop-bot