Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Very slow reads of datasets with compession by gathering or DSG ragged arrays #260

Closed
davidhassell opened this issue Jun 19, 2023 · 0 comments · Fixed by #261
Closed

Very slow reads of datasets with compession by gathering or DSG ragged arrays #260

davidhassell opened this issue Jun 19, 2023 · 0 comments · Fixed by #261
Labels
bug Something isn't working netCDF read Relating to reading netCDF datasets performance Relating to speed and memory performance
Milestone
1.10.1.1

Comments

@davidhassell
Copy link
Contributor

In versions 1.10.0.1 to 1.10.1.0, reading datasets with compression by gathering or DSG ragged arrays is very slow, taking seconds rather than milliseconds.

This is due to a bug in cfdm.Data._original_filenames, which causes every integer element in the list/count/index variable to be assessed for an original filename, which, of course, it does not have!

PR to follow.
@davidhassell davidhassell added bug Something isn't working performance Relating to speed and memory performance labels Jun 19, 2023
@davidhassell davidhassell added this to the Next release milestone Jun 19, 2023
This was referenced Jun 19, 2023
Merged
Merged
@davidhassell davidhassell added the netCDF read Relating to reading netCDF datasets label Nov 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working netCDF read Relating to reading netCDF datasets performance Relating to speed and memory performance
Projects
None yet
Milestone
1.10.1.1
Development

Successfully merging a pull request may close this issue.

Fix very slow reads of datasets with compession by gathering or DSG ragged arrays davidhassell/cfdm
1 participant
@davidhassell