Fix parquet statistics for ListingTable and Utf8View with schema_force_string_view, rename config option to schema_force_view_types
#12232
Conversation
github-actions
bot
added
core
github-actions
bot
added
the
documentation
…ll solve the issue
alamb
changed the title
schema_force_string_view.schema_force_string_view.
There was a problem hiding this comment.
Thank you @wiedld -- this PR is really nicely thought out and implemented.
I think the idea of putting the "should we use view types" on to the ParquetFormat is much nicer than passing around boolean flags
I left some suggestoons on this PR for naming and comments
I think now that the next arrow release is in #12032 , perhaps you can merge up this branch and fixup the tests in preparation for merge?
| @@ -380,6 +380,10 @@ config_namespace! { | |||
| /// the filters are applied in the same order as written in the query | |||
| pub reorder_filters: bool, default = false | |||
|
|
|||
| /// (reading) If true, parquet reader will read columns of `Utf8/Utf8Large` with `Utf8View`, | |||
There was a problem hiding this comment.
is this change so that all the reading configuration values are before the writing ones?
There was a problem hiding this comment.
While reviewing / considering this PR, I wonder if we should (in a follow on PR) rename this config flag to be schema_force_view_types as it also applies to binary columns 🤔
There was a problem hiding this comment.
Ah...I just did it all in this PR because it felt weird to have 2 naming conventions at once. Hopefully that's ok. 😅
github-actions
bot
added
sqllogictest
wiedld
force-pushed
the
12123/view-type-on-parquet-read
branch
from
cb30be7 to
2438136
Compare
alamb
changed the title
schema_force_string_view.schema_force_string_view, rename config option to schema_force_view_types
…ew types are used
wiedld
force-pushed
the
12123/view-type-on-parquet-read
branch
from
2438136 to
02036fb
Compare
|
Thanks again @wiedld |
Which issue does this PR close?
Closes #12123
Rationale for this change
On write: parquet file written with utf8/large-utf & binary/large-binary schema (so is in metadata).
On read: we would like to be able to read as the more performant view types.
Previous work has already used the
schema_force_string_viewto read into utf8view and binaryview, by passing around a bool to the ParquetOpener.This work is to focused on getting the parquet statistics, on read, to properly compute when reading as view types.
What changes are included in this PR?
schema_force_string_viewup a few lines, to be with the "read" (not write) config options.schema_force_string_viewThere are two tests marked as incomplete: an expected panic and a commented out test. Both tests will successfully pass once the next (anticipated) arrow release occurs. In order to prove this, there is a PoC over here which branches of the in-progress WIP for the arrow upgrade and then adds the commits from this PR.. I merged in the latest main with the arrow-rs release (thank you!) and those tests are now passing.Are these changes tested?
Yes.
Are there any user-facing changes?
No.