fix(11367): parquet writer defaults #34
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alamb
reviewed
Jul 17, 2024
There was a problem hiding this comment.
Thanks @wiedld -- this PR looks nice. I left some comments
Once we merge apache#11444 I think this would be a great PR to send upstream
| "datafusion's default is None" | ||
| ); | ||
|
|
||
| // TODO: matches once create WriterProps, but only due to parquet's override |
There was a problem hiding this comment.
I don't understand this comment -- is there any action item here?
There was a problem hiding this comment.
I added another commit to re-work the code comments for this test. Hopefully that makes the misalignments more clear.
Please let me know @alamb if I should change anything else before moving this over into an upstream PR.
|
Have opened the upstream PR. |
wiedld
pushed a commit
that referenced
this pull request
Oct 9, 2024
* Postgres: enforce required `NUMERIC` type for `round` scalar function (#34) Includes initial support for dialects to override scalar functions unparsing * Document scalar_function_to_sql_overrides fn
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WIP: first we need consensus on the desired defaults. Second, decide if the extern parquet's defaults can (or cannot) override the datafusion default.
Which issue does this PR close?
Closes apache#11367
Rationale for this change
When we switched from using the parquet's ArrowWriter (with options) to the parallelized parquet writer (with it's own options), we ran into unintended behaviors due to different default settings.
Here are the places where the current defaults differ:
† For these settings, datafusion has no default (None). However, once datafusion's ParquetOptions are used by the extern parquet (a.k.a. converted to parquet's ArrowWriterOptions) then it uses the extern parquet's defaults. Refer to the newly added tests.
.
Additionally, there are differences in the bloom filter configurations based upon partial definition (a.k.a. leaving some as default, and some as defined):
What changes are included in this PR?
The first commit is adding tests to define and demonstrate the differences in the defaults.
After discussion and consensus, we'll add other commits (as needed) for implementing the desired changes.
Are these changes tested?
Yes.
Are there any user-facing changes?
No API changes. Only potential future changes to alleviate unintended consequences.