Update to arrow/parquet 53.0.0, tonic, prost, object_store, pyo3
#12032
Conversation
Co-authored-by: tison <[email protected]>
… alamb/update_prost
github-actions
bot
added
physical-expr
| Some(1), | ||
| Some(10), | ||
| None, | ||
| Some(0), |
There was a problem hiding this comment.
Due to apache/arrow-rs#6216 (where null counts is now Option)
alamb
changed the title
arrow/parquet `53.0.0arrow/parquet 53.0.0, tonic, prost, etc
| async-trait = "0.1.73" | ||
| aws-config = "1.5.5" | ||
| # begin pin aws-sdk crates otherwise CI MSRV check fails |
There was a problem hiding this comment.
Here is an example failure https://github.com/apache/datafusion/actions/runs/10690110122/job/29633769353?pr=12032
The same thing actually happens on main if you do cargo update in the datafusion-cli directory, just no one had hit that yet
There was a problem hiding this comment.
That is an unfortunate choice by aws.
|
This PR is now ready for review 🎉 (finally, 53.0.0 is released) |
| @@ -401,8 +401,7 @@ fn _regexp_replace_static_pattern_replace<T: OffsetSizeTrait>( | |||
| DataType::Utf8View => { | |||
| let string_view_array = as_string_view_array(&args[0])?; | |||
|
|
|||
| let mut builder = StringViewBuilder::with_capacity(string_view_array.len()) | |||
| .with_block_size(1024 * 1024 * 2); | |||
| let mut builder = StringViewBuilder::with_capacity(string_view_array.len()); | |||
There was a problem hiding this comment.
does that mean the block size is not used in the builder for this case?
There was a problem hiding this comment.
yes, we don't need it after this pr is merged: apache/arrow-rs#6136
I ran a local benchmark and show no perf diff
|
Thanks for the review @comphead and all the help @XiangpengHao -- let's get this merged! |
…`, `pyo3` (apache#12032) * Update prost, prost-derive, pbjson * udpate more * Update datafusion/substrait/Cargo.toml Co-authored-by: tison <[email protected]> * Update vendored code * revert upgrade in datafusion-examples until arrow-flight is updated * Pin to pre-release arrow-rs * update pyo3 * Update to use new arrow apis * update for new api * Update tonic in examples * update prost * update datafusion-cli/cargo * update test output * update * updates * updates * update math * update more * fix scalar tests * Port statistics to use new API * factor into a function * update generated files * Update test * add new test * update tests * tapelo format * Update other tests * Update datafusion pin * Update for API change * Update to arrow 53.0.0 sha * Update cli deps * update cargo.lock * Update expected output * Remove patch * update datafusion-cli cargo * Pin some aws sdks whose update caused CI failures --------- Co-authored-by: tison <[email protected]>
Draft as this is waiting on the actual arrow 53.0.0 release: apache/arrow-rs#6016Which issue does this PR close?
N/A
Rationale for this change
We need to update the DataFusion ecosystem
What changes are included in this PR?
arrow/parquet`53.0.0prost,prost-derive,pbjson,tonicecosystem #11372object_storeto 0.11.0 (TBD): Release object store0.11.0(breaking API) around Aug 15 2024 arrow-rs#6121Are these changes tested?
Yes by CI
Are there any user-facing changes?