Patched DataFusion, Oct 15, 2024

[wiedld]

This takes us up to Oct-15 SHA 399e840.

It advances DF by adding these oct-7 to oct-15 commits: 577e4bb...influxdata:arrow-datafusion:399e8403d2fb5a058fb263bc2449dbc9d7529841

Patches applied

• We are still manually patching upstream commits, for these fixes:

  • oct-16 = commits from pr Fix logical vs physical schema mismatch for aliased now() apache/datafusion#12951
  • Oct-20 = apache@8d4614d
  • Oct-23 = commits from pr Fix logical vs physical schema mismatch for UNION where some inputs are constants apache/datafusion#12954
  • Nov-3 = apache@85f92ef
  • Nov-11 = commits from pr: Fix ExprSchema extraction of metadata for Cast expressions. apache/datafusion#13305
  • clippy updates (presumably these are a bunch of separate upstream commits)

• SanityCheckPlan

  • I'm not sure when we can stop skipping the SanityCheckPlan. I'm unclear how it relates to the additional sort pushdown fixes, including a recent fix done here: [bug]: Fix wrong order by removal from plan apache/datafusion#13497

• Patches still not fixed:

  • commit reverted due to CPU spinning bug (new)
  • This empty projection patch, since I only ever found these plans in iox-created influqxl LP.
    • patch: 001865d
    • iox issue: https://github.com/influxdata/influxdb_iox/issues/12407

[wiedld]

@alamb -- I had to add a single additional patch commit, due to an upstream commit from oct-8.

The test case correctly shows that bit_length(utf8view) statement should fail.
However, I'm not really sure how this statement was properly failing before (on datafusion main)?

Steps I took to debug:

1. without changes:
   • we get a utf8 view type for the column, post coercion
   • the statement bit_length(utf8view) was not failing on our DF upgrade branch.
2. I changed:
   • made the signature of bit_length only accept utf8 and large utf (not the views), since I think that oct-8 upstream commit made a wrong change. Note that the evoke does not handle view types.
3. but the tests still passed.
   • it looks like type coercion changed the utf8 view to large utf8.

So this branch now passes all DF tests, but I'm still unsure how DF main is ok with this test. (This was also called out in the PR itself.) 🤔

[alamb]

Patches applied

Thank you for this list

However, I'm not really sure how this statement was properly failing before (on datafusion main)?

I believe this was fixed in a newer version of arrow-rs that was released after that DataFusion commit

• Support Utf8View for bit_length kernel apache/arrow-rs#6671

Since IOx is now running a newer arrow in my mind that explains the test difference

[alamb]

We are still manually patching upstream commits, for these fixes:

BTW I have added a discussion item for our next meeting about these patches (aka how should we be thinking about them upstream)

[wiedld]

We are now on Oct-30th. Closing this PR.