Add contains function, and support in datafusion substrait consumer #10879
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
waynexia
reviewed
Jun 12, 2024
| ) -> Result<ArrayRef, DataFusionError> { | ||
| let mod_str = as_generic_string_array::<T>(&args[0])?; | ||
| let match_str = as_generic_string_array::<T>(&args[1])?; | ||
| let res = arrow::compute::kernels::comparison::regexp_is_match_utf8( |
There was a problem hiding this comment.
Do we need to escape search string as it's used in regexp? Wondering what's the result of contains("abcdefg", ".*")
There was a problem hiding this comment.
@Lordworms can you possibly respond to @waynexia 's comment?
There was a problem hiding this comment.
To keep the review queue clear, I filed #10929 to follow up on this item and will merge this PR in
waynexia
approved these changes
Jun 12, 2024
alamb
changed the title
alamb
changed the title
alamb
reviewed
Jun 12, 2024
There was a problem hiding this comment.
Can we also please add an entry to the documentation about this new function? https://datafusion.apache.org/user-guide/sql/scalar_functions.html#scalar-functions
alamb
reviewed
Jun 12, 2024
| @@ -149,6 +149,9 @@ pub mod expr_fn { | |||
| ),( | |||
| uuid, | |||
| "returns uuid v4 as a string value", | |||
| ), ( | |||
| contains, | |||
| "Return true if search_string is found within string.", | |||
There was a problem hiding this comment.
I think we should also mention here that search_string is treated like a regex if that is the intention
Sure, sorry for the ignorance |
Weijun-H
reviewed
Jun 14, 2024
Co-authored-by: Alex Huang <[email protected]>
|
Please add an entry here datafusion/docs/source/user-guide/sql/scalar_functions.md Lines 645 to 646 in 1a2a1bf |
|
The CI failure seems unrelated to the code in this PR: https://github.com/apache/datafusion/actions/runs/9510703358?pr=10879 |
|
Thanks again @waynexia @Lordworms and @Weijun-H ! |
devinjdangelo
pushed a commit
to devinjdangelo/arrow-datafusion
that referenced
this pull request
Jun 16, 2024
…pache#10879) * adding new function contains * adding substrait test * adding doc * adding doc * Update docs/source/user-guide/sql/scalar_functions.md Co-authored-by: Alex Huang <[email protected]> * adding entry --------- Co-authored-by: Alex Huang <[email protected]>
alamb
added a commit
that referenced
this pull request
Jun 28, 2024
…11060) * wip create and register ext file types with session * Add contains function, and support in datafusion substrait consumer (#10879) * adding new function contains * adding substrait test * adding doc * adding doc * Update docs/source/user-guide/sql/scalar_functions.md Co-authored-by: Alex Huang <[email protected]> * adding entry --------- Co-authored-by: Alex Huang <[email protected]> * logical planning updated * compiling * removing filetype enum * compiling * working on tests * fix some tests * test fixes * cli fix * cli fmt * Update datafusion/core/src/datasource/file_format/mod.rs Co-authored-by: Andrew Lamb <[email protected]> * Update datafusion/core/src/execution/session_state.rs Co-authored-by: Andrew Lamb <[email protected]> * review comments * review comments * review comments * typo fix * fmt * fix err log style * fmt --------- Co-authored-by: Lordworms <[email protected]> Co-authored-by: Alex Huang <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>
comphead
pushed a commit
to comphead/arrow-datafusion
that referenced
this pull request
Jul 2, 2024
…pache#11060) * wip create and register ext file types with session * Add contains function, and support in datafusion substrait consumer (apache#10879) * adding new function contains * adding substrait test * adding doc * adding doc * Update docs/source/user-guide/sql/scalar_functions.md Co-authored-by: Alex Huang <[email protected]> * adding entry --------- Co-authored-by: Alex Huang <[email protected]> * logical planning updated * compiling * removing filetype enum * compiling * working on tests * fix some tests * test fixes * cli fix * cli fmt * Update datafusion/core/src/datasource/file_format/mod.rs Co-authored-by: Andrew Lamb <[email protected]> * Update datafusion/core/src/execution/session_state.rs Co-authored-by: Andrew Lamb <[email protected]> * review comments * review comments * review comments * typo fix * fmt * fix err log style * fmt --------- Co-authored-by: Lordworms <[email protected]> Co-authored-by: Alex Huang <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>
findepi
pushed a commit
to findepi/datafusion
that referenced
this pull request
Jul 16, 2024
…pache#10879) * adding new function contains * adding substrait test * adding doc * adding doc * Update docs/source/user-guide/sql/scalar_functions.md Co-authored-by: Alex Huang <[email protected]> * adding entry --------- Co-authored-by: Alex Huang <[email protected]>
findepi
pushed a commit
to findepi/datafusion
that referenced
this pull request
Jul 16, 2024
…pache#11060) * wip create and register ext file types with session * Add contains function, and support in datafusion substrait consumer (apache#10879) * adding new function contains * adding substrait test * adding doc * adding doc * Update docs/source/user-guide/sql/scalar_functions.md Co-authored-by: Alex Huang <[email protected]> * adding entry --------- Co-authored-by: Alex Huang <[email protected]> * logical planning updated * compiling * removing filetype enum * compiling * working on tests * fix some tests * test fixes * cli fix * cli fmt * Update datafusion/core/src/datasource/file_format/mod.rs Co-authored-by: Andrew Lamb <[email protected]> * Update datafusion/core/src/execution/session_state.rs Co-authored-by: Andrew Lamb <[email protected]> * review comments * review comments * review comments * typo fix * fmt * fix err log style * fmt --------- Co-authored-by: Lordworms <[email protected]> Co-authored-by: Alex Huang <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #10861
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?