Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move Pruning into physical-optimizer crate #13485

Merged
merged 4 commits into from
Nov 20, 2024
Merged

Conversation

irenjj
Copy link
Contributor

@irenjj irenjj commented Nov 19, 2024

Which issue does this PR close?

part of #11502.

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions bot added optimizer Optimizer rules core Core DataFusion crate labels Nov 19, 2024
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @irenjj -- this looks very nice!

I think we need to clean up some of the dependencies, but it is really close

Thank you!

Comment on lines 41 to 42
datafusion-expr = { workspace = true, default-features = true }
datafusion-functions-nested = { workspace = true }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please move datafusion-functions-nested to dev-dependencies?

I also think we can avoid the dependency on datafusion-expr by changing

use datafusion_expr::Operator;

to

use datafusion_expr_common::operator::Operator;

@@ -33,13 +33,17 @@ workspace = true

[dependencies]
arrow = { workspace = true }
arrow-array = { workspace = true }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If possible, can you avoid this explicit dependency? Whatever is used in arrow-array should be available in arrow and then we can keep the dependency chain simpler

@@ -22,12 +22,10 @@
use std::collections::HashSet;
Copy link
Contributor Author

@irenjj irenjj Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

datafusion-expr dependency is needed by //! [Expr]: crate::prelude::Expr, should we remove the comment to avoid introducing new dependencies? @alamb

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In other places, we have changed the link to docs.rs directly. So in this case, change the link to

https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.Expr.html

recursive = { workspace = true }

[dev-dependencies]
datafusion-expr = { workspace = true }
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needed in test.

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great to me -- thank you @irenjj

@jayzhan211 jayzhan211 merged commit 30ff48e into apache:main Nov 20, 2024
27 checks passed
@jayzhan211
Copy link
Contributor

Thanks @irenjj @alamb

@irenjj irenjj deleted the move_pruning branch November 20, 2024 03:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Core DataFusion crate optimizer Optimizer rules
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants