Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[minor] make recursive package dependency optional #13778

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

[minor] make recursive package dependency optional #13778

wants to merge 5 commits into from
+67 −64

Conversation

buraksenn
Copy link
Contributor

@buraksenn buraksenn commented Dec 14, 2024 •
edited
Loading

Which issue does this PR close?

Closes #13766

Rationale for this change

Adding recursive package causes issues for downstream projects

What changes are included in this PR?

Made recursive package optional in packages that use it instead of top level dependency

Are these changes tested?

Existing tests. Especially this one that fails when recursive is off:

    #[test]
    fn test_large_tree() {
        let mut item = TestTreeNode::new_leaf("initial".to_string());
        for i in 0..3000 {
            item = TestTreeNode::new(vec![item], format!("parent-{}", i));
        }

        let mut visitor =
            TestVisitor::new(Box::new(visit_continue), Box::new(visit_continue));

        item.visit(&mut visitor).unwrap();
    }

@github-actions github-actions bot added sql SQL Planner logical-expr Logical plan and expressions optimizer Optimizer rules common Related to common crate labels Dec 14, 2024
@buraksenn buraksenn changed the title minor make recursive optional [minor] make recursive package dependency optional Dec 14, 2024
@blaginin
Copy link
Contributor

should we add it to the docs?

@github-actions github-actions bot added the documentation Improvements or additions to documentation label Dec 14, 2024
@buraksenn
Copy link
Contributor Author

should we add it to the docs?

Thanks. Added to README.md file

alamb
alamb reviewed Dec 15, 2024
View reviewed changes
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @buraksenn

I wonder if there is some way to test this (aka add a CI test that will fail with recursive protection -- perhaps the WASM compile test 🤔 ) that doesn't fail when the feature is disabled?

That way we can ensure we have the configuration settings hooked up right

@buraksenn buraksenn force-pushed the make-recursive-optional branch from c352f8a to f92ee10 Compare December 15, 2024 16:12
findepi
findepi reviewed Dec 17, 2024
View reviewed changes
Copy link
Member

@findepi findepi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for working on this!

Cargo.toml
@@ -143,7 +143,6 @@ pbjson = { version = "0.7.0" }
prost = "0.13.1"
prost-derive = "0.13.1"
rand = "0.8"
recursive = "0.1.1"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert, the version should still be defined here

datafusion/common/Cargo.toml
@@ -61,7 +63,7 @@ object_store = { workspace = true, optional = true }
parquet = { workspace = true, optional = true, default-features = true }
paste = "1.0.15"
pyo3 = { version = "0.22.0", optional = true }
recursive = { workspace = true }
recursive = { version = "0.1.1", optional = true }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
recursive = { version = "0.1.1", optional = true }
recursive = { workspace = true, optional = true }

same in other places

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, define default recursive-protection feature and make it a default feature
add dep:recursive to it

datafusion/common/Cargo.toml
avro = ["apache-avro"]
backtrace = []
pyarrow = ["pyo3", "arrow/pyarrow", "parquet"]
force_hash_collisions = []
recursive-protection = ["recursive"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
recursive-protection = ["recursive"]
recursive-protection = ["dep:recursive"]

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see https://doc.rust-lang.org/cargo/reference/features.html#optional-dependencies

image

datafusion/expr/Cargo.toml
@@ -36,6 +36,8 @@ name = "datafusion_expr"
path = "src/lib.rs"

[features]
default = ["recursive-protection"]
recursive-protection = ["recursive"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dep:

datafusion/expr/Cargo.toml
@@ -48,7 +50,7 @@ datafusion-functions-window-common = { workspace = true }
datafusion-physical-expr-common = { workspace = true }
indexmap = { workspace = true }
paste = "^1.0"
recursive = { workspace = true }
recursive = { version = "0.1.1", optional = true }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use workspace version

findepi
findepi reviewed Dec 17, 2024
View reviewed changes
datafusion/optimizer/src/eliminate_cross_join.rs
" TableScan: t1 [a:UInt32, b:UInt32, c:UInt32]",
" TableScan: t2 [a:UInt32, b:UInt32, c:UInt32]",
"Filter: t2.c < UInt32(15) OR t2.c = UInt32(688) [a:UInt32, b:UInt32, c:UInt32, a:UInt32, b:UInt32, c:UInt32]",
" Inner Join: t1.a + UInt32(100) = t2.a * UInt32(2) [a:UInt32, b:UInt32, c:UInt32, a:UInt32, b:UInt32, c:UInt32]",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unrelated change?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alamb alamb alamb left review comments

@findepi findepi findepi left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
common Related to common crate documentation Improvements or additions to documentation logical-expr Logical plan and expressions optimizer Optimizer rules sql SQL Planner
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Making the recursive dependency an optional feature
4 participants
@buraksenn @blaginin @findepi @alamb