Interaction constraints #3126

btrotta · 2020-05-28T07:45:25Z

Implement interaction constraints feature requested in #2884.

jameslamb

Thanks @btrotta ! I left some minor comments, but will defer to C++ maintainers so I'll just leave a "Comment" review.

jameslamb · 2020-05-30T01:19:36Z

docs/Parameters.rst

+
+   -  for Python, specify as list of lists, e.g. ``[[0, 1, 2], [2, 3]]``
+
+   -  for R, specify as a string similar to the CLI format (ensuring there are no spaces), e.g. ``"[0,1,2],[2,3]"``


For R, I think that we should make the interface "a list of character vectors, where each vector is a list of feature names", and then inside the R package turn the feature names into indices.

I think passing a string of an array in a comma-delimited list puts too much burden on R users. @Laurae2 what do you think?

@jameslamb Good suggestion, thanks! I've tried to implement this, but I'm not that familiar with R so would be great if you could take a look when you get a chance.

I think it would be ok to say here that it is not supported in the R package yet, and then create an issue like [R-package] add support for interaction_constraints and add it to #2302 . Then I or another R maintainer can get to it without delaying this pull request.

Done, see #3130

jameslamb · 2020-05-30T01:21:52Z

python-package/lightgbm/basic.py

-            pairs.append(str(key) + '=' + ','.join(map(str, val)))
+            def to_string(x):
+                if isinstance(x, list):
+                    return '[' + ','.join(map(str, x)) + ']'


Suggested change

return '[' + ','.join(map(str, x)) + ']'

return "[{}]".format(','.join(map(str, x)))

I think this would be a little easier to read using string interpolation. I recommend .format() since it's an option supported in Python 2 and 3. @StrikerRUS what do you think?

jameslamb · 2020-05-31T17:19:51Z

R-package/R/lgb.train.R

@@ -124,6 +124,60 @@ lgb.train <- function(params = list(),
    end_iteration <- begin_iteration + nrounds - 1L
  }

+  # Convert interaction constraints to feature numbers
+  if (!is.null(params[["interaction_constraints"]])) {


thanks for this!

@btrotta I'll leave some comments here and you can decide, but I think it would make sense to do this in a separate pull request so it doesn't hold this one up.

Unfortunately we have a lot of duplication between lgb.train() and lgb.cv() today
(https://github.com/microsoft/LightGBM/blob/master/R-package/R/lgb.cv.R), so handling for this parameter needs to be added in both places. I think it would make sense to create a new internal function like .check_interaction_constraints(), similar to how we have lgb.check.obj(), that is then called by both functions. That would also make it easier to unit test the handling of this parameter.

We should add some validation that interaction_constraints is a list of either numeric or character vectors. Like this

interaction_constraints <- params[["interaction_constraints"]] if (!is.null(interaction_constraints)){ # validation if (!methods::is(interaction_constraints, "list")){ stop("interaction_constraints must be a list") } if (!all(sapply(interaction_constraints, function(x){is.character(x) || is.numeric(x)}))){ stop("every element in interaction_constraints must be a character vector or numeric vector) } # rest of the code }

I think the uses of as.list() in this code should be replaced with as.integer()

I'd like to have unit tests, for both lgb.train() and lgb.cv(). That will catch some issues and prevent us from breaking this functionality in the future.

validation that you get an informative error if you don't pass a list object

validation that you get an informative error if you pass a list of something other than character vectors and numeric vectors

validation that you get an informative error for indices that are greater than max_features

test training with only passing indices

test training with only passing feature names

test training with passing a mix of indices and feature names

Ok sure, I have moved these changes to new PR #3136.

guolinke · 2020-06-11T00:26:17Z

@btrotta is this ready to review?

btrotta · 2020-06-11T01:43:14Z

@guolinke Yes

guolinke · 2020-06-12T12:26:12Z

src/treelearner/col_sampler.hpp

-      return std::vector<int8_t>(train_data_->num_features(), 1);
+  std::vector<int8_t> GetByNode(const Tree* tree, int leaf) {
+    // get interaction constraints for current branch
+    std::unordered_set<int> allowed_features;


is that possible to cache the used features of all branches, rather than re-generate them every time?

I've changed the code so that we keep track of the features in each branch as we build the tree.

Thanks so much! However, it seems the tracking is still enabled even when interaction constraint is not used?

Thanks, I changed it now so it only keeps track if interaction constraint is enabled.

guolinke

Thanks very much! Looks good to me

include/LightGBM/config.h

tests/python_package_test/test_engine.py

btrotta · 2020-06-19T06:49:41Z

@StrikerRUS Thanks, have updated according to your suggestions.

StrikerRUS

Many thanks!

mayer79 · 2020-07-05T11:34:35Z

@btrotta: Simply fantastic, thank you so much for digging into this and the great implementation.

I made some tests in R and all make perfect sense:

library(lightgbm)
library(flashlight)

n <- 1000
m <- 4

set.seed(1)
X <- matrix(rnorm(n * m), ncol = m, dimnames = list(NULL, paste0("X", 1:m)))
cols <- colnames(X)
y <- X[, 1] * X[, 2] * X[, 3] + X[, 4] + rnorm(n)

train <- lgb.Dataset(X, label = y)

Example 1: Additive model with depth > 1

params <- list(
  learning_rate = 0.1,
  num_leaves = 15, 
  objective = "regression",
  interaction_constraints = list(1, 2, 3, 4)
)

fit <- lgb.train(params, train, nrounds = 50)

fl <- flashlight(model = fit, data = data.frame(y, X), y = "y", label = "lgb",
                 predict_fun = function(m, X) predict(m, data.matrix(X[, cols])))
plot(imp <- light_interaction(fl, v = cols, pairwise = TRUE))

Okay -> pairwise H-statistic values are all 0. H measures the interaction strength in a model agnostic way based on bivariate partial dependence curves.

Some additive, some pairwise

params <- list(
  learning_rate = 0.1,
  num_leaves = 15, 
  objective = "regression",
  interaction_constraints = list(c(1, 2), 3, 4)
)

fit <- lgb.train(params, train, nrounds = 50)

fl <- flashlight(model = fit, data = data.frame(y, X), y = "y", label = "lgb",
                 predict_fun = function(m, X) predict(m, data.matrix(X[, cols])))
plot(imp <- light_interaction(fl, v = cols, pairwise = TRUE))

Okay -> there is only one interacting variable pair.

Overlapping sets


params <- list(
  learning_rate = 0.1,
  num_leaves = 15, 
  objective = "regression",
  interaction_constraints = list(c(1, 2), c(1, 3))
)

fit <- lgb.train(params, train, nrounds = 50)

fl <- flashlight(model = fit, data = data.frame(y, X), y = "y", label = "lgb",
                 predict_fun = function(m, X) predict(m, data.matrix(X[, cols])))
plot(imp <- light_interaction(fl, v = cols, pairwise = TRUE))

-> Okay as well! X2 and X3 are not interacting, as intended.

And note to myself: Unlisted variables are modelled without interaction.

btrotta · 2020-07-08T06:53:34Z

@mayer79 Thanks for doing those additional tests, good to know it's working as expected!

github-actions · 2023-08-24T12:20:44Z

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.

Add interaction constraints functionality.

0296efa

btrotta requested review from chivee, guolinke, henry0312, jameslamb, Laurae2, StrikerRUS and wxchan as code owners May 28, 2020 07:45

btrotta added 4 commits May 28, 2020 18:27

Minor fixes.

f992ff2

Minor fixes.

3fd5a26

Change lambda to function.

76c0d40

Fix gpu bug, remove extra blank lines.

a23aaa0

btrotta requested a review from huanzhang12 as a code owner May 28, 2020 10:46

btrotta added 4 commits May 28, 2020 21:08

Fix gpu bug.

3b9cdd5

Fix style issues.

b06c11c

Try to fix segfault on MACOS.

072ab16

Fix bug.

935038b

StrikerRUS added the feature label May 28, 2020

btrotta added 2 commits May 29, 2020 12:56

Fix bug.

abadde9

Fix bugs.

ed3afc4

jameslamb reviewed May 30, 2020

View reviewed changes

btrotta added 4 commits May 30, 2020 18:25

Change parameter format for R.

2a4cf11

Fix R style issues.

a047f5f

Change string formatting code.

c1c3ea9

Change docs to say R package not supported.

7cb5fe9

btrotta mentioned this pull request May 31, 2020

[R-package] add support for interaction_constraints #3130

Closed

jameslamb reviewed May 31, 2020

View reviewed changes

Remove R functionality, moving to separate PR.

9129906

btrotta mentioned this pull request Jun 1, 2020

[R-package] Interface for interaction constraints #3136

Merged

Merge.

4b292e1

guolinke reviewed Jun 12, 2020

View reviewed changes

btrotta added 4 commits June 15, 2020 16:24

Keep track of branch features in tree object.

6af7aa1

Merge.

f8a50d2

Only track branch features when feature interactions are enabled.

bd9c043

Fix lint error.

c81b9b3

guolinke approved these changes Jun 18, 2020

View reviewed changes

StrikerRUS reviewed Jun 18, 2020

View reviewed changes

include/LightGBM/config.h Outdated Show resolved Hide resolved

tests/python_package_test/test_engine.py Outdated Show resolved Hide resolved

tests/python_package_test/test_engine.py Outdated Show resolved Hide resolved

Update docs and simplify tests.

bdeaa53

StrikerRUS approved these changes Jun 19, 2020

View reviewed changes

Merge branch 'master' into interaction

83686a4

StrikerRUS merged commit bca2da9 into microsoft:master Jun 23, 2020

StrikerRUS mentioned this pull request Jun 23, 2020

Interaction constraints #2884

Closed

btrotta mentioned this pull request Jun 26, 2020

Fix bug with interaction constraints #3189

Merged

shiyu1994 mentioned this pull request Nov 9, 2021

[CUDA] Feature Interaction Constraints on New CUDA Version #4785

Closed

github-actions bot locked as resolved and limited conversation to collaborators Aug 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Interaction constraints #3126

Interaction constraints #3126

btrotta commented May 28, 2020

jameslamb left a comment

jameslamb May 30, 2020

btrotta May 30, 2020

jameslamb May 30, 2020

btrotta May 31, 2020

jameslamb May 30, 2020

jameslamb May 31, 2020 •

edited

Loading

btrotta Jun 1, 2020

guolinke commented Jun 11, 2020

btrotta commented Jun 11, 2020

guolinke Jun 12, 2020

btrotta Jun 15, 2020

guolinke Jun 17, 2020

btrotta Jun 18, 2020

guolinke left a comment

btrotta commented Jun 19, 2020

StrikerRUS left a comment

mayer79 commented Jul 5, 2020

btrotta commented Jul 8, 2020

github-actions bot commented Aug 24, 2023


		- for Python, specify as list of lists, e.g. ``[[0, 1, 2], [2, 3]]``

		- for R, specify as a string similar to the CLI format (ensuring there are no spaces), e.g. ``"[0,1,2],[2,3]"``

	return '[' + ','.join(map(str, x)) + ']'
	return "[{}]".format(','.join(map(str, x)))

Interaction constraints #3126

Interaction constraints #3126

Conversation

btrotta commented May 28, 2020

jameslamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jameslamb May 31, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guolinke commented Jun 11, 2020

btrotta commented Jun 11, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guolinke left a comment

Choose a reason for hiding this comment

btrotta commented Jun 19, 2020

StrikerRUS left a comment

Choose a reason for hiding this comment

mayer79 commented Jul 5, 2020

Example 1: Additive model with depth > 1

Some additive, some pairwise

Overlapping sets

btrotta commented Jul 8, 2020

github-actions bot commented Aug 24, 2023

jameslamb May 31, 2020 •

edited

Loading