gblinear non-determinism

[honzasterba]

The last calls of the code below produce very different results for each call. This is not reproducible when using gbtree booster.

import xgboost as xgb
import pandas as pd

# read in data
train = pd.read_csv('train.csv')
test = pd.read_csv('test.csv')
dtrain = xgb.DMatrix(train, label=train["ID"])
dtest = xgb.DMatrix(test, label=test["ID"])
param = {
    'booster': 'gblinear',
    'max_depth': 5, 
    'response_column':'AGE',
    'objective': "reg:gamma",
    'verbosity': 3,
    'seed': 1
}
print(xgb.train(param, dtrain, 1).predict(dtest))
print(xgb.train(param, dtrain, 1).predict(dtest))
print(xgb.train(param, dtrain, 1).predict(dtest))

[honzasterba]

data.zip

[trivialfis]

Could you try providing a seed parameter?

[honzasterba]

setting seed still ends up vastly different results for each run (i am updating the sample code)

[RAMitchell]

gblinear uses the shotgun algorithm by default which is strongly nondeterministic. Try setting "updater":"coord_descent".

[honzasterba]

thanks that makes it deterministic, in case you consider gblinear with seed set indeterminism not a bug we cal close this issue

[trivialfis]

Closing. I forgot that the shotgun is a thread sanitizer killer.

[michalkurka]

@trivialfis imho this should not be closed without at least some fix - people expect the model will be reproducible if they set a seed - an improvement of documentation and perhaps a warning when gblinear is run with user defined seed seem appropriate in this case

[trivialfis]

@michalkurka Good point.

[RAMitchell]

This is documented under xgboost parameters. @michalkurka where would you expect to see this?

Choice of algorithm to fit linear model

shotgun: Parallel coordinate descent algorithm based on shotgun algorithm. Uses ‘hogwild’ parallelism and therefore produces a nondeterministic solution on each run.

coord_descent: Ordinary coordinate descent algorithm. Also multithreaded but still produces a deterministic solution.```

[michalkurka]

@RAMitchell This is something I would ideally want to see in the python documentation, eg. here: https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBClassifier

I would certainly not expect the default is non-deterministic, IMHO a warning on this page would make sense.