[R-package] MAPE metric not being minimized in R

[parayamelo]

Hi,

I am running Lightgbm in R (R version 4.0.0, Lightgbm version 2.3.2), with metric = "mape". I see that when I run cross validation (lgb.cv), the best iteration is always the one with the maximum value of mape.

For example, I run cross validation with nrounds = 200. Last value:

[200]: valid's mape:0.47669+0.033924

best.iter <- lgb.cv$best_iter

<lgb.CVBooster>
Public:
best_iter: 200
best_score: 0.476690009429843
boosters: list
initialize: function (x)
record_evals: list
reset_parameter: function (new_params)

However, if I see other values,

[151]: valid's mape:0.436838+0.0311321

In this case, the best iteration should be 151 and not 200.

I do not see this behavior in python.

Am I doing something wrong?

Thank you.

[jameslamb]

Thanks for the report @parayamelo, and sorry for the issue !

I will try to reproduce this behavior on a sample dataset. If I can't reproduce it, I'll need some more information from you.

[parayamelo]

Thank you @jameslamb! Yes, please, let me know if I can help you with something else.

[StrikerRUS]

@jameslamb We should make this regex to not catch MAPE

LightGBM/R-package/R/lgb.Booster.R

Line 599 in 1f3e72c

private$higher_better_inner_eval <- grepl("^ndcg|^map|^auc", names)

[jameslamb]

@jameslamb We should make this regex to not catch MAPE

LightGBM/R-package/R/lgb.Booster.R

Line 599 in 1f3e72c

private$higher_better_inner_eval <- grepl("^ndcg|^map|^auc", names)

😱 thank you for pointing that out. I won't have time to investigate for a few more hours, that will help!

[jameslamb]

@parayamelo , @StrikerRUS was right, the issue was #3099 (comment)

mape conflicted with map (mean average precision, used in ranking tasks) and gets classified as a maximization problem.

I think I've fixed this in #3101. If you'd like to test, you can build from that pull request:

git clone https://github.com/jameslamb/LightGBM.git
cd LightGBM
git fetch origin fix/r-metrics
git checkout fix/r-metrics

Rscript build_r.R

Thank you very much for reporting this issue and for using LightGBM!

[parayamelo]

Thanks @jameslamb for the fix and @StrikerRUS for pointing out where the error was! I will test it and come back to you.

[parayamelo]

@jameslamb I tested the solution on 2 different datasets, in one that I trusted it worked, in the other one it worked, but I think there is a mistake on how I construct the dataset, because it finds the lowest mape at the first iteration. I will use this branch for the moment, will this be fix for future releases? Thank you once again!

[jameslamb]

Glad it's working!

I will use this branch for the moment, will this be fix for future releases

Yes absolutely. #3101 will be merged in the next few days. It just needs to go through our normal review process.

[parayamelo]

Great! Thanks again, I will close the issue then.

[jameslamb]

@parayamelo thanks! I am actually going to leave this open. Until that pull request is merged, other users might experience the same problem and then open a new issue if they don't see this one.

[jameslamb]

@parayamelo this fix has been merged to master, thanks again for using LightGBM and for reporting this issue!

[parayamelo]

@jameslamb Thank you!

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.