fix predict with header #2643

guolinke · 2019-12-19T10:25:47Z

guolinke · 2019-12-19T11:11:36Z

@araman0 can you try this fix?

araman0 · 2019-12-20T02:15:43Z

src/io/parser.cpp

  }

-  if (label_idx < 0) {
+  if (output_label_index < 0 && label_idx >= 0) {


@guolinke
It doesn't output "Data file drop_label.tsv doesn't contain a label column."

$ head tmp.tsv label qid f1 f2 f3 f4 0 100 0.13835814976295646 0.4402769613464669 0.946869449084728 0.5359206091455541 0 100 0.01386793487399507 0.7304609364663149 0.6480159106639183 0.41084065861637076 0 100 0.36082263352323185 0.19255932163030642 0.5832995411454666 0.4231199512157455 1 100 0.24434117994194782 0.40654813181749727 0.5760772294720323 0.5689696349004272 2 100 0.2640113555206345 0.9101757126052573 0.1468593731027389 0.6202184617895963 0 101 0.6445492878877913 0.9393752647865824 0.8782811206461665 0.06155760291074741 0 101 0.0391478117208669 0.8289305808544266 0.15732825570638398 0.810192132529246 0 101 0.41636558182363037 0.06300000298506869 0.19768374800057087 0.8448096990814711 1 101 0.7191740187766613 0.945746842262128 0.8570465518925168 0.2125836005675651 $ LightGBM/lightgbm task=predict input_model=lgbm_model.txt data=tmp.tsv header=true output_result=pred_tmp.txt [LightGBM] [Info] Finished loading parameters [LightGBM] [Info] Finished initializing prediction, total used 100 iterations [LightGBM] [Info] Finished prediction $ head pred_tmp.txt -0.02745108988247158 -0.39943097926145144 0.63986234187669611 1.0541387019913457 1.6918502361811527 -1.0965348952564287 0.81806604138389494 -1.1307052283260748 -0.97471278396427918 1.3047786665317376 $ head drop_label.tsv qid f1 f2 f3 f4 100 0.13835814976295646 0.4402769613464669 0.946869449084728 0.5359206091455541 100 0.01386793487399507 0.7304609364663149 0.6480159106639183 0.41084065861637076 100 0.36082263352323185 0.19255932163030642 0.5832995411454666 0.4231199512157455 100 0.24434117994194782 0.40654813181749727 0.5760772294720323 0.5689696349004272 100 0.2640113555206345 0.9101757126052573 0.1468593731027389 0.6202184617895963 101 0.6445492878877913 0.9393752647865824 0.8782811206461665 0.06155760291074741 101 0.0391478117208669 0.8289305808544266 0.15732825570638398 0.810192132529246 101 0.41636558182363037 0.06300000298506869 0.19768374800057087 0.8448096990814711 101 0.7191740187766613 0.945746842262128 0.8570465518925168 0.2125836005675651 $ LightGBM/lightgbm task=predict input_model=lgbm_model.txt data=drop_label.tsv header=true output_result=pred_drop_label.txt [LightGBM] [Info] Finished loading parameters [LightGBM] [Info] Finished initializing prediction, total used 100 iterations [LightGBM] [Info] Finished prediction $ head pred_drop_label.txt -0.02745108988247158 -0.39943097926145144 0.63986234187669611 1.0541387019913457 1.6918502361811527 -1.0965348952564287 0.81806604138389494 -1.1307052283260748 -0.97471278396427918 1.3047786665317376

yes, this output is designed for the file without the header.
When there is no header, we need to exclude the label column, otherwise, we will read the wrong features. Provide this output is to let the user confirm whether we process his input correctly or not.

When there is a header, we don't need to exclude the label column, as we can get feature by its name.

chivee · 2019-12-20T02:08:23Z

src/application/predictor.hpp

-            feature_names_map_[i] = j;
-            break;
-          }
+        if (header_mapper.count(header_words[i]) > 0) {


never be used?

to avoid duplicated feature names

araman0 · 2019-12-20T03:45:22Z

@guolinke I've confirmed it outputs same predictions. thx!

$ LightGBM/lightgbm task=predict input_model=lgbm_model.txt data=tmp.tsv header=true output_result=pred_tmp.txt
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Info] Finished initializing prediction, total used 100 iterations
[LightGBM] [Info] Finished prediction
$ head pred_tmp.txt 
-0.67235347824716607
-0.32698080395337359
-0.28923179728330589
0.18131331367689701
1.018185089266975
-1.2743996252961152
-1.3109041864684552
-0.81249028101368681
-0.27020354133159391
0.12807112089932743
$ LightGBM/lightgbm task=predict input_model=lgbm_model.txt data=shuffle1.tsv header=true output_result=pred_shuffle1.txt
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Info] Finished initializing prediction, total used 100 iterations
[LightGBM] [Info] Finished prediction
$ head pred_shuffle1.txt 
-0.67235347824716607
-0.32698080395337359
-0.28923179728330589
0.18131331367689701
1.018185089266975
-1.2743996252961152
-1.3109041864684552
-0.81249028101368681
-0.27020354133159391
0.12807112089932743
$ LightGBM/lightgbm task=predict input_model=lgbm_model.txt data=shuffle2.tsv header=true output_result=pred_shuffle2.txt
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Info] Finished initializing prediction, total used 100 iterations
[LightGBM] [Info] Finished prediction
$ head pred_shuffle2.txt 
-0.67235347824716607
-0.32698080395337359
-0.28923179728330589
0.18131331367689701
1.018185089266975
-1.2743996252961152
-1.3109041864684552
-0.81249028101368681
-0.27020354133159391
0.12807112089932743
$ LightGBM/lightgbm task=predict input_model=lgbm_model.txt data=shuffle3.tsv header=true output_result=pred_shuffle3.txt
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Info] Finished initializing prediction, total used 100 iterations
[LightGBM] [Info] Finished prediction
$ head pred_shuffle3.txt 
-0.67235347824716607
-0.32698080395337359
-0.28923179728330589
0.18131331367689701
1.018185089266975
-1.2743996252961152
-1.3109041864684552
-0.81249028101368681
-0.27020354133159391
0.12807112089932743

$ LightGBM/lightgbm task=predict input_model=lgbm_ignore_model.txt data=tmp.tsv header=true output_result=pred_tmp_ignore.txt
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Info] Finished initializing prediction, total used 100 iterations
[LightGBM] [Info] Finished prediction
$ head pred_tmp_ignore.txt
-0.43058259683609235
-0.098525361939025474
-0.2048474171242694
0.026815875041871856
0.94411773547211675
-0.89330324800246408
-1.3645662587572376
-0.5720641564103417
-0.46146592181818258
0.069007612700339171
$ LightGBM/lightgbm task=predict input_model=lgbm_ignore_model.txt data=drop_column.tsv header=true output_result=pred_drop_column_ignore.txt
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Info] Finished initializing prediction, total used 100 iterations
[LightGBM] [Warning] Feature (f4) is missed in data file. If it is weight/query/group/ignore_column, you can ignore this warning.
[LightGBM] [Info] Finished prediction
$ head pred_drop_column_ignore.txt
-0.43058259683609235
-0.098525361939025474
-0.2048474171242694
0.026815875041871856
0.94411773547211675
-0.89330324800246408
-1.3645662587572376
-0.5720641564103417
-0.46146592181818258
0.069007612700339171
$ LightGBM/lightgbm task=predict input_model=lgbm_ignore_model.txt data=add_column.tsv header=true output_result=pred_add_column_ignore.txt
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Info] Finished initializing prediction, total used 100 iterations
[LightGBM] [Warning] Feature (f4) is missed in data file. If it is weight/query/group/ignore_column, you can ignore this warning.
[LightGBM] [Info] Finished prediction
$ head pred_add_column_ignore.txt
-0.43058259683609235
-0.098525361939025474
-0.2048474171242694
0.026815875041871856
0.94411773547211675
-0.89330324800246408
-1.3645662587572376
-0.5720641564103417
-0.46146592181818258
0.069007612700339171

fix predict with header

2f33985

guolinke requested a review from chivee as a code owner December 19, 2019 10:25

Merge remote-tracking branch 'origin/master' into predict-with-header

cda5171

avoid duplicated feature names

cee38c1

araman0 reviewed Dec 20, 2019

View reviewed changes

chivee approved these changes Dec 20, 2019

View reviewed changes

guolinke merged commit ae320e5 into master Dec 20, 2019

guolinke deleted the predict-with-header branch December 20, 2019 03:53

guolinke added the fix label Mar 1, 2020

lock bot locked as resolved and limited conversation to collaborators Mar 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix predict with header #2643

fix predict with header #2643

guolinke commented Dec 19, 2019

guolinke commented Dec 19, 2019

araman0 Dec 20, 2019

guolinke Dec 20, 2019

chivee Dec 20, 2019

guolinke Dec 20, 2019 •

edited

Loading

araman0 commented Dec 20, 2019

fix predict with header #2643

fix predict with header #2643

Conversation

guolinke commented Dec 19, 2019

guolinke commented Dec 19, 2019

araman0 Dec 20, 2019

Choose a reason for hiding this comment

guolinke Dec 20, 2019

Choose a reason for hiding this comment

chivee Dec 20, 2019

Choose a reason for hiding this comment

guolinke Dec 20, 2019 • edited Loading

Choose a reason for hiding this comment

araman0 commented Dec 20, 2019

guolinke Dec 20, 2019 •

edited

Loading