LightGBM results differently depending on the order of the dataTune hyperparameters for cost-sensitive classificationR - Error in KNN - Test and training differLearning Algorithm that decide which model gives better results for each testing instanceSignificant overfitting with CVwhy is mse training drastically different from the begining of each training with Encoder-DecoderClassification using Orange 3What to report in the build model, asses model and evaluate results steps of CRISP-DM?Using GridSearchCV and a Random Forest Regressor with the same parameters gives different resultsHyper-parameter tuning when you don't have an access to the test dataUsing the validation data
Hostile Divisor Numbers
Should I simplify my writing in a foreign country?
Python 3 - simple temperature program
Are sleeping system R-ratings additive?
Handling Null values (and equivalents) routinely in Python
How does summation index shifting work?
How do LIGO and VIRGO know that a gravitational wave has its origin in a neutron star or a black hole?
Formatting Datetime.now()
Can my 2 children, aged 10 and 12, who are US citizens, travel to the USA on expired American passports?
Why would a military not separate its forces into different branches?
Adding command shortcuts to /bin
It is as simple as ABC
What do "Sech" and "Vich" mean in this sentence?
Should homeowners insurance cover the cost of the home?
Is 'contemporary' ambiguous and if so is there a better word?
Nested loops to process groups of pictures
SOQL query WHERE filter by specific months
How to pass hash as password to ssh server
What do I do if my advisor made a mistake?
Gerrymandering Puzzle - Rig the Election
What is the closest airport to the center of the city it serves?
Will 700 more planes a day fly because of the Heathrow expansion?
Where are the "shires" in the UK?
What is a common way to tell if an academic is "above average," or outstanding in their field? Is their h-index (Hirsh index) one of them?
LightGBM results differently depending on the order of the data
Tune hyperparameters for cost-sensitive classificationR - Error in KNN - Test and training differLearning Algorithm that decide which model gives better results for each testing instanceSignificant overfitting with CVwhy is mse training drastically different from the begining of each training with Encoder-DecoderClassification using Orange 3What to report in the build model, asses model and evaluate results steps of CRISP-DM?Using GridSearchCV and a Random Forest Regressor with the same parameters gives different resultsHyper-parameter tuning when you don't have an access to the test dataUsing the validation data
$begingroup$
I have two datasets A and B which are exactly the same in terms of the number of columns, name of columns and the values. The only difference is the order of those columns. I then train the LightGBM model on each of the two datasets with the following steps
- Divide each dataset into training and testing (use the same random seed and ratio for both A and B)
- Leave the hyperparameters as pretty much default
- Set a random state as a fixed number (for reproduction)
- Tune the learning_rate using a Grid Search
- Train a LightGBM model on the training set and test it on the
testing set - Learning rate with the best performance on the testing set will be
chosen
The output models on the two datasets are very different, which makes me thinks that the order of columns does affect the performance of the model training using LightGBM.
Do you know why this is the case?
machine-learning classification
New contributor
$endgroup$
add a comment |
$begingroup$
I have two datasets A and B which are exactly the same in terms of the number of columns, name of columns and the values. The only difference is the order of those columns. I then train the LightGBM model on each of the two datasets with the following steps
- Divide each dataset into training and testing (use the same random seed and ratio for both A and B)
- Leave the hyperparameters as pretty much default
- Set a random state as a fixed number (for reproduction)
- Tune the learning_rate using a Grid Search
- Train a LightGBM model on the training set and test it on the
testing set - Learning rate with the best performance on the testing set will be
chosen
The output models on the two datasets are very different, which makes me thinks that the order of columns does affect the performance of the model training using LightGBM.
Do you know why this is the case?
machine-learning classification
New contributor
$endgroup$
add a comment |
$begingroup$
I have two datasets A and B which are exactly the same in terms of the number of columns, name of columns and the values. The only difference is the order of those columns. I then train the LightGBM model on each of the two datasets with the following steps
- Divide each dataset into training and testing (use the same random seed and ratio for both A and B)
- Leave the hyperparameters as pretty much default
- Set a random state as a fixed number (for reproduction)
- Tune the learning_rate using a Grid Search
- Train a LightGBM model on the training set and test it on the
testing set - Learning rate with the best performance on the testing set will be
chosen
The output models on the two datasets are very different, which makes me thinks that the order of columns does affect the performance of the model training using LightGBM.
Do you know why this is the case?
machine-learning classification
New contributor
$endgroup$
I have two datasets A and B which are exactly the same in terms of the number of columns, name of columns and the values. The only difference is the order of those columns. I then train the LightGBM model on each of the two datasets with the following steps
- Divide each dataset into training and testing (use the same random seed and ratio for both A and B)
- Leave the hyperparameters as pretty much default
- Set a random state as a fixed number (for reproduction)
- Tune the learning_rate using a Grid Search
- Train a LightGBM model on the training set and test it on the
testing set - Learning rate with the best performance on the testing set will be
chosen
The output models on the two datasets are very different, which makes me thinks that the order of columns does affect the performance of the model training using LightGBM.
Do you know why this is the case?
machine-learning classification
machine-learning classification
New contributor
New contributor
edited 2 days ago
Duy Bui
New contributor
asked Apr 30 at 17:09
Duy BuiDuy Bui
1313
1313
New contributor
New contributor
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
A possible explanation is this:
When the order of the columns differ, there is a little difference in the procedure.
What LightGBM, XGBoost, CatBoost, amongst other do is to select different columns from the features in your dataset in every step in the training.
The selections of these columns is done randomly: Let's say your dataset has 20 columns. The root node selects the features 1st, 3rd and 18th, on both datasets the 1st, 3rd and 18th features are different in both possible datasets. This is repeatedly done and in every step there is randomness affecting your ultimate result.
$endgroup$
$begingroup$
How can we control that randomness when the algorithm selects a subset of features to build a decision tree? That was also my only thought to answer this situation. Moreover, I guess if we always select all features per tree, the algorithm will use Gini (or something similar) to calculate the feature importance at each step, which won't create an randomness.
$endgroup$
– Duy Bui
2 days ago
$begingroup$
lightgbm
allows the user to set the random seeds used for row and column sampling.
$endgroup$
– bradS
2 days ago
1
$begingroup$
@bradS: I didn't set the seed as a hyperparameter in the LightGBM but I checked again and seeds should be set as a fixed number by default. That means it should have the same result, which is not the case here. lightgbm.readthedocs.io/en/latest/Parameters.html
$endgroup$
– Duy Bui
2 days ago
add a comment |
$begingroup$
While the ordering of data is inconsequential in theory, it is important in practice. Considering you took steps to ensure reproducibility, Different ordering of data will alter your train-test split logic(unless you know for certain that the train sets and test sets in both cases are exactly the same). Though you don’t specify how you split the data it is highly possible that a certain assortment of data points makes the machine more robust to outliers and therefore offering better model performance.
In the case that the train and test data is the same in both cases, you’d likely have to see if there is a seed/reproducibility measure (in any part of your code) that you have not taken.
New contributor
$endgroup$
$begingroup$
Sorry, I forgot to mention that. Will update my query. Train and test are exactly the same because I split them using the same random seed.
$endgroup$
– Duy Bui
2 days ago
$begingroup$
@DuyBui a few suggestions to try: 1) if you are using Gpu set gpu_use_dp to true From: github.com/Microsoft/LightGBM/pull/560#issuecomment-304561654 2) set num_threads to a fixed number From: github.com/Microsoft/LightGBM/issues/632;
$endgroup$
– gbdata
yesterday
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Duy Bui is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f51188%2flightgbm-results-differently-depending-on-the-order-of-the-data%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
A possible explanation is this:
When the order of the columns differ, there is a little difference in the procedure.
What LightGBM, XGBoost, CatBoost, amongst other do is to select different columns from the features in your dataset in every step in the training.
The selections of these columns is done randomly: Let's say your dataset has 20 columns. The root node selects the features 1st, 3rd and 18th, on both datasets the 1st, 3rd and 18th features are different in both possible datasets. This is repeatedly done and in every step there is randomness affecting your ultimate result.
$endgroup$
$begingroup$
How can we control that randomness when the algorithm selects a subset of features to build a decision tree? That was also my only thought to answer this situation. Moreover, I guess if we always select all features per tree, the algorithm will use Gini (or something similar) to calculate the feature importance at each step, which won't create an randomness.
$endgroup$
– Duy Bui
2 days ago
$begingroup$
lightgbm
allows the user to set the random seeds used for row and column sampling.
$endgroup$
– bradS
2 days ago
1
$begingroup$
@bradS: I didn't set the seed as a hyperparameter in the LightGBM but I checked again and seeds should be set as a fixed number by default. That means it should have the same result, which is not the case here. lightgbm.readthedocs.io/en/latest/Parameters.html
$endgroup$
– Duy Bui
2 days ago
add a comment |
$begingroup$
A possible explanation is this:
When the order of the columns differ, there is a little difference in the procedure.
What LightGBM, XGBoost, CatBoost, amongst other do is to select different columns from the features in your dataset in every step in the training.
The selections of these columns is done randomly: Let's say your dataset has 20 columns. The root node selects the features 1st, 3rd and 18th, on both datasets the 1st, 3rd and 18th features are different in both possible datasets. This is repeatedly done and in every step there is randomness affecting your ultimate result.
$endgroup$
$begingroup$
How can we control that randomness when the algorithm selects a subset of features to build a decision tree? That was also my only thought to answer this situation. Moreover, I guess if we always select all features per tree, the algorithm will use Gini (or something similar) to calculate the feature importance at each step, which won't create an randomness.
$endgroup$
– Duy Bui
2 days ago
$begingroup$
lightgbm
allows the user to set the random seeds used for row and column sampling.
$endgroup$
– bradS
2 days ago
1
$begingroup$
@bradS: I didn't set the seed as a hyperparameter in the LightGBM but I checked again and seeds should be set as a fixed number by default. That means it should have the same result, which is not the case here. lightgbm.readthedocs.io/en/latest/Parameters.html
$endgroup$
– Duy Bui
2 days ago
add a comment |
$begingroup$
A possible explanation is this:
When the order of the columns differ, there is a little difference in the procedure.
What LightGBM, XGBoost, CatBoost, amongst other do is to select different columns from the features in your dataset in every step in the training.
The selections of these columns is done randomly: Let's say your dataset has 20 columns. The root node selects the features 1st, 3rd and 18th, on both datasets the 1st, 3rd and 18th features are different in both possible datasets. This is repeatedly done and in every step there is randomness affecting your ultimate result.
$endgroup$
A possible explanation is this:
When the order of the columns differ, there is a little difference in the procedure.
What LightGBM, XGBoost, CatBoost, amongst other do is to select different columns from the features in your dataset in every step in the training.
The selections of these columns is done randomly: Let's say your dataset has 20 columns. The root node selects the features 1st, 3rd and 18th, on both datasets the 1st, 3rd and 18th features are different in both possible datasets. This is repeatedly done and in every step there is randomness affecting your ultimate result.
answered Apr 30 at 19:30
Juan Esteban de la CalleJuan Esteban de la Calle
1,10324
1,10324
$begingroup$
How can we control that randomness when the algorithm selects a subset of features to build a decision tree? That was also my only thought to answer this situation. Moreover, I guess if we always select all features per tree, the algorithm will use Gini (or something similar) to calculate the feature importance at each step, which won't create an randomness.
$endgroup$
– Duy Bui
2 days ago
$begingroup$
lightgbm
allows the user to set the random seeds used for row and column sampling.
$endgroup$
– bradS
2 days ago
1
$begingroup$
@bradS: I didn't set the seed as a hyperparameter in the LightGBM but I checked again and seeds should be set as a fixed number by default. That means it should have the same result, which is not the case here. lightgbm.readthedocs.io/en/latest/Parameters.html
$endgroup$
– Duy Bui
2 days ago
add a comment |
$begingroup$
How can we control that randomness when the algorithm selects a subset of features to build a decision tree? That was also my only thought to answer this situation. Moreover, I guess if we always select all features per tree, the algorithm will use Gini (or something similar) to calculate the feature importance at each step, which won't create an randomness.
$endgroup$
– Duy Bui
2 days ago
$begingroup$
lightgbm
allows the user to set the random seeds used for row and column sampling.
$endgroup$
– bradS
2 days ago
1
$begingroup$
@bradS: I didn't set the seed as a hyperparameter in the LightGBM but I checked again and seeds should be set as a fixed number by default. That means it should have the same result, which is not the case here. lightgbm.readthedocs.io/en/latest/Parameters.html
$endgroup$
– Duy Bui
2 days ago
$begingroup$
How can we control that randomness when the algorithm selects a subset of features to build a decision tree? That was also my only thought to answer this situation. Moreover, I guess if we always select all features per tree, the algorithm will use Gini (or something similar) to calculate the feature importance at each step, which won't create an randomness.
$endgroup$
– Duy Bui
2 days ago
$begingroup$
How can we control that randomness when the algorithm selects a subset of features to build a decision tree? That was also my only thought to answer this situation. Moreover, I guess if we always select all features per tree, the algorithm will use Gini (or something similar) to calculate the feature importance at each step, which won't create an randomness.
$endgroup$
– Duy Bui
2 days ago
$begingroup$
lightgbm
allows the user to set the random seeds used for row and column sampling.$endgroup$
– bradS
2 days ago
$begingroup$
lightgbm
allows the user to set the random seeds used for row and column sampling.$endgroup$
– bradS
2 days ago
1
1
$begingroup$
@bradS: I didn't set the seed as a hyperparameter in the LightGBM but I checked again and seeds should be set as a fixed number by default. That means it should have the same result, which is not the case here. lightgbm.readthedocs.io/en/latest/Parameters.html
$endgroup$
– Duy Bui
2 days ago
$begingroup$
@bradS: I didn't set the seed as a hyperparameter in the LightGBM but I checked again and seeds should be set as a fixed number by default. That means it should have the same result, which is not the case here. lightgbm.readthedocs.io/en/latest/Parameters.html
$endgroup$
– Duy Bui
2 days ago
add a comment |
$begingroup$
While the ordering of data is inconsequential in theory, it is important in practice. Considering you took steps to ensure reproducibility, Different ordering of data will alter your train-test split logic(unless you know for certain that the train sets and test sets in both cases are exactly the same). Though you don’t specify how you split the data it is highly possible that a certain assortment of data points makes the machine more robust to outliers and therefore offering better model performance.
In the case that the train and test data is the same in both cases, you’d likely have to see if there is a seed/reproducibility measure (in any part of your code) that you have not taken.
New contributor
$endgroup$
$begingroup$
Sorry, I forgot to mention that. Will update my query. Train and test are exactly the same because I split them using the same random seed.
$endgroup$
– Duy Bui
2 days ago
$begingroup$
@DuyBui a few suggestions to try: 1) if you are using Gpu set gpu_use_dp to true From: github.com/Microsoft/LightGBM/pull/560#issuecomment-304561654 2) set num_threads to a fixed number From: github.com/Microsoft/LightGBM/issues/632;
$endgroup$
– gbdata
yesterday
add a comment |
$begingroup$
While the ordering of data is inconsequential in theory, it is important in practice. Considering you took steps to ensure reproducibility, Different ordering of data will alter your train-test split logic(unless you know for certain that the train sets and test sets in both cases are exactly the same). Though you don’t specify how you split the data it is highly possible that a certain assortment of data points makes the machine more robust to outliers and therefore offering better model performance.
In the case that the train and test data is the same in both cases, you’d likely have to see if there is a seed/reproducibility measure (in any part of your code) that you have not taken.
New contributor
$endgroup$
$begingroup$
Sorry, I forgot to mention that. Will update my query. Train and test are exactly the same because I split them using the same random seed.
$endgroup$
– Duy Bui
2 days ago
$begingroup$
@DuyBui a few suggestions to try: 1) if you are using Gpu set gpu_use_dp to true From: github.com/Microsoft/LightGBM/pull/560#issuecomment-304561654 2) set num_threads to a fixed number From: github.com/Microsoft/LightGBM/issues/632;
$endgroup$
– gbdata
yesterday
add a comment |
$begingroup$
While the ordering of data is inconsequential in theory, it is important in practice. Considering you took steps to ensure reproducibility, Different ordering of data will alter your train-test split logic(unless you know for certain that the train sets and test sets in both cases are exactly the same). Though you don’t specify how you split the data it is highly possible that a certain assortment of data points makes the machine more robust to outliers and therefore offering better model performance.
In the case that the train and test data is the same in both cases, you’d likely have to see if there is a seed/reproducibility measure (in any part of your code) that you have not taken.
New contributor
$endgroup$
While the ordering of data is inconsequential in theory, it is important in practice. Considering you took steps to ensure reproducibility, Different ordering of data will alter your train-test split logic(unless you know for certain that the train sets and test sets in both cases are exactly the same). Though you don’t specify how you split the data it is highly possible that a certain assortment of data points makes the machine more robust to outliers and therefore offering better model performance.
In the case that the train and test data is the same in both cases, you’d likely have to see if there is a seed/reproducibility measure (in any part of your code) that you have not taken.
New contributor
New contributor
answered Apr 30 at 17:47
gbdatagbdata
335
335
New contributor
New contributor
$begingroup$
Sorry, I forgot to mention that. Will update my query. Train and test are exactly the same because I split them using the same random seed.
$endgroup$
– Duy Bui
2 days ago
$begingroup$
@DuyBui a few suggestions to try: 1) if you are using Gpu set gpu_use_dp to true From: github.com/Microsoft/LightGBM/pull/560#issuecomment-304561654 2) set num_threads to a fixed number From: github.com/Microsoft/LightGBM/issues/632;
$endgroup$
– gbdata
yesterday
add a comment |
$begingroup$
Sorry, I forgot to mention that. Will update my query. Train and test are exactly the same because I split them using the same random seed.
$endgroup$
– Duy Bui
2 days ago
$begingroup$
@DuyBui a few suggestions to try: 1) if you are using Gpu set gpu_use_dp to true From: github.com/Microsoft/LightGBM/pull/560#issuecomment-304561654 2) set num_threads to a fixed number From: github.com/Microsoft/LightGBM/issues/632;
$endgroup$
– gbdata
yesterday
$begingroup$
Sorry, I forgot to mention that. Will update my query. Train and test are exactly the same because I split them using the same random seed.
$endgroup$
– Duy Bui
2 days ago
$begingroup$
Sorry, I forgot to mention that. Will update my query. Train and test are exactly the same because I split them using the same random seed.
$endgroup$
– Duy Bui
2 days ago
$begingroup$
@DuyBui a few suggestions to try: 1) if you are using Gpu set gpu_use_dp to true From: github.com/Microsoft/LightGBM/pull/560#issuecomment-304561654 2) set num_threads to a fixed number From: github.com/Microsoft/LightGBM/issues/632;
$endgroup$
– gbdata
yesterday
$begingroup$
@DuyBui a few suggestions to try: 1) if you are using Gpu set gpu_use_dp to true From: github.com/Microsoft/LightGBM/pull/560#issuecomment-304561654 2) set num_threads to a fixed number From: github.com/Microsoft/LightGBM/issues/632;
$endgroup$
– gbdata
yesterday
add a comment |
Duy Bui is a new contributor. Be nice, and check out our Code of Conduct.
Duy Bui is a new contributor. Be nice, and check out our Code of Conduct.
Duy Bui is a new contributor. Be nice, and check out our Code of Conduct.
Duy Bui is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f51188%2flightgbm-results-differently-depending-on-the-order-of-the-data%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown