How to determine the optimal threshold to achieve the highest accuracyWhy is accuracy not the best measure for assessing classification models?Classification probability thresholdIs accuracy an improper scoring rule in a binary classification setting?How to find the best input value for this simple problem?How do I deal with datasets that have many values out of range / over threshold?Threshold in precision/recall curveFinding the optimal threshold parameterWhat is F1 Optimal Threshold? How to calculate it?Do I do threshold selection for my logit model on the testing or training subset?Training threshold vs validation threshold for better prediction results?Decision rule for Bayesian variable selectionStatistically prove classification accuracy is acceptableGeneral rule uniform distributed classes
How do we explain the E major chord in this progression?
A planet illuminated by a black hole?
What are the exact meanings of roll, pitch and yaw?
How can I stop myself from micromanaging other PCs' actions?
How is the uk visa 180 calculated
Why can't my huge trees be chopped down?
What is the max number of outlets on a GFCI circuit?
Where to place an artificial gland in the human body?
Does the Intel 8086 CPU have user mode and kernel mode?
How to write a sincerely religious protagonist without preaching or affirming or judging their worldview?
Iterate over non-const variables in C++
Strange Cron Job takes up 100% of CPU Ubuntu 18 LTS Server
Commercial jet accompanied by small plane near Seattle
Why are off grid solar setups only 12, 24, 48 VDC?
Does academia have a lazy work culture?
What should I say when a company asks you why someone (a friend) who was fired left?
Is my employer paying me fairly? Going from 1099 to W2
Why are so many countries still in the Commonwealth?
Which Roman general was killed by his own soldiers for not letting them to loot a newly conquered city?
What is the lowest-speed bogey a jet fighter can intercept/escort?
What is the difference between 1/3, 1/2, and full casters?
Spin vs orbital angular momenta in QFT
Explanation for a joke about a three-legged dog that walks into a bar
How can I prevent corporations from growing their own workforce?
How to determine the optimal threshold to achieve the highest accuracy
Why is accuracy not the best measure for assessing classification models?Classification probability thresholdIs accuracy an improper scoring rule in a binary classification setting?How to find the best input value for this simple problem?How do I deal with datasets that have many values out of range / over threshold?Threshold in precision/recall curveFinding the optimal threshold parameterWhat is F1 Optimal Threshold? How to calculate it?Do I do threshold selection for my logit model on the testing or training subset?Training threshold vs validation threshold for better prediction results?Decision rule for Bayesian variable selectionStatistically prove classification accuracy is acceptableGeneral rule uniform distributed classes
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
$begingroup$
I have a list of probabilities outputted by a classifier on a balanced dataset. The metric I want to maximize is accuracy ($fracTP+TNP+N$). Is there a way to calculate the best threshold (without iterating over many threshold values an selecting the best one), given the probabilities and their true labels.
optimization threshold
asked Jul 16 at 11:51
ShakShak
183 bronze badges
$endgroup$
add a comment |
$begingroup$
I have a list of probabilities outputted by a classifier on a balanced dataset. The metric I want to maximize is accuracy ($fracTP+TNP+N$). Is there a way to calculate the best threshold (without iterating over many threshold values an selecting the best one), given the probabilities and their true labels.
optimization threshold
asked Jul 16 at 11:51
ShakShak
183 bronze badges
$endgroup$
2
$begingroup$
Do not use accuracy to evaluate a classifier: Why is accuracy not the best measure for assessing classification models? Is accuracy an improper scoring rule in a binary classification setting? Classification probability threshold. That said, it's an interesting theoretical question.
$endgroup$
– Stephan Kolassa
Jul 16 at 11:59
add a comment |
$begingroup$
I have a list of probabilities outputted by a classifier on a balanced dataset. The metric I want to maximize is accuracy ($fracTP+TNP+N$). Is there a way to calculate the best threshold (without iterating over many threshold values an selecting the best one), given the probabilities and their true labels.
optimization threshold
asked Jul 16 at 11:51
ShakShak
183 bronze badges
$endgroup$
I have a list of probabilities outputted by a classifier on a balanced dataset. The metric I want to maximize is accuracy ($fracTP+TNP+N$). Is there a way to calculate the best threshold (without iterating over many threshold values an selecting the best one), given the probabilities and their true labels.
optimization threshold
optimization threshold
asked Jul 16 at 11:51
ShakShak
183 bronze badges
asked Jul 16 at 11:51
ShakShak
183 bronze badges
asked Jul 16 at 11:51
ShakShak
183 bronze badges
asked Jul 16 at 11:51
ShakShak
183 bronze badges
asked Jul 16 at 11:51
ShakShak
183 bronze badges
183 bronze badges
2
$begingroup$
Do not use accuracy to evaluate a classifier: Why is accuracy not the best measure for assessing classification models? Is accuracy an improper scoring rule in a binary classification setting? Classification probability threshold. That said, it's an interesting theoretical question.
$endgroup$
– Stephan Kolassa
Jul 16 at 11:59
add a comment |
2
$begingroup$
Do not use accuracy to evaluate a classifier: Why is accuracy not the best measure for assessing classification models? Is accuracy an improper scoring rule in a binary classification setting? Classification probability threshold. That said, it's an interesting theoretical question.
$endgroup$
– Stephan Kolassa
Jul 16 at 11:59
2
2
$begingroup$
Do not use accuracy to evaluate a classifier: Why is accuracy not the best measure for assessing classification models? Is accuracy an improper scoring rule in a binary classification setting? Classification probability threshold. That said, it's an interesting theoretical question.
$endgroup$
– Stephan Kolassa
Jul 16 at 11:59
$begingroup$
Do not use accuracy to evaluate a classifier: Why is accuracy not the best measure for assessing classification models? Is accuracy an improper scoring rule in a binary classification setting? Classification probability threshold. That said, it's an interesting theoretical question.
$endgroup$
– Stephan Kolassa
Jul 16 at 11:59
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
I suspect that the answer is "no", i.e., that there is no such way.
Here is an illustration, where we plot the predicted probabilities against the true labels:
Since the denominator $P+N$ in the formula for accuracy does not change, what you are trying to do is to shift the horizontal red line up or down (the height being the threshold you are interested in) in order to maximize the number of "positive" dots above the line plus the number of "negative" dots below the line. Where this optimal line lies depends entirely on the shape of the two point clouds, i.e., the conditional distribution of the predicted probabilities per true label.
Your best bet is likely a bisection search.
That said, I recommend you look at
- Why is accuracy not the best measure for assessing classification models?
- Is accuracy an improper scoring rule in a binary classification setting?
- Classification probability threshold
answered Jul 16 at 12:14
Stephan KolassaStephan Kolassa
53.3k9 gold badges105 silver badges199 bronze badges
$endgroup$
1
$begingroup$
Thank you, the graphical explanation is really good.
$endgroup$
– Shak
Jul 16 at 12:25
add a comment |
$begingroup$
Agreeing to @StephanKolassa, I'll just look from an algorithmic perspective. You'll need to sort your samples with respect to produced probabilities, which is $O(nlog n)$, if you've $n$ data samples. Then, your true class labels will order like
$$0 0 1 0 0 1 ... 1 1 0 1 $$
Then, we'll put a separator $|$ at some position in this array; this'll represent your threshold. At most there are $n+1$ positions to put it. Even if you calculate the accuracy for each of these positions, you won't be worse than the sorting complexity. After getting the maximum accuracy, the threshold may just be chosen as the average of the neighboring samples.
answered Jul 16 at 12:11
gunesgunes
12.1k1 gold badge5 silver badges22 bronze badges
$endgroup$
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "65"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f417660%2fhow-to-determine-the-optimal-threshold-to-achieve-the-highest-accuracy%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
I suspect that the answer is "no", i.e., that there is no such way.
Here is an illustration, where we plot the predicted probabilities against the true labels:
Since the denominator $P+N$ in the formula for accuracy does not change, what you are trying to do is to shift the horizontal red line up or down (the height being the threshold you are interested in) in order to maximize the number of "positive" dots above the line plus the number of "negative" dots below the line. Where this optimal line lies depends entirely on the shape of the two point clouds, i.e., the conditional distribution of the predicted probabilities per true label.
Your best bet is likely a bisection search.
That said, I recommend you look at
- Why is accuracy not the best measure for assessing classification models?
- Is accuracy an improper scoring rule in a binary classification setting?
- Classification probability threshold
answered Jul 16 at 12:14
Stephan KolassaStephan Kolassa
53.3k9 gold badges105 silver badges199 bronze badges
$endgroup$
1
$begingroup$
Thank you, the graphical explanation is really good.
$endgroup$
– Shak
Jul 16 at 12:25
add a comment |
$begingroup$
I suspect that the answer is "no", i.e., that there is no such way.
Here is an illustration, where we plot the predicted probabilities against the true labels:
Since the denominator $P+N$ in the formula for accuracy does not change, what you are trying to do is to shift the horizontal red line up or down (the height being the threshold you are interested in) in order to maximize the number of "positive" dots above the line plus the number of "negative" dots below the line. Where this optimal line lies depends entirely on the shape of the two point clouds, i.e., the conditional distribution of the predicted probabilities per true label.
Your best bet is likely a bisection search.
That said, I recommend you look at
- Why is accuracy not the best measure for assessing classification models?
- Is accuracy an improper scoring rule in a binary classification setting?
- Classification probability threshold
answered Jul 16 at 12:14
Stephan KolassaStephan Kolassa
53.3k9 gold badges105 silver badges199 bronze badges
$endgroup$
1
$begingroup$
Thank you, the graphical explanation is really good.
$endgroup$
– Shak
Jul 16 at 12:25
add a comment |
$begingroup$
I suspect that the answer is "no", i.e., that there is no such way.
Here is an illustration, where we plot the predicted probabilities against the true labels:
Since the denominator $P+N$ in the formula for accuracy does not change, what you are trying to do is to shift the horizontal red line up or down (the height being the threshold you are interested in) in order to maximize the number of "positive" dots above the line plus the number of "negative" dots below the line. Where this optimal line lies depends entirely on the shape of the two point clouds, i.e., the conditional distribution of the predicted probabilities per true label.
Your best bet is likely a bisection search.
That said, I recommend you look at
- Why is accuracy not the best measure for assessing classification models?
- Is accuracy an improper scoring rule in a binary classification setting?
- Classification probability threshold
answered Jul 16 at 12:14
Stephan KolassaStephan Kolassa
53.3k9 gold badges105 silver badges199 bronze badges
$endgroup$
I suspect that the answer is "no", i.e., that there is no such way.
Here is an illustration, where we plot the predicted probabilities against the true labels:
Since the denominator $P+N$ in the formula for accuracy does not change, what you are trying to do is to shift the horizontal red line up or down (the height being the threshold you are interested in) in order to maximize the number of "positive" dots above the line plus the number of "negative" dots below the line. Where this optimal line lies depends entirely on the shape of the two point clouds, i.e., the conditional distribution of the predicted probabilities per true label.
Your best bet is likely a bisection search.
That said, I recommend you look at
- Why is accuracy not the best measure for assessing classification models?
- Is accuracy an improper scoring rule in a binary classification setting?
- Classification probability threshold
answered Jul 16 at 12:14
Stephan KolassaStephan Kolassa
53.3k9 gold badges105 silver badges199 bronze badges
answered Jul 16 at 12:14
Stephan KolassaStephan Kolassa
53.3k9 gold badges105 silver badges199 bronze badges
answered Jul 16 at 12:14
Stephan KolassaStephan Kolassa
53.3k9 gold badges105 silver badges199 bronze badges
answered Jul 16 at 12:14
Stephan KolassaStephan Kolassa
53.3k9 gold badges105 silver badges199 bronze badges
53.3k9 gold badges105 silver badges199 bronze badges
1
$begingroup$
Thank you, the graphical explanation is really good.
$endgroup$
– Shak
Jul 16 at 12:25
add a comment |
1
$begingroup$
Thank you, the graphical explanation is really good.
$endgroup$
– Shak
Jul 16 at 12:25
1
1
$begingroup$
Thank you, the graphical explanation is really good.
$endgroup$
– Shak
Jul 16 at 12:25
$begingroup$
Thank you, the graphical explanation is really good.
$endgroup$
– Shak
Jul 16 at 12:25
add a comment |
$begingroup$
Agreeing to @StephanKolassa, I'll just look from an algorithmic perspective. You'll need to sort your samples with respect to produced probabilities, which is $O(nlog n)$, if you've $n$ data samples. Then, your true class labels will order like
$$0 0 1 0 0 1 ... 1 1 0 1 $$
Then, we'll put a separator $|$ at some position in this array; this'll represent your threshold. At most there are $n+1$ positions to put it. Even if you calculate the accuracy for each of these positions, you won't be worse than the sorting complexity. After getting the maximum accuracy, the threshold may just be chosen as the average of the neighboring samples.
answered Jul 16 at 12:11
gunesgunes
12.1k1 gold badge5 silver badges22 bronze badges
$endgroup$
add a comment |
$begingroup$
Agreeing to @StephanKolassa, I'll just look from an algorithmic perspective. You'll need to sort your samples with respect to produced probabilities, which is $O(nlog n)$, if you've $n$ data samples. Then, your true class labels will order like
$$0 0 1 0 0 1 ... 1 1 0 1 $$
Then, we'll put a separator $|$ at some position in this array; this'll represent your threshold. At most there are $n+1$ positions to put it. Even if you calculate the accuracy for each of these positions, you won't be worse than the sorting complexity. After getting the maximum accuracy, the threshold may just be chosen as the average of the neighboring samples.
answered Jul 16 at 12:11
gunesgunes
12.1k1 gold badge5 silver badges22 bronze badges
$endgroup$
add a comment |
$begingroup$
Agreeing to @StephanKolassa, I'll just look from an algorithmic perspective. You'll need to sort your samples with respect to produced probabilities, which is $O(nlog n)$, if you've $n$ data samples. Then, your true class labels will order like
$$0 0 1 0 0 1 ... 1 1 0 1 $$
Then, we'll put a separator $|$ at some position in this array; this'll represent your threshold. At most there are $n+1$ positions to put it. Even if you calculate the accuracy for each of these positions, you won't be worse than the sorting complexity. After getting the maximum accuracy, the threshold may just be chosen as the average of the neighboring samples.
answered Jul 16 at 12:11
gunesgunes
12.1k1 gold badge5 silver badges22 bronze badges
$endgroup$
Agreeing to @StephanKolassa, I'll just look from an algorithmic perspective. You'll need to sort your samples with respect to produced probabilities, which is $O(nlog n)$, if you've $n$ data samples. Then, your true class labels will order like
$$0 0 1 0 0 1 ... 1 1 0 1 $$
Then, we'll put a separator $|$ at some position in this array; this'll represent your threshold. At most there are $n+1$ positions to put it. Even if you calculate the accuracy for each of these positions, you won't be worse than the sorting complexity. After getting the maximum accuracy, the threshold may just be chosen as the average of the neighboring samples.
answered Jul 16 at 12:11
gunesgunes
12.1k1 gold badge5 silver badges22 bronze badges
answered Jul 16 at 12:11
gunesgunes
12.1k1 gold badge5 silver badges22 bronze badges
answered Jul 16 at 12:11
gunesgunes
12.1k1 gold badge5 silver badges22 bronze badges
answered Jul 16 at 12:11
gunesgunes
12.1k1 gold badge5 silver badges22 bronze badges
12.1k1 gold badge5 silver badges22 bronze badges
add a comment |
add a comment |
Thanks for contributing an answer to Cross Validated!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f417660%2fhow-to-determine-the-optimal-threshold-to-achieve-the-highest-accuracy%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
$begingroup$
Do not use accuracy to evaluate a classifier: Why is accuracy not the best measure for assessing classification models? Is accuracy an improper scoring rule in a binary classification setting? Classification probability threshold. That said, it's an interesting theoretical question.
$endgroup$
– Stephan Kolassa
Jul 16 at 11:59