When upgrading nodes, why does kubernetes terminate pods before the new pod is in a ready state?Exposing service in KubernetesIntermittent DNS failures in Google Container EngineAny way to delay termination of an older pod, to “wait” for the new pod to be ready to serve requests?What makes a kubernetes node unhealthy?kubernetes flannel daemonset not starting (Clean Ubuntu 16 and 18)GKE lost nodes and Google Console shows different number of nodesGKE lost node tag and static ip after auto-upgradeKubernetes node breaks on restarting containerdGoogle Cloud Kuberbetes run-away systemd 100% CPU usageSetup Local Persistent Volume on physical machines with Kubernetes
70s Sci-Fi Story including/describing Conway's solitaire game "life"
The most secure way to handle someone forgetting to verify their account?
Why are flying carpets banned while flying brooms are not?
To what extent does asymmetric cryptography secure bitcoin transactions?
Extract the attribute names from a large number of Shapefiles
Discontinuous Tube visualization
What's a German word for »Sandbagger«?
Deleting a point in METAFONT
You have no, but can try for yes
Is encryption still applied if you ignore the SSL certificate warning for self-signed certs?
How should I interpret a promising preprint that was never published in a peer-reviewed journal?
Why don't humans perceive sound waves as twice the frequency they are?
Get Chord Name From a Given Set of Notes
Why is this guy handcuffed censored?
How do you name this compound using IUPAC system (including steps)?
How did Jayne know when to shoot?
Does a hash function have a Upper bound on input length?
Transistor power dissipation rating
Applying for jobs with an obvious scar
Is "repository" pronounced /rɪˈpɒzɪt(ə)ri/ or ri-ˈpä-zə-ˌtȯr-ē or /rəˈpäzəˌtôrē/?
A spacecraft is travelling at X units per hour. But relative to what exactly? Does it depend on orbit? How?
Why teach C using scanf without talking about command line arguments?
A "Replace" sort problem. Basic but haunts me
Are there foods that astronauts are explicitly never allowed to eat?
When upgrading nodes, why does kubernetes terminate pods before the new pod is in a ready state?
Exposing service in KubernetesIntermittent DNS failures in Google Container EngineAny way to delay termination of an older pod, to “wait” for the new pod to be ready to serve requests?What makes a kubernetes node unhealthy?kubernetes flannel daemonset not starting (Clean Ubuntu 16 and 18)GKE lost nodes and Google Console shows different number of nodesGKE lost node tag and static ip after auto-upgradeKubernetes node breaks on restarting containerdGoogle Cloud Kuberbetes run-away systemd 100% CPU usageSetup Local Persistent Volume on physical machines with Kubernetes
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
As you can see in the picture below, when upgrading nodes to a new version of Kubernetes, the pods on each node are recreated on a new node with a newer version of Kubernetes.
However, it seems that the old pods are being destroyed (Terminating) without waiting for the new pods to be in a Ready state, leading to downtime as the new pods are still in a ContainerCreating state.
Is there an explanation for this, or am I doing something wrong?
kubernetes google-kubernetes-engine
asked Jul 11 at 2:32
NickNick
1033 bronze badges
add a comment |
As you can see in the picture below, when upgrading nodes to a new version of Kubernetes, the pods on each node are recreated on a new node with a newer version of Kubernetes.
However, it seems that the old pods are being destroyed (Terminating) without waiting for the new pods to be in a Ready state, leading to downtime as the new pods are still in a ContainerCreating state.
Is there an explanation for this, or am I doing something wrong?
kubernetes google-kubernetes-engine
asked Jul 11 at 2:32
NickNick
1033 bronze badges
add a comment |
As you can see in the picture below, when upgrading nodes to a new version of Kubernetes, the pods on each node are recreated on a new node with a newer version of Kubernetes.
However, it seems that the old pods are being destroyed (Terminating) without waiting for the new pods to be in a Ready state, leading to downtime as the new pods are still in a ContainerCreating state.
Is there an explanation for this, or am I doing something wrong?
kubernetes google-kubernetes-engine
asked Jul 11 at 2:32
NickNick
1033 bronze badges
As you can see in the picture below, when upgrading nodes to a new version of Kubernetes, the pods on each node are recreated on a new node with a newer version of Kubernetes.
However, it seems that the old pods are being destroyed (Terminating) without waiting for the new pods to be in a Ready state, leading to downtime as the new pods are still in a ContainerCreating state.
Is there an explanation for this, or am I doing something wrong?
kubernetes google-kubernetes-engine
kubernetes google-kubernetes-engine
asked Jul 11 at 2:32
NickNick
1033 bronze badges
asked Jul 11 at 2:32
NickNick
1033 bronze badges
asked Jul 11 at 2:32
NickNick
1033 bronze badges
asked Jul 11 at 2:32
NickNick
1033 bronze badges
asked Jul 11 at 2:32
NickNick
1033 bronze badges
1033 bronze badges
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
When a node is upgraded, GKE will first drain the pods from the node before removing the deleting the virtual machine and removing from the node from the cluster. The process for terminating a pod is explained in detail here and what you should realize is that this process starts before the new pods is created, scheduled, or starts running.
If you have a deployment and want to keep a specific number of replicas up during the node upgrade process, you should configure pod disruption budgets, with which you can ensure that the node upgrade process will proceed at a slow enough rate to keep enough containers running for your service to handle traffic without any downtime.
answered Jul 11 at 5:11
Robert BaileyRobert Bailey
5093 silver badges6 bronze badges
Thanks! I wonder why Kubernetes doesn't do this by default.
– Nick
Jul 11 at 6:25
Which part? Graceful termination periods and disruption budgets are application specific settings and need to be customized for each application / system separately.
– Robert Bailey
Jul 12 at 3:18
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "2"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f974771%2fwhen-upgrading-nodes-why-does-kubernetes-terminate-pods-before-the-new-pod-is-i%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
When a node is upgraded, GKE will first drain the pods from the node before removing the deleting the virtual machine and removing from the node from the cluster. The process for terminating a pod is explained in detail here and what you should realize is that this process starts before the new pods is created, scheduled, or starts running.
If you have a deployment and want to keep a specific number of replicas up during the node upgrade process, you should configure pod disruption budgets, with which you can ensure that the node upgrade process will proceed at a slow enough rate to keep enough containers running for your service to handle traffic without any downtime.
answered Jul 11 at 5:11
Robert BaileyRobert Bailey
5093 silver badges6 bronze badges
Thanks! I wonder why Kubernetes doesn't do this by default.
– Nick
Jul 11 at 6:25
Which part? Graceful termination periods and disruption budgets are application specific settings and need to be customized for each application / system separately.
– Robert Bailey
Jul 12 at 3:18
add a comment |
When a node is upgraded, GKE will first drain the pods from the node before removing the deleting the virtual machine and removing from the node from the cluster. The process for terminating a pod is explained in detail here and what you should realize is that this process starts before the new pods is created, scheduled, or starts running.
If you have a deployment and want to keep a specific number of replicas up during the node upgrade process, you should configure pod disruption budgets, with which you can ensure that the node upgrade process will proceed at a slow enough rate to keep enough containers running for your service to handle traffic without any downtime.
answered Jul 11 at 5:11
Robert BaileyRobert Bailey
5093 silver badges6 bronze badges
Thanks! I wonder why Kubernetes doesn't do this by default.
– Nick
Jul 11 at 6:25
Which part? Graceful termination periods and disruption budgets are application specific settings and need to be customized for each application / system separately.
– Robert Bailey
Jul 12 at 3:18
add a comment |
When a node is upgraded, GKE will first drain the pods from the node before removing the deleting the virtual machine and removing from the node from the cluster. The process for terminating a pod is explained in detail here and what you should realize is that this process starts before the new pods is created, scheduled, or starts running.
If you have a deployment and want to keep a specific number of replicas up during the node upgrade process, you should configure pod disruption budgets, with which you can ensure that the node upgrade process will proceed at a slow enough rate to keep enough containers running for your service to handle traffic without any downtime.
answered Jul 11 at 5:11
Robert BaileyRobert Bailey
5093 silver badges6 bronze badges
When a node is upgraded, GKE will first drain the pods from the node before removing the deleting the virtual machine and removing from the node from the cluster. The process for terminating a pod is explained in detail here and what you should realize is that this process starts before the new pods is created, scheduled, or starts running.
If you have a deployment and want to keep a specific number of replicas up during the node upgrade process, you should configure pod disruption budgets, with which you can ensure that the node upgrade process will proceed at a slow enough rate to keep enough containers running for your service to handle traffic without any downtime.
answered Jul 11 at 5:11
Robert BaileyRobert Bailey
5093 silver badges6 bronze badges
answered Jul 11 at 5:11
Robert BaileyRobert Bailey
5093 silver badges6 bronze badges
answered Jul 11 at 5:11
Robert BaileyRobert Bailey
5093 silver badges6 bronze badges
answered Jul 11 at 5:11
Robert BaileyRobert Bailey
5093 silver badges6 bronze badges
5093 silver badges6 bronze badges
Thanks! I wonder why Kubernetes doesn't do this by default.
– Nick
Jul 11 at 6:25
Which part? Graceful termination periods and disruption budgets are application specific settings and need to be customized for each application / system separately.
– Robert Bailey
Jul 12 at 3:18
add a comment |
Thanks! I wonder why Kubernetes doesn't do this by default.
– Nick
Jul 11 at 6:25
Which part? Graceful termination periods and disruption budgets are application specific settings and need to be customized for each application / system separately.
– Robert Bailey
Jul 12 at 3:18
Thanks! I wonder why Kubernetes doesn't do this by default.
– Nick
Jul 11 at 6:25
Thanks! I wonder why Kubernetes doesn't do this by default.
– Nick
Jul 11 at 6:25
Which part? Graceful termination periods and disruption budgets are application specific settings and need to be customized for each application / system separately.
– Robert Bailey
Jul 12 at 3:18
Which part? Graceful termination periods and disruption budgets are application specific settings and need to be customized for each application / system separately.
– Robert Bailey
Jul 12 at 3:18
add a comment |
Thanks for contributing an answer to Server Fault!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f974771%2fwhen-upgrading-nodes-why-does-kubernetes-terminate-pods-before-the-new-pod-is-i%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
