/sig apps

@annajung @JamesLaverack james, you mentioned in the sig-apps slack channel that this enhancement is at risk, can you clarify why? it meets the criteria.

@ahg-g Just to follow up here too, we discussed in Slack and this was due to a delay in reviewing. We've now marked this as "Tracked" on the enhancements spreadsheet for 1.21.

Thank you for getting back to us. :)

Hi @ahg-g,

Since your Enhancement is scheduled to be in 1.21, please keep in mind the important upcoming dates:

• Tuesday, March 9th: Week 9 — Code Freeze
• Tuesday, March 16th: Week 10 — Docs Placeholder PR deadline
  • If this enhancement requires new docs or modification to existing docs, please follow the steps in the Open a placeholder PR doc to open a PR against k/website repo.

As a reminder, please link all of your k/k PR(s) and k/website PR(s) to this issue so we can track them.

Thanks!

Hi @ahg-g,

Since your Enhancement is scheduled to be in 1.21, please keep in mind the important upcoming dates:

• Tuesday, March 9th: Week 9 — Code Freeze

• Tuesday, March 16th: Week 10 — Docs Placeholder PR deadline

  • If this enhancement requires new docs or modification to existing docs, please follow the steps in the Open a placeholder PR doc to open a PR against k/website repo.

As a reminder, please link all of your k/k PR(s) and k/website PR(s) to this issue so we can track them.

Thanks!

done.

Hi @ahg-g

Enhancements team is currently tracking the following PRs

• Implements pod deletion cost kubernetes#99163

As this PR is merged, can we mark this enhancement complete for code freeze or do you have other PR(s) that are being worked on as part of the release?

Hi @JamesLaverack , yes the k/k code is merged, docs PR still open though.

/stage beta

/milestone v1.22

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

/remove-lifecycle rotten

Hey all watching! After thinking more about how we can make pod-deletion-cost GA, I believe I have an idea that will address most of the annotation-related concerns of the current implementation (while still maintaining backward compatibility with annotations, if they are present).

I still need to write up a full proposal and KEP, but my initial thoughts can be found at:

• PROPOSAL configurable down-scaling behaviour in ReplicaSets & Deployments kubernetes#107598 (comment).

The gist of the idea is that we can make pod-deletion-cost a more transient value (rather than only storing it in annotations), by extending the /apis/apps/v1/namespaces/{namespace}/deployments/{name}/scale API so when a caller sends a PATCH which reduces replicas, they can include the pod-deletion-cost of one-or-more Pods, these costs will only affect the current down-scale (unlike the annotations, which must be manually cleared after scaling to remove their effect).

Many thanks for all your works and reflections on this subject, I'm strongly interest in the capacity to choose pods to be evicted during scale in and I try to follow up corresponding discussions and feature developments or proposals.

I searched for a long time how to achieve this correctly, I was happy with the PodDeletionCost but now I am a little disappointed as it seems that it will stay a beta (please do not remove this feature until an equivalent one is released).
To give my two cents, I will share my understanding of the issue and maybe, I hope, help to solve it in a simple and globally compatible manner (may be I should post elsewhere, I'm not familiar with your processes).

My need (which is maybe different of yours) is to selectively evict or replace terminated pods to keep a dynamic number of fresh pod replicas without terminating potentially running pods (I mean pods running applications currently processing something).
It is more or less a pool of pods with a minimum and maximum replicas, a current number of replica varying in function of external demands, and the rule to forbid to terminate a pod with activity inside.

I may be wrong, but I think the root cause of the problem is the incompatibility between the automatic pod restart and the scale-in features.
If the ReplicaSet automatically restarts terminated pods then it gives no chance to the application itself to indicate which pod to be evicted during scale in (I mean without using the API).

Without PodDeletionCost, one known workaround is to:

• stop or delete the ReplicaSet
• delete selected pods
• decrease replica count accordingly
• start or recreate the ReplicaSet.

For me this workaround speaks in favor the incompatibility between ReplicaSet and scale-in features to select pods to be evicted : currently that cannot work when mixed together.

Also I think one should avoid any controller to terminate a pod, it should be the application inside the pod that terminates, implying the its pod to terminate, then a controller could evict only already terminated pod.

Here is my proposal :

• add an option to ReplicaSet or Deployment etc to not restart terminated pods (succeeded and/or failed).
  Currently the restart policy can only be Always.
• during scale-in prioritize terminated pods to be evicted (maybe that's already the case ?)

With these behaviors, scale-in will select pods to be evicted based on the inside pod applications termination status (here Succeeded or Failed) instead of external indicators.
If a custom controller is used to maintain a dynamic number of replica it will be able to remove or replace terminated pods just by decreasing the replica count or deleting them.

If this proposal is acceptable and can work it is maybe achievable with a minimal coding effort.

What do you think ?

Maybe my need is different because I need to automatically replace or delete terminated pods.
I have been able to select pods to remove from the replicaset by not terminating pods but setting the pod-delection-cost annotation instead, then a custom controller decrease the replica or delete pods accordingly.
Like evoked in PROPOSAL configurable down-scaling behaviour in ReplicaSets & Deployments something like a pod deletion cost probe would be better than the annotation to let the application indicates by itself that it must be prioritized for deletion.

I think there are two cases to distinguish during scale-in: the capacity to remove terminated pods from replicaset (without replacing them, which imply a ReplicaSet restartPolicy different than Always), and the capacity to remove running pods (using the probe).

/milestone clear

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@ahg-g I'm not in love with annotations as APIs. Do we REALLY think this is the best answer?

@ahg-g I'm not in love with annotations as APIs. Do we REALLY think this is the best answer?

I think we have a reasonable counter proposal in kubernetes/kubernetes#107598 (comment); can we hold this in its current beta state until that proposal makes progress?

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

/remove-lifecycle stale

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

/remove-lifecycle stale

would the following be an acceptable design pattern?

• Pod gets a sidecar with permission to set its own PodDeletionCost
• sidecar polls metric-server for current CPU usage of its own Pod
• sidecar sets its own PodDeletionCost to the number of millicores returned by metrics-server
  • next time scale-in happens, the most idle Pods in the ReplicaSet get deleted, and the busy Pods can stay busy
• dev tweaks poll interval to the use case, and the load it causes on the cluster

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

/remove-lifecycle stale