/sig apps
/wg batch

/assign

/assign

/sig scheduling
/sig api-machinery

Hello @alculquicondor 👋, 1.25 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PT on Thursday June 23, 2022, which is just over 2 days from now.

For note, This enhancement is targeting for stage alpha for 1.25 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP file using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable
• KEP has a updated detailed test plan section filled out
• KEP has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

The open PR #3374 is addressing all the listed criteria above. We would just require getting it merged by the Enhancements Freeze.

For note, the status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

With KEP PR #3374 merged, the enhancement is ready for the 1.25 Enhancements Freeze.

For note, the status is now marked as tracked. Thank you so much!

Hello @alculquicondor 👋, 1.25 Release Docs Lead here.
This enhancement is marked as ‘Needs Docs’ for 1.25 release.

Please follow the steps detailed in the documentation to open a PR against dev-1.25 branch in the k/website repo. This PR can be just a placeholder at this time, and must be created by August 4.
 Also, take a look at Documenting for a release to familiarize yourself with the docs requirement for the release.

Thank you!

Hi @alculquicondor @mimowo, Enhancements team here again 👋

Checking in as we approach Code Freeze at 01:00 UTC on Wednesday, 3rd August 2022.

Please ensure that the following items are completed before the code-freeze:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
  Refactor gc_controller to do not use the deletePod stub kubernetes#111070
  Refactor taint_manager to do not use getPod and getNode stubs kubernetes#111084
  Add integration test for podgc kubernetes#111091
  Append new pod conditions when deleting pods to indicate the reason for pod deletion kubernetes#110959
  Support handling of pod failures with respect to the configured rules kubernetes#111113
  Add worker to clean up stale DisruptionTarget condition kubernetes#111475
• All PRs are fully merged by the code freeze deadline.
  Append new pod conditions when deleting pods to indicate the reason for pod deletion kubernetes#110959
  Support handling of pod failures with respect to the configured rules kubernetes#111113
  Add worker to clean up stale DisruptionTarget condition kubernetes#111475

Let me know if there are any additional k/k PRs besides the ones listed above

Currently, the status of the enhancement is marked as at-risk

Thanks :)

@Atharva-Shinde @alculquicondor there is one more PR that should be included before the code freeze: kubernetes/kubernetes#111475

thank you @mimowo,
I have updated my comment with the PR and have also tagged you for future reference :)

Yes, that is my proposal

I don't want to stand in the way of solving real problems, but I worry that this becomes conceptual debt that we will never pay off (or even remember!)

@kerthcet already has an open WIP KEP :)
And we have already received good user feedback about the failure policy at the job level.

This could also play nicely with #3967 . Also currently limited to restartPolicy=Never, but we could extend to restartPolicy=OnFailure, and define it in this case to skip counting individual container restarts

That's the point I think Job API can leverage the maxRestartTimesOnFailure to achieve Backoff Limit Per Index.

The max-restarts KEP isn't the same as restart-rules, though, right? They
all seem complementary but not the same.

Yes, currently max-restarts KEP only accounts for the restartPolicy=OnFailure, because restartPolicy=Never & maxRestartTimes=x looks conflict.

The max-restarts KEP isn't the same as restart-rules, though, right? They
all seem complementary but not the same.

Yes, these are the pieces of work that need to be done to support fully restart policy Never and OnFailure:

1. restartPolicy=Never (this KEP, currently Beta without blocking issues open)
2. maxRestartTimes (started as a new KEP)
3. enable restartPolicy=OnFailure (not started, but seems a small follow-up KEP); plus change to Job's backoffLimit when maxRestartTimes is specified
4. containerRestartPolicy in pod spec (not started, AFAIK purely node changes) (enabling API similar to the one in Retriable and non-retriable Pod failures for Jobs #3329 (comment))

Note that:

• (2.) is a prerequisite for (3.)
• (3.) and (4.) and independent technically

IMO doing (1.) - (4.) under this KEP would prolong its graduation substantially, and different points have different priorities, so it is useful to de-couple. Recently we got this slack ask to support OnFailure. We will need to keep track of how much demand there is for (3.) and (4.) and prioritize accordingly, but de-coupling allows that.

Is this planning to go to beta in 1.28? It's on the board for PRR but I don't see an open KEP PR linked here?

@johnbelamaric the KEP is already in Beta.

There are two updates for the third iteration of Beta:

1. https://github.com/kubernetes/enhancements/pull/3965/files (just reflecting a bugfix that is already merged, and status update)
2. https://github.com/kubernetes/enhancements/pull/3940/files/8eb45d267c0885159d53ed9355d5b7995eefdd39#diff-01c599fe2c09bcb8104563c0fd5162bd711cc04814d049357c6dda0ac7bf2a2a (and update currently coupled with the KEP-3939) to delay pod recreation until we have analyzed with pod failure policy the failed pod.

Do you think (2.) should be decoupled as a dedicated PR for this KEP -3329 or can stay as is?

Do you think (2.) should be decoupled as a dedicated PR for this KEP -3329 or can stay as is?

Also asking @wojtek-t who is PRR reviewer of the other KEP (3939).

The other KEP is close to merge so seems safe to keep the update coupled there.

I discussed this with @deads2k and he was ok having a single PR for changes in both KEPs, because changes in 3329 are minimal and highly related to the other KEP

Sure that's fine

Sure that's fine

I reviewed that integration part from the PRR POV carefully too.

Hi @mimowo and @alculquicondor, just checking in again before the enhancement freeze coming up on 01:00 UTC Friday, 16th June 2023. This enhancement is marked as at risk, but it looks like the only thing left is for #3940 to go in. Let me know if I've missed anything. Thanks!

#3940 just merged 😄

With all the requirements fulfilled this enhancement is now marked as tracked for the upcoming enhancements freeze🚀

Hi @alculquicondor

1.28 Docs Shadow here.

Does this enhancement work planned for 1.28 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.28 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 20th July 2023.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

Hey again @alculquicondor 👋

Just checking in as we approach Code freeze at 01:00 UTC Friday, 19th July 2023 .

Here’s the enhancement’s state for the upcoming code freeze:

• All the PRs that are related to your enhancement are linked in the above issue description (for tracking purposes). This includes code, tests, and documentation related PR/s.
• All code related PR/s are merged or are in merge-ready state ( i.e they have approved and lgtm labels applied) by the code freeze deadline. This includes any tests related PR/s too.
  • Add pod disruption conditions for kubelet-initiated failures kubernetes#112360
  • Extend metrics with the new labels kubernetes#113324
  • Use SSA to add pod failure conditions kubernetes#113304
  • Enable the "Retriable and non-retriable pod failures for jobs" feature into beta kubernetes#113360
  • Add e2e test for job pod failure policy used to match pod disruption kubernetes#113812
  • Fix disruption controller permissions to allow patching pod's status kubernetes#113580
  • Fix match onExitCodes when Pod is not terminated kubernetes#113856
  • Wait for Pods to finish before considering Failed in Job kubernetes#113860
  • Add e2e test to ignore failures with 137 exit code kubernetes#113927
  • Fix clearing of rate-limiter for the queue of checks for cleaning stale pod disruption conditions kubernetes#114770
  • Adjust DisruptionTarget condition message to do not include preemptor pod metadata kubernetes#114914
  • PodGC should not add DisruptionTarget condition for pods which are in terminal phase kubernetes#115056
  • Give terminal phase correctly to all pods that will not be restarted kubernetes#115331
  • API-initiated eviction: handle deleteOptions correctly kubernetes#116554
  • Add DisruptionTarget condition when preempting for critical pod kubernetes#117586

For this enhancement, it looks like the following code related PR/s are open and they need to be merged or should be in merge-ready state before the code freeze commences :

• Job: create replacement pods only after terminated kubernetes#117015

Also please let me know if there are other PRs in k/k we should be tracking for this KEP.
As always, we are here to help if any questions come up. Thanks!

We will also have an improvement to this KEP as part of: kubernetes/kubernetes#117015 in 1.28.

Hey @alculquicondor , could you please create a docs PR even if it is a draft PR with no content yet against dev-1.28 branch in the k/website repo. The deadline to create this draft PR is Thursday 20th July 2023.

Thanks @Rishit-dagli, that's gonna be kubernetes/website#41745 (yes, the enhancements are tightly coupled).

I added the necessary PRs to the issue description above.

Hey @alculquicondor 👋 Enhancements Lead here

With all the implementation(code related) PRs merged as per the issue description:
kubernetes/kubernetes#112360
kubernetes/kubernetes#113324
kubernetes/kubernetes#113304
kubernetes/kubernetes#113360
kubernetes/kubernetes#113812
kubernetes/kubernetes#113580
kubernetes/kubernetes#113856
kubernetes/kubernetes#113860
kubernetes/kubernetes#113927
kubernetes/kubernetes#114770
kubernetes/kubernetes#114914
kubernetes/kubernetes#115056
kubernetes/kubernetes#115331
kubernetes/kubernetes#116554
kubernetes/kubernetes#117586

This enhancement is now marked as tracked for the v1.28 Code Freeze!

removed.

/remove-label lead-opted-in

@alculquicondor please update the list of code changes for Beta with this PR: kubernetes/kubernetes#121103.

Hello 👋 1.30 Enhancements Lead here,

I'm closing milestone 1.28 now,
If you wish to progress this enhancement in v1.30, please follow the instructions here to opt in the enhancement and make sure the lead-opted-in label is set so it can get added to the tracking board and finally add /milestone v1.30. Thanks!

/milestone clear