In-Place Update of Pod Resources #1287

vinaykul · 2019-10-08T04:46:18Z

Enhancement Description

Please to keep this description up to date. This will help the Enhancement Team track efficiently the evolution of the enhancement

~~Identify CRI changes needed for UpdateContainerResources API, define response message for UpdateContainerResources~~
- ~~Extend UpdateContainerResources API to return info such as ‘not supported’, ‘not enough memory’, ‘successful’, ‘pending page evictions’ etc.~~
- ~~Define expected behavior for runtime when UpdateContainerResources is invoked. Define timeout duration of the CRI call.~~
  - Resolution: Separate KEP for CRI changes.
    - Discussed draft CRI changes with SIG-Node on Oct 22, and we agreed to do this as an incremental change outside the scope of this KEP, in a new mini-KEP. It does not block implementation of this KEP.
Define behavior when multiple containers are being resized, and UpdateContainerResources fails for one or more containers.
- One Possible solution:
  - Do not update Status.Resources.Limits if UpdateContainerResources API fails, and keep retrying until it succeeds.
~~Check with API reviewers if we can keep maps instead list of named sub-objects for ResizePolicy.~~
- After discussion with @liggitt , we are going to use list of named subobjects for extensibility.
Can we find a more intuitive name for ResizePolicy?
Can we use ResourceVersion to figure out the ordering of Pod resize requests?
~~Do we need to add back the ‘RestartPod’ resize policy? Is there a strong use-case for it?~~
- Resolution: No.
  - Discussed with SIG-Node on Oct 15th, not adding RestartPod policy for simplicity, will revisit if we encounter problems.

Alpha Feature Code Issues:
These are Items and issues discovered during code review that need further discussion and need to be addressed before Beta.

~~Can we figure out GetPodQOS differently once it is determined on pod create? See In-place Pod Vertical Scaling feature kubernetes#102884 (comment)~~
- Fixed by Perf optimization: GetPodQOS() returns persisted value of PodStatus.QOSClass, if set. kubernetes#119665
How do we deal with a pod that requests 1m/1m cpu requests/limits. See In-place Pod Vertical Scaling feature kubernetes#102884 (comment)
Add internal representation of ContainerStatus.Resources in kubeContainer. Convert it to ContainerStatus.Resources in kubelet_pods generate functions. See In-place Pod Vertical Scaling feature kubernetes#102884 (comment) and In-place Pod Vertical Scaling feature kubernetes#102884 (comment) and In-place Pod Vertical Scaling feature kubernetes#102884 (comment)
Can we get rid of resize mutex? Is there a better way to handle resize retries? See In-place Pod Vertical Scaling feature kubernetes#102884 (comment)
Can we recover from resize checkpoint store failures? See In-place Pod Vertical Scaling feature kubernetes#102884 (comment)
CRI clarification for ContainerStatus.Resources and how to handle runtimes that don't support it. See In-place Pod Vertical Scaling feature kubernetes#102884 (comment)
~~Add real values to dockershim test for ContainerStatus.Resources In-place Pod Vertical Scaling feature kubernetes#102884 (comment)~~
- Resolution: Not required due to dockershim deprecation.
~~Change PodStatus.Resources from v1.ResourceRequirements to *v1.ResourceRequirements~~
- Resolution: Fixed
Address all places in the code that has 'TODO(vinaykul)'
Current implementation does not work with node toploogy manager enabled. This limitation is not capturedi in the KEP. Add this to the release documentation for alpha, we will address this in beta. See In-place Pod Vertical Scaling feature kubernetes#102884 (comment)

vinaykul · 2019-10-08T04:47:10Z

/assign @vinaykul

jeremyrickard · 2019-10-09T14:26:31Z

👋 Hey there @vinaykul. I'm a shadow on the 1.17 Release Team, working on Enhancements. We're tracking issues for the 1.17 release and I wanted to reach out and ask we should track this (or more specifically I guess the In-Place Update of Pod Resources feature) for 1.17?

The current release schedule is:

Monday, September 23 - Release Cycle Begins
Tuesday, October 15, EOD PST - Enhancements Freeze
Thursday, November 14, EOD PST - Code Freeze
Tuesday, November 22 - Docs must be completed and reviewed
Monday, December 9 - Kubernetes 1.17.0 Released

We're only 5 days away from the Enhancements Freeze, so if you intend to graduate this capability in the 1.17 release, here are the requirements that you'll need to satisfy:

KEP must be merged in implementable state
KEP must define graduation criteria
KEP must have a test plan defined

Thanks @vinaykul

vinaykul · 2019-10-10T21:43:23Z

KEP must be merged in implementable state

KEP must define graduation criteria

KEP must have a test plan defined

Hi @jeremyrickard I'll do my best to get this KEP to implementable state by next Tuesday, but it looks like a stretch at this point - the major item is to complete API review with @thockin , and that depends on his availability.

The actual code changes are not that big. Nevertheless, the safe option would be to track this for 1.18.0 release, I'll update you by next Monday.

CC: @dashpole @derekwaynecarr @dchen1107

vinaykul · 2019-10-14T16:38:37Z

@jeremyrickard @mrbobbytables This KEP will take some more discussion - key thing is API review. It does not look like @thockin or another API reviewer is available soon. Could we please track this KEP for v1.18?
Thanks,

jeremyrickard · 2019-10-14T16:49:39Z

/milestone v1.18

vinaykul · 2019-10-22T05:25:59Z

@PatrickLang Here's a first stab at the proposed CRI change to allow UpdateContainerResources to work with Windows. Please take a look.. let's discuss in tomorrow's sig meeting

root@skibum:~/km16/staging/src/k8s.io/cri-api# git diff --cached .
diff --git a/staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto b/staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto
index 0290d0f..b05bb56 100644
--- a/staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto
+++ b/staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto
@@ -924,14 +924,33 @@ message ContainerStatusResponse {
     map<string, string> info = 2;
 }
 
+// ContainerResources holds the fields representing a container's resource limits
+message ContainerResources {
+    // Resource configuration specific to Linux container.
+    LinuxContainerResources linux = 1;
+    // Resource configuration specific to Windows container.
+    WindowsContainerResources windows = 2;
+}
+
 message UpdateContainerResourcesRequest {
     // ID of the container to update.
     string container_id = 1;
-    // Resource configuration specific to Linux containers.
+    // Resource configuration specific to Linux container.
     LinuxContainerResources linux = 2;
+    // Resource configuration specific to Windows container.
+    WindowsContainerResources windows = 3;
 }
 
-message UpdateContainerResourcesResponse {}
+message UpdateContainerResourcesResponse {
+    // ID of the container that was updated.
+    string container_id = 1;
+    // Resource configuration currently applied to the Linux container.
+    LinuxContainerResources linux = 2;
+    // Resource configuration currently applied to the Windows container.
+    WindowsContainerResources windows = 3;
+    // Error message if UpdateContainerResources fails in the runtime.
+    string error_message = 4;
+}
 
 message ExecSyncRequest {
     // ID of the container.
diff --git a/staging/src/k8s.io/cri-api/pkg/apis/services.go b/staging/src/k8s.io/cri-api/pkg/apis/services.go
index 9a22ecb..9f1d893 100644
--- a/staging/src/k8s.io/cri-api/pkg/apis/services.go
+++ b/staging/src/k8s.io/cri-api/pkg/apis/services.go
@@ -44,7 +44,7 @@ type ContainerManager interface {
        // ContainerStatus returns the status of the container.
        ContainerStatus(containerID string) (*runtimeapi.ContainerStatus, error)
        // UpdateContainerResources updates the cgroup resources for the container.
-       UpdateContainerResources(containerID string, resources *runtimeapi.LinuxContainerResources) error
+       UpdateContainerResources(containerID string, resources *runtimeapi.ContainerResources) error
        // ExecSync executes a command in the container, and returns the stdout output.
        // If command exits with a non-zero exit code, an error is returned.
        ExecSync(containerID string, cmd []string, timeout time.Duration) (stdout []byte, stderr []byte, err error)

dashpole · 2019-10-24T20:23:04Z

@vinaykul It looks like since the above PR was merged, this was removed from the API review queue. I believe you need to open a new PR that moves the state to implementable, and then add the API-review label to get it back in the queue and get a reviewer.

Edit: you should also include any other changes (e.g. windows CRI changes) required to move the feature to implementable in the PR as well.

vinaykul · 2019-10-29T00:38:18Z

@vinaykul It looks like since the above PR was merged, this was removed from the API review queue. I believe you need to open a new PR that moves the state to implementable, and then add the API-review label to get it back in the queue and get a reviewer.

Edit: you should also include any other changes (e.g. windows CRI changes) required to move the feature to implementable in the PR as well.

@dashpole Thanks!

I've started a provisional mini-KEP per our discussion last week for the CRI changes (Dawn mentioned last week that we should take that up separately). imho the CRI changes does not block the implementation of this KEP, as it is between Kubelet and runtime, and user is not affected by it.

In a second commit to the same PR, I've addressed another key issue (update api failure handling), and requested change to move primary KEP to implementable.

With this, everything is in one place, and we can use it for API review.

palnabarun · 2020-01-13T07:04:31Z

Hey there @vinaykul -- 1.18 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating to alpha in 1.18?

The current release schedule is:

Monday, January 6th - Release Cycle Begins
Tuesday, January 28th EOD PST - Enhancements Freeze
Thursday, March 5th, EOD PST - Code Freeze
Monday, March 16th - Docs must be completed and reviewed
Tuesday, March 24th - Kubernetes 1.18.0 Released

To be included in the release,

The KEP PR must be merged
The KEP must be in an implementable state
The KEP must have test plans and graduation criteria.

If you would like to include this enhancement, once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

We'll be tracking enhancements here: http://bit.ly/k8s-1-18-enhancements

Thanks! :)

vinaykul · 2020-01-14T16:47:56Z

Hey there @vinaykul -- 1.18 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating to alpha in 1.18?

The current release schedule is:

Monday, January 6th - Release Cycle Begins

Tuesday, January 28th EOD PST - Enhancements Freeze

Thursday, March 5th, EOD PST - Code Freeze

Monday, March 16th - Docs must be completed and reviewed

Tuesday, March 24th - Kubernetes 1.18.0 Released

To be included in the release,

The KEP PR must be merged

The KEP must be in an implementable state

The KEP must have test plans and graduation criteria.

If you would like to include this enhancement, once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

We'll be tracking enhancements here: http://bit.ly/k8s-1-18-enhancements

Thanks! :)

@palnabarun Yes, I'm planning to work towards alpha code targets for this feature in 1.18. I've updated the KEP adding test plan and graduation criteria sections that I will be reviewing with SIG-Node this week and hope to get it implementable before Jan 28. I'll update this thread if anything changes.

palnabarun · 2020-01-14T17:00:19Z

Thank you @vinaykul for the updates. :)

palnabarun · 2020-01-14T17:00:27Z

/stage alpha

palnabarun · 2020-01-14T17:00:36Z

/milestone v1.18

salehsedghpour · 2024-01-16T23:24:49Z

Hello 👋 1.30 Enhancements Lead here,

I'm closing milestone 1.29 now,
If you wish to progress this enhancement in v1.30, please follow the instructions here to opt in the enhancement and make sure the lead-opted-in label is set so it can get added to the tracking board and finally add /milestone v1.30. Thanks!

/milestone clear

pacoxu · 2024-01-18T02:44:30Z

kubernetes/kubernetes#122701 revert a change in resource quota. Link here to make your attentions.

carlory · 2024-01-19T06:41:14Z

@liggitt @pacoxu should I cherry-pick this PR into 1.29,1.28 and 1.27?

the feature was merged in 1.27 kubernetes/kubernetes#102884

Jeffwan · 2024-01-19T07:13:24Z

Hello 👋 1.30 Enhancements Lead here,

I'm closing milestone 1.29 now, If you wish to progress this enhancement in v1.30, please follow the instructions here to opt in the enhancement and make sure the lead-opted-in label is set so it can get added to the tracking board and finally add /milestone v1.30. Thanks!

/milestone clear

we do have few known changes want to check in to v1.30. I will follow the instruction and make it

liggitt · 2024-01-19T07:32:09Z

No, this is an alpha feature, which we don't do backports for, see https://github.com/kubernetes/community/blob/master/contributors/devel/sig-release/cherry-picks.md

…

On Thu, Jan 18, 2024 at 10:41 PM 杨朱 · Kiki ***@***.***> wrote: @liggitt <https://github.com/liggitt> should I cherry-pick this PR into 1.29,1.28 an 1.27? the feature was merged in 1.27 kubernetes/kubernetes#102884 <kubernetes/kubernetes#102884> — Reply to this email directly, view it on GitHub <#1287 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHPI4XYO3H2MXLN3NEZTXDYPIIRTAVCNFSM4I6M33OKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBZHE4DKMRQGA4A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Jeffwan · 2024-01-23T14:22:34Z

@salehsedghpour #4433 keep it tracked in v1.30

Jeffwan · 2024-01-23T14:39:18Z

v1.30 tracking issues

SergeyKanzhelev · 2024-01-26T23:15:21Z

/stage alpha
/milestone v1.30

salehsedghpour · 2024-01-29T00:00:36Z

Hello @Jeffwan , 1.30 Enhancements team here! Is this enhancement targeting 1.30? If it is, can you follow the instructions here to opt in the enhancement and make sure the lead-opted-in label is set so it can get added to the tracking board? Thanks!

Jeffwan · 2024-02-06T16:26:34Z

@salehsedghpour related issues has been correctly labeled.

Jeffwan · 2024-02-06T16:38:12Z

tracking issues for v1.30

salehsedghpour · 2024-02-09T03:30:52Z

This enhancement didn't have the lead-opted-in label and thus was not tracked.
/milestone clear

thockin · 2024-02-09T03:52:01Z

I think this was a flub - exception?

salehsedghpour · 2024-02-09T08:27:35Z

Hi @Jeffwan and @thockin .

Yes, an exception seems possible. And the remaining parts for this enhancement to be tracked and included in this release are:

Make sure that the lead-opted-in label is added.
Make sure that the status in kep.yaml is marked with implementable.
And make sure that KEP-4176: A Static Policy Option to spread hyperthreads across physical CPUs #4177 is being merged.

Jeffwan · 2024-02-09T11:24:23Z

@salehsedghpour @thockin I just filed an exception for this KEP-4176

esotsal · 2024-03-01T10:51:13Z

Current implementation does not work with node toploogy manager enabled. This limitation is not capturedi in the KEP. Add this to the release documentation for alpha, we will address this in beta. See kubernetes/kubernetes#102884 (comment)

kubernetes/kubernetes#123319 is an attempt to fix it

k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Oct 8, 2019

k8s-ci-robot assigned vinaykul Oct 8, 2019

vinaykul mentioned this issue Oct 8, 2019

KEP: in-place update of pod resources #686

Merged

mrbobbytables added sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling. sig/node Categorizes an issue or PR as relevant to SIG Node. labels Oct 14, 2019

k8s-ci-robot removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 14, 2019

mrbobbytables added this to the v1.17 milestone Oct 14, 2019

k8s-ci-robot modified the milestones: v1.17, v1.18 Oct 14, 2019

jeremyrickard added tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team and removed tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team labels Oct 14, 2019

k8s-ci-robot added the stage/alpha Denotes an issue tracking an enhancement targeted for Alpha status label Jan 14, 2020

palnabarun removed the tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team label Jan 14, 2020

kannon92 mentioned this issue Jan 15, 2024

In-Place Pod Resource Resizing always restarts pod kubernetes/kubernetes#122760

Open

k8s-ci-robot removed this from the v1.29 milestone Jan 16, 2024

haircommander mentioned this issue Jan 17, 2024

InPlacePodVerticalScaling can make node requests be over 100 percent kubernetes/kubernetes#122602

Closed

Jeffwan mentioned this issue Jan 23, 2024

KEP-1287: Keep in-place pod resize KEP alpha/implementable for v1.30 #4433

Merged

k8s-ci-robot added this to the v1.30 milestone Jan 26, 2024

k8s-ci-robot removed this from the v1.30 milestone Feb 9, 2024

sftim mentioned this issue Feb 27, 2024

Vertical Scaling of Pods #21

Closed

18 tasks

joelsmith mentioned this issue Mar 6, 2024

Add InPlacePodVerticalScaling featuregate (TechPreviewNoUpgrade) openshift/api#1782

Open

bplasmeijer mentioned this issue Mar 19, 2024

VPA with "in-place" patch of resource allocation for pods Azure/AKS#3950

Closed

esotsal mentioned this issue Mar 22, 2024

WIP: Static CPU management policy alongside InPlacePodVerticalScaling kubernetes/kubernetes#123319

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

In-Place Update of Pod Resources #1287

In-Place Update of Pod Resources #1287

vinaykul commented Oct 8, 2019 •

edited by pacoxu

vinaykul commented Oct 8, 2019

jeremyrickard commented Oct 9, 2019 •

edited

vinaykul commented Oct 10, 2019

vinaykul commented Oct 14, 2019

jeremyrickard commented Oct 14, 2019

vinaykul commented Oct 22, 2019

dashpole commented Oct 24, 2019 •

edited

vinaykul commented Oct 29, 2019

palnabarun commented Jan 13, 2020 •

edited

vinaykul commented Jan 14, 2020

palnabarun commented Jan 14, 2020

palnabarun commented Jan 14, 2020

palnabarun commented Jan 14, 2020

salehsedghpour commented Jan 16, 2024

pacoxu commented Jan 18, 2024

carlory commented Jan 19, 2024 •

edited

Jeffwan commented Jan 19, 2024

liggitt commented Jan 19, 2024 via email

Jeffwan commented Jan 23, 2024 •

edited

Jeffwan commented Jan 23, 2024

SergeyKanzhelev commented Jan 26, 2024

salehsedghpour commented Jan 29, 2024

Jeffwan commented Feb 6, 2024

Jeffwan commented Feb 6, 2024 •

edited

salehsedghpour commented Feb 9, 2024

thockin commented Feb 9, 2024

salehsedghpour commented Feb 9, 2024

Jeffwan commented Feb 9, 2024

esotsal commented Mar 1, 2024 •

edited

In-Place Update of Pod Resources #1287

In-Place Update of Pod Resources #1287

Comments

vinaykul commented Oct 8, 2019 • edited by pacoxu

Enhancement Description

Please to keep this description up to date. This will help the Enhancement Team track efficiently the evolution of the enhancement

vinaykul commented Oct 8, 2019

jeremyrickard commented Oct 9, 2019 • edited

vinaykul commented Oct 10, 2019

vinaykul commented Oct 14, 2019

jeremyrickard commented Oct 14, 2019

vinaykul commented Oct 22, 2019

dashpole commented Oct 24, 2019 • edited

vinaykul commented Oct 29, 2019

palnabarun commented Jan 13, 2020 • edited

vinaykul commented Jan 14, 2020

palnabarun commented Jan 14, 2020

palnabarun commented Jan 14, 2020

palnabarun commented Jan 14, 2020

salehsedghpour commented Jan 16, 2024

pacoxu commented Jan 18, 2024

carlory commented Jan 19, 2024 • edited

Jeffwan commented Jan 19, 2024

liggitt commented Jan 19, 2024 via email

Jeffwan commented Jan 23, 2024 • edited

Jeffwan commented Jan 23, 2024

SergeyKanzhelev commented Jan 26, 2024

salehsedghpour commented Jan 29, 2024

Jeffwan commented Feb 6, 2024

Jeffwan commented Feb 6, 2024 • edited

salehsedghpour commented Feb 9, 2024

thockin commented Feb 9, 2024

salehsedghpour commented Feb 9, 2024

Jeffwan commented Feb 9, 2024

esotsal commented Mar 1, 2024 • edited

vinaykul commented Oct 8, 2019 •

edited by pacoxu

jeremyrickard commented Oct 9, 2019 •

edited

dashpole commented Oct 24, 2019 •

edited

palnabarun commented Jan 13, 2020 •

edited

carlory commented Jan 19, 2024 •

edited

Jeffwan commented Jan 23, 2024 •

edited

Jeffwan commented Feb 6, 2024 •

edited

esotsal commented Mar 1, 2024 •

edited