/sig scheduling

/label lead-opted-in

/milestone v1.26

Hey @kerthcet 👋, 1.26 Enhancements team here!

Just checking in as we approach Enhancements Freeze on 18:00 PDT on Thursday 6th October 2022.

This enhancement is targeting for stage alpha for 1.26 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP file using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable
• KEP has an updated detailed test plan section filled out
• KEP has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

For this KEP, we would need to:

• Change the status of kep.yaml to implementable (I've also added a comment review)
• Include the new updated PR of this KEP in the Issue Description and get it merged before Enhancements Freeze to make this enhancement eligible for 1.26 release.

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well.
Thank you :)

Thanks @Atharva-Shinde.

• The KEP is updated to marked as implementable
• Waiting for @johnbelamaric or @wojtek-t to be the approver of PRR: KEP: Pod Scheduling Readiness #3522 (comment)

Hello @Huang-Wei 👋, just a quick check-in again, as we approach the 1.26 Enhancements freeze.

Please plan to get the PR #3522 merged before Enhancements freeze on 18:00 PDT on Thursday 6th October 2022 i.e tomorrow

For note, the current status of the enhancement is marked at-risk :)

Thanks for the reminder. It's 99% accomplished atm, just some final comments waiting for the approver to +1.

With #3522 merged, we have this marked as tracked for v1.26.

Hi @Huang-Wei 👋,

Checking in once more as we approach 1.26 code freeze at 17:00 PDT on Tuesday 8th November 2022.

Please ensure the following items are completed:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PRs are fully merged by the code freeze deadline.

For this enhancement, it looks like the following PRs are open and need to be merged before code freeze:

• [KEP-3521] Part 1: New Pod API .spec.schedulingGates kubernetes#113274
• [KEP-3521] Part 2: Core scheduling implementation kubernetes#113275
• [KEP-3521] Part 3: Bug fixes, integration & E2E Test kubernetes#113442

Please ensure all of these PRs are linked to this issue as well and mentioned in the initial issue description.

As always, we are here to help should questions come up. Thanks!

@marosset yes, those 3 PRs are code implementation for this KEP in alpha stage. I just updated the issue description to get them linked.

Hello @Huang-Wei 👋, 1.26 Release Docs Lead here. This enhancement is marked as ‘Needs Docs’ for 1.26 release.

Please follow the steps detailed in the documentation to open a PR against dev-1.26 branch in the k/website repo. This PR can be just a placeholder at this time, and must be created by November 9. Also, take a look at Documenting for a release to familiarize yourself with the docs requirement for the release.

Any doubt, reach us! Thank you!

Hi @Huang-Wei 👋,

Checking in once more as we approach 1.26 code freeze at 17:00 PDT on Tuesday 8th November 2022.

Please ensure the following items are completed:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PRs are fully merged by the code freeze deadline.

For this enhancement, it looks like the following PRs are open and need to be merged before code freeze:

• [KEP-3521] Part 2: Core scheduling implementation kubernetes#113275
• [KEP-3521] Part 3: Bug fixes, integration & E2E Test kubernetes#113442

As always, we are here to help should questions come up. Thanks!

@marosset ACK, I'm working with reviewers to get 2 pending PRs merged by tomorrow.

I'm working with reviewers to get 2 pending PRs merged by tomorrow.

All dev work has been merged.

For people wanting to request a specific node but still use the scheduling lifecycle / scheduling gates, etc, they should do what the daemonset controller does, and use nodeAffinity to target a single node without setting spec.nodeName

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - <node-name-here>

.spec.nodeName was historically designed as a terminal state of pod's destination to run. You may find quite some enforcements building atop it:

• .spec.nodeName can only be set once (there was a flaw that we didn't make it RBAC-ed in the first place though)
• .spec.nodeName cannot be cleared once set
• recommended way to set .spec.nodeName is to create a v1.Binding request and delegate the "setting .spec.nodeName" to API Server - which is the way kube-scheduler does, but not always for other schedulers
• setting .spec.nodeName directly would cause conflicting scheduling results (it'd behave like you have multi scheduler running at the same time) - it was the motivation of enforcing scheduler to take over all Pod's scheduling including Daemonset pods

So, to adhere to existing semantics, which we cannot change basically, like Jordan said, express the desire of "placing a pod to node X" via nodeAffinity/nodeSelector rather than setting .spec.nodeName.

.spec.nodeName can only be set once (there was a flaw that we didn't make it RBAC-ed in the first place though)

RBAC didn't exist until Kube 1.6, spec.nodeName was there from the start :-)

Nice summary, just one follow-up for my own education:

recommended way to set .spec.nodeName is to create a v1.Binding request and delegate the "setting .spec.nodeName" to API Server

Why is this better than setting it directly (for example, via a patch)?

@alculquicondor

There was a time in the Kubernetes history when Status was supposed to only contain information that could be restored from something else. The node node name could not have been restored from somwhere else, so the decision was made to put it in the Spec.

That assumption is no longer true today. Status can be treated as a source of truth and it would be more appropriate for nodeName to be in Status. But it's not, because it would be a breaking change.

I (and all of us, I believe) fully agree. That's a bad, unrecommended and confusing field that would have never been able to get in today and it is there only for historical reasons.

That said, that doesn't imply that the nodeName means or ever meant "I want to land on node X". It means: "this pod is in node X".

With this statement I don't fully agree.
You're treating this field as if it's a purely status field, but in essence it's not, it's a spec field. And while it's unrecommended to use it, people still use it and it is valid as long as this field is not deprecated. Not only that, but as a user it would be completely unintuitive to me that I need to set node affinity and not use this field.

Furthermore, it's not accurate that is means "this pod is in node X". For example, if I try to run a pod with nodeName set but that asks for more resources than available on the node, I'll see this:

> k get pods -w
NAME     READY   STATUS        RESTARTS   AGE
fedora   0/1     OutOfmemory   0          4s

What actually happened here?
What happened is that the user said "the desired state for my pod is that it needs to run on node X". But, in practice, the actual state is that the pod cannot run on that node. Saying that this pod is "on node X" is not accurate. It's more accurate to say "the pod wants to run on node X, but it is not possible".

Yes, it is reasonable because that's not what the nodeName is intented for.

Would you also say that for standard Resource Quota?
In other words, would it be reasonable in your opinion that if a ResourceQuota exists on a namespace it would deny pods that set nodeName? I would assume that it would be completely unacceptable, and that the community would demand to support this scenario (and that's indeed how it's implemented). Would you agree?

The nodeName field is a field that is for schedulers to manage, not end users or other kinds of controllers. It is as if the users of this field say "I want to bypass scheduler or an external quota mechanism".

Letting the user say "I want to bypass external quota mechanism" is absolutely against the concept of quota in the first place. The whole idea is forcing users under a certain namespace to comply to the quota. Effectively this means that external quota mechanisms have to work that out, like we do in the AAQ operator. In my opinion this is a major issue since one of the main goals of this KEP is to support such external mechanisms.

And again, the user saying "I want to bypass quota" is absolutely unacceptable with the standard Resource Quota, so I feel it's a bit unfair to apply this rule only for external mechanisms.

I guess that my perspective can be boiled to: nodeName is either a valid field in spec that's acceptable to use by users to specify their pods' desired state or it is an internal field that should be never used by users.

If it is a valid spec field - we should treat setting it as a completely valid use-case that should be supported with both standard and external k8s mechanisms.

If it's an internal field that should never be set by users - we should deprecate it and disallow its usage.

Saying "it's a valid spec field, but users should never use it, therefore it's fine to not support it, but we won't deprecate it because we don't want to break it" seems contradicting to me.

For people wanting to request a specific node but still use the scheduling lifecycle / scheduling gates, etc, they should do what the daemonset controller does, and use nodeAffinity to target a single node without setting spec.nodeName

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - <node-name-here>

Thanks @liggitt.

For my education: is there a user-case where someone would need to use nodeName instead of node-affinity? Or is it true that in 100% of the times it's better to set node affinity instead of nodeName?

That said, that doesn't imply that the nodeName means or ever meant "I want to land on node X". It means: "this pod is in node X".

With this statement I don't fully agree.

As soon as a pod with this field is created, it appears in the corresponding node's pod watch (filtered to spec.nodeName=$myName). The field does mean the pod is scheduled to the node.

As you point out, the node can still reject it or terminate it, but the pod is being handled by the node at that point, for better or worse.

For my education: is there a user-case where someone would need to use nodeName instead of node-affinity? Or is it true that in 100% of the times it's better to set node affinity instead of nodeName?

If they are writing something that is intended to roll pod creation and scheduling into a single step, and are acting as both pod creator and scheduler, setting spec.nodeName on create is logically coherent. Anyone intending to use the normal scheduling flow should not set spec.nodeName.

If it's an internal field that should never be set by users - we should deprecate it and disallow its usage.

"Internal" is a blurry line for an API-driven system. Is a custom scheduler "internal"? Is a custom create-and-schedule-in-a-single-step integration "internal"?

We will not break the current use of the field. We can improve documentation about it.

Why is this better than setting it directly (for example, via a patch)?

@pohly by setting .spec.nodeName, it's complex to be guarded against (you have to build admission control or leverage OPA); while via a separate API endpoint (v1.Binding), it's easier to be guarded by RBAC.

So the goal was to make setting .spec.nodeName only possible through v1.Binding, it's just not enforced? Is that still the intent or has that goal been discarded as infeasible (major breaking change)?

Please try to look at this from a user perspective. Setting .spec.nodeName seems like the intuitive choice when you want a pod to be deployed on a specific node. It appears natural and convenient. However, the correct setting, which might not be immediately intuitive, involves a long and complex expression:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
            - key: metadata.name
              operator: In
              values:
              - <node-name-here>

If they are writing something that is intended to roll pod creation and scheduling into a single step, and are acting as both pod creator and scheduler, setting spec.nodeName on create is logically coherent. Anyone intending to use the normal scheduling flow should not set spec.nodeName.

Is there an example of a practical case when this is needed? I still don't see why nodeAffinity is not always the right choice in practice.

There is another option:

spec:
  nodeSelector:
    kubernetes.io/hostname: <node-hostname>

Is there an example of a practical case when this is needed?

To oversimplify: only use .spec.nodeName if you are a scheduler.

I still don't see why nodeAffinity is not always the right choice in practice.

From a user perspective, it is the right choice. Unless you are an administrator trying to run a pod in an emergency, for example, if the scheduler is down.

There is another option:

spec:
  nodeSelector:
    kubernetes.io/hostname: <node-hostname>

this label can be modified

If they are writing something that is intended to roll pod creation and scheduling into a single step, and are acting as both pod creator and scheduler, setting spec.nodeName on create is logically coherent. Anyone intending to use the normal scheduling flow should not set spec.nodeName.

To oversimplify: only use .spec.nodeName if you are a scheduler.

@liggitt @alculquicondor

I'm trying to understand if there's an actual use-case for non-schedulers to use this field.
Except for the scheduler being down, what do I gain from setting nodeName instead of node affinity? Is there any difference from a user's perspective whatsoever? Even if I would write a custom scheduler, what would I gain from using nodeName instead node affinity?

I'm not sure I fully understand why we'd want to keep this field around. Do we want to grant users the possibility of bypassing the scheduler, especially if there is never a valuable use-case that justifies it?

The following crazy idea pops into my head:

1. Add a .status.nodeName field. This is the field that should be watched by k8s components / external controllers to understand to which node the pod is scheduled.
2. If deprecating .spec.nodeName is completely unacceptable for backward compatibility reasons, k8s can mutate it to the equivalent node-affinity, perhaps alongside a warning that notifies the user of what happened.
3. Longer term - if possible - deprecate and remove this field entirely.

If there's a must-have concrete reason (I haven't seen one yet) to let users to bypass the scheduler, we can introduce a field with a clearer name like spec.bypassScheduling so it would be clear what its use is.

Just a crazy idea :)

Even if I would write a custom scheduler, what would I gain from using nodeName instead node affinity?

An integration that both created and scheduled the pod would set spec.nodeName directly, instead of creating the pod and immediately calling pods/binding to set spec.nodeName. Translating spec.nodeName on create into affinity would break that integration.

I'm not sure I fully understand why we'd want to keep this field around.

There are lots of readers of the field, so we would never remove it. We continue to allow writing it on pod create for compatibility with existing integrations that set it on create.

we can introduce a field with a clearer name like spec.bypassScheduling so it would be clear what its use is.

Requiring a new field to be set to keep existing behavior is just as breaking for compatibility :-/

I'm trying to understand if there's an actual use-case for non-schedulers to use this field.

Again, oversimplifying, there is no use-case.

this label can be modified

Yes, it can, but you might be breaking a few first-party and third-party controllers that assume that this label matches the nodeName or at least that it is unique. The label is documented as well-known, so it should be treated with care https://kubernetes.io/docs/reference/labels-annotations-taints/#kubernetesiohostname

A good example of how .spec.nodeName means "scheduling done" is by setting nodeName and an arbitrary resource, the resource will be ignored, and the pod fails with

…
status:
  phase: Failed
…
  message: 'Pod was rejected: Node didn''t have enough resource: cpu, requested: 400000000, used: 400038007, capacity: 159500'
  reason: OutOfcpu
…
  containerStatuses:
    - name: nginx
      state:
        terminated:
          exitCode: 137
          reason: ContainerStatusUnknown
          message: The container could not be located when the pod was terminated
…
      image: 'nginx:1.14.2'
      started: false

To some extent .spec.nodeName could be considered to be a scheduling gate as well - a special one: Only if it is set, then we can (are) schedule(d).
However, due to it's special meaning in the coordination with kubelet, we can not change it's seamtics.

I do not know the current state, but I do wonder - if we are not already doing this today - a Pod with scheduling gates and spec.nodeName should be rejected at admission time.

A good example of how .spec.nodeName means "scheduling done" is by setting nodeName and an arbitrary resource, the resource will be ignored, and the pod fails with

…
status:
  phase: Failed
…
  message: 'Pod was rejected: Node didn''t have enough resource: cpu, requested: 400000000, used: 400038007, capacity: 159500'
  reason: OutOfcpu
…
  containerStatuses:
    - name: nginx
      state:
        terminated:
          exitCode: 137
          reason: ContainerStatusUnknown
          message: The container could not be located when the pod was terminated
…
      image: 'nginx:1.14.2'
      started: false

To some extent .spec.nodeName could be considered to be a scheduling gate as well - a special one: Only if it is set, then we can (are) schedule(d). However, due to it's special meaning in the coordination with kubelet, we can not change it's seamtics.

I do not know the current state, but I do wonder - if we are not already doing this today - a Pod with scheduling gates and spec.nodeName should be rejected at admission time.

@fabiand Yes, it will be rejected at the admission.
However, the Application-Aware Quota adds the scheduling gate to pods at the admission time as well.
Therefore, pods created with .spec.nodeName will get rejected in the namespaces where AAQ operates.
One example is with kubectl debug node which explicitly sets .spec.nodeName in the pod spec.

I share that it's a general problem, but due to the special handling of .spec.nodeName I do not see how we can resolve the problem in

1. a backwards compatile manner
2. keeping the user spec

I do fear that - in your example - kubectl debug or oc debug should change and use affinity instead.

The core problem is that kubelet starts to react once nodeName is set.

Was it considered to change kubelet to only start acting once nodeName is set and schedulingGates is empty?

Was it considered to change kubelet to only start acting once nodeName is set and schedulingGates is empty?

According to the version skew policy, the change would have to be in the kubelet for 3 versions before we can relax the validation in apiserver.

I guess that could be backwards compatible if we start in 1.31 and we allow scheduling_gates + nodeName in 1.34.
@liggitt wdyt?

One example is with kubectl debug node which explicitly sets .spec.nodeName in the pod spec.

IMO, that falls under the special case where it might make sense to skip scheduler or an external quota system. You probably wouldn't even want to set requests in a debug pod.

probably wouldn't even want to set requests in a debug pod.

FWIW - I do wonder if debug pods should actually be guaranteed. I had a couple of cases where debug pods (as best effort) got killed quickly on busy nodes.

I guess that could be backwards compatible if we start in 1.31 and we allow scheduling_gates + nodeName in 1.34.
@liggitt wdyt?

that seems like making the problem and confusion around use of spec.nodeName worse to me... I don't see a compelling reason to do that

@alculquicondor

IMO, that falls under the special case where it might make sense to skip scheduler or an external quota system.

TBH, I still try to understand how skipping the scheduler is ever helpful (when you're not using a custom scheduler).
Can you please elaborate on how skipping the scheduler helps in this case? In other words, if we would change kubectl debug to use node affinity instead of nodeName, what would be the implications?

You probably wouldn't even want to set requests in a debug pod.

While this might be correct, the question to me is who makes the decision. Granting a user a knob to skip quota mechanisms feels to me like granting a linux user to bypass permission checks when writing to a file. In both cases the whole idea is to restrict the users and enforce them to comply to a certain policy. Handing the user the possibility to bypass such mechanisms seems entirely contradicting to me and de-facto it makes external quota mechanisms unpractical.

@liggitt

that seems like making the problem and confusion around use of spec.nodeName worse to me... I don't see a compelling reason to do that

Are you open to discussion on that?
IMHO this behavior is consistent with scheduling gates' intent to (as the KEP states) force creating pods in a "not-ready to schedule" state. I understand that technically the pod is not scheduled at all with nodeName set (although I think we all agree it's a surprising behavior that's kept for backward compatibility), however having kubelet to wait for the scheduling gates to be removed before running the pod with a some kind of "not ready" condition sounds very intuitive and consistent to me.

This way we can avoid breaking backward compatibility, support external quota mechanisms and extend scheduling-gates in a consistent matter, which IMHO makes the exceptional nodeName case to be less exceptional.

having kubelet to wait for the scheduling gates to be removed before running the pod with a some kind of "not ready" condition sounds very intuitive and consistent to me.

Not to me... expecting pods which are already assigned to a node to run through scheduling gate removal phases (which couldn't do anything to affect the selected node) seems more confusing than the current behavior which simply forbids that combination. I don't think we should relax validation there and increase confusion.

I (sadly) concur - If the nodeName is set, then it's a matter of fact that the scheduling has happened. This is the contract we can not break.

Fun fact: I created a deployment with pod template that had nodeName defined AND a resource (cpu: 400k) which can not be met. The Deployment had an RC of 2. But due to this scheduling contradiction it ended up with 12k failed pods.

IOW: I wonder if this spec.nodeName is troublesome for other controllers in kube as well.

I share that it's a general problem, but due to the special handling of .spec.nodeName I do not see how we can resolve the problem in

1. a backwards compatile manner
2. keeping the user spec

I do fear that - in your example - kubectl debug or oc debug should change and use affinity instead.

The reason kubectl debug or oc debug creates pods using .spec.nodeName instead of affinity is likely because it allows the debug pod to bypass taint constraints.

Although it might be achievable with tolerations as well.