/sig node
/sig scheduling
/wg device-management

I think sig-scheduling lead needs to opt-in for this feature for it to be tracked for 1.33.

/assign

/label lead-opted-in

Hello @pohly 👋, v1.33 Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 14th February 2025 / 19:00 PDT Thursday 13th February 2025.

This enhancement is targeting stage alpha for v1.33 (correct me, if otherwise)
/stage alpha

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.32.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 6th February 2025 so that the PRR team has enough time to review your KEP.

For this KEP, we would need to update the following:

• Create the KEP readme using the latest template and merge it in the k/enhancements repo.
• Ensure that the KEP has undergone a production readiness review and has been merged into k/enhancements.

It looks like #5034 will address most of these issues.

• Get KEP-5027 + 5055: DRA: admin-controlled device attributes + taints #5034 merged before the enhancements freeze.

The status of this enhancement is marked as At risk for enhancements freeze. Please keep the issue description up-to-date with appropriate stages as well

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

Hi @pohly 👋, 1.33 Enhancements team here,

Just a quick friendly reminder as we approach the enhancements freeze later this week, at 02:00 UTC Friday 14th February 2025 / 19:00 PDT Thursday 13th February 2025.

The current status of this enhancement is marked as At risk for enhancement freeze. There are a few requirements mentioned in the comment #5055 (comment) that still need to be completed.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

Hello @pohly 👋, 1.33 Enhancements team here,

With PR #5034 has been merged, all the KEP requirements are in place and merged into k/enhancements.

Before the enhancement freeze, it would be appreciated if following nits could be addressed:

• Update issue description to add direct link to KEP README.md.

  • Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/5055-dra-device-taints-and-tolerations/README.md

• Update issue description to add the KEP update PR for alpha stage

  • Alpha
    • KEP (k/enhancements) update PR(s): KEP-5027 + 5055: DRA: admin-controlled device attributes + taints #5034

Aside from the minor nits mentioned above, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is now marked as tracked for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

/label tracked/yes

Thanks for the update and for clarifying the changes for KEP-5055. I’ll go ahead and mark this as tracked again, since it was previously marked removed from milestone on 1.33 project board after the docs PR was closed.

(cc Docs Lead: @rayandas, we might need to track the reopened Docs PR kubernetes/website#49822))

@pohly After reading the KEP, I have a couple of questions from the perspective of Node autoscaling. Do we expect the device taints to behave/be interacted with similarly to Node taints?

Cluster Autoscaler uses existing Nodes to reason about how a new Node from the same NodeGroup would look like. While doing that, it has to ignore Node taints that won't be present on a new Node. This includes well-known status/transient taints, but CA also allows the user to configure which taints are "startup" (can be present on every Node at the beginning, then are expected to disappear), and which are "status" (can be present on only some Nodes, new Nodes don't automatically get them). Any other taint will be copied to the NodeGroup template and prevent CA from scaling up for pods that don't tolerate it (even if the taint were to disappear later on).

Should we just extend these mechanisms and the general taint handling to device taints and essentially treat them the same as Node taints? Or are there any considerations that make device taints different from Node taints wrt the logic above?

For taints applied by the admin we probably should literally do what the admin says: if the DeviceTaintRule applies to all devices of a driver regardless of the pool, it still applies also to the fictional node.

For taints published by the driver it's as ambiguous as node taints. They may or may not be "startup", so I suppose a configuration option will be needed.

For taints applied by the admin we probably should literally do what the admin says: if the DeviceTaintRule applies to all devices of a driver regardless of the pool, it still applies also to the fictional node.

Are you saying we should treat some DeviceTaintRules as special cases? Can we easily detect if a DeviceTaintRule "applies to all devices of a driver regardless of the pool"?

FYI, I updated kubernetes/autoscaler#7947 accordingly.

I think Patrick is suggesting that in general, taints defined by DeviceTaintRules are more likely to be like "startup" taints (i.e. can be extrapolated to new nodes), whereas taints defined in ResourceSlices could very well be either "startup" or "status." There's nothing stopping some kind of controller from managing "status"-like taints in DeviceTaintRules though, so that's not a perfect heuristic.

One issue CA might have making that distinction though is that the ResourceSlice tracker doesn't record the origin of taints, so it's impossible to tell only by looking at a ResourceSlice from the tracker whether or not a taint came from a DeviceTaintRule. Maybe that's only a sign that "from DeviceTaintRule => 'startup'" isn't the right line to draw in the sand. IMO if some user configuration to make the startup/status distinction would be required for taints defined by drivers in ResourceSlices, that same configuration should also be required and apply the same way for taints in DeviceTaintRules.

Hi @pohly 👋, 1.34 Enhancements Lead here.

I am closing the v1.33 milestone now.

If you'd like to work on this enhancement in v1.34, please have the SIG lead opt-in by adding the lead-opted-in label, which ensures it gets added to the tracking board. Also, please set the milestone to v1.34 using /milestone v1.34.

Thanks!

/remove-label lead-opted-in
/remove-label tracked/yes
/milestone clear

@pohly is this targeting alpha or beta for 1.34?

This will remain in alpha for 1.34. No KEP update is planned for this cycle, but there might be some minor code PRs.

Hi, @pohly! To be clear, any code changes that introduce behavioral changes or user-facing changes, no matter how minor, still require tracking by the release team even if you will not be graduating to the next stage. Just making sure that's understood, since there was significant confusion around this in the last cycle. If they're refactors or bug fixes, that's totally fine and we don't need to track it.

In this case, "behavioral changes or user-facing changes" would imply updating the KEP upfront. I can't think of anything right now, so I suppose it's fine to not track it.