/sig node

Discussion Link: N/A (or... at multiple conferences during the last years when presenting CRIU and container migration, there was always the question when will we see container migration in Kubernetes)

Responsible SIGs: maybe node

We recommend actively socializing your KEP with the appropriate sig to gain visibility, consensus and also for scheduling. Also as you are not sure of what SIG will sponsor this, reaching out to the SIGs to get clarity on that will be helpful to move your KEP forward.

Hi @adrianreber

Any updates on whether this will be included in 1.20?

Enhancements Freeze is October 6th and by that time we require:

The KEP must be merged in an implementable state
The KEP must have test plans
The KEP must have graduation criteria
The KEP must have an issue in the milestone

Best,
Kirsten

Hello @kikisdeliveryservice

Any updates on whether this will be included in 1.20?

Sorry, but how would I decide this? There has not been a lot of feedback on the corresponding KEP which makes it really difficult for me to answer that question. On the other hand, maybe the missing feedback is a good sign that it will take some more time. So probably this will not be included in 1.20.

Normally the sig would give a clear signal that it would be included. That would be by : reviewing the KEP, agreeing to the milestone proposals in the KEP etc.. I'd encourage you to keep in touch with them and start the 1.21 conversation early if this does not end up getting reviewed/merged properly by October 6th.

Best,
Kirsten

@kikisdeliveryservice Thanks for the guidance. Will do.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

@fejta-bot: Closing this issue.

/reopen
/remove-lifecycle rotten

@adrianreber: Reopened this issue.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

/remove-lifecycle stale

Still working on it.

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

First attempt to provide checkpoint via kubectl: kubernetes/kubernetes#120898

Hi @adrianreber, I created a simple microservice pod and tried to migrate it. I found that when I just started it and used a counter-like function, I was able to checkpoint it, but when I used it to connect to the message broker and send the message, checkpointing it will raise the following error, is there any way to solve it?

checkpointed: checkpointing of default/order-service-7c69b4d88b-n56xq/order-service failed
(rpc error: code = Unknown desc = failed to checkpoint container b0ae35f30db3124341035041d02ce85bd83be2b38180081ab919a9f89e16c3af:

running "/usr/local/bin/runc"
["checkpoint"
"--image-path" "/var/lib/containers/storage/overlay-containers/b0ae35f30db3124341035041d02ce85bd83be2b38180081ab919a9f89e16c3af/userdata/checkpoint"
"--work-path" "/var/lib/containers/storage/overlay-containers/b0ae35f30db3124341035041d02ce85bd83be2b38180081ab919a9f89e16c3af/userdata"
"--leave-running" "b0ae35f30db3124341035041d02ce85bd83be2b38180081ab919a9f89e16c3af"]

failed: /usr/local/bin/runc --root /run/runc --systemd-cgroup checkpoint --image-path /var/lib/containers/storage/overlay-containers/b0ae35f30db3124341035041d02ce85bd83be2b38180081ab919a9f89e16c3af/userdata/checkpoint --work-path /var/lib/containers/storage/overlay-containers/b0ae35f30db3124341035041d02ce85bd83be2b38180081ab919a9f89e16c3af/userdata --leave-running b0ae35f30db3124341035041d02ce85bd83be2b38180081ab919a9f89e16c3af

failed: time="2023-10-24T16:40:11Z"
level=error
msg="criu failed: type NOTIFY errno 0\nlog file: /var/lib/containers/storage/overlay-containers/b0ae35f30db3124341035041d02ce85bd83be2b38180081ab919a9f89e16c3af/userdata/dump.log"

@Tobeabellwether Please open a bug at CRI-O with the dump.log attached.

@Tobeabellwether Please open a bug at CRI-O with the dump.log attached.

@adrianreber Thanks for the tip, I checked that dump.log file and found:

(00.134562) Error (criu/sk-inet.c:191): inet: Connected TCP socket, consider using --tcp-established option.
(00.134634) ----------------------------------------
(00.134654) Error (criu/cr-dump.c:1669): Dump files (pid: 1533879) failed with -1

So, I tried to forcefully interrupt the TCP connection between the pod and the message broker, and the checkpoint was successfully created. Should this still be considered a bug?

Ah, good to know. No, this is not a real bug. We probably at some point need the ability to pass down different parameters from Kubernetes to CRIU. But this is something for the far future.

You can also control CRIU options with a CRIU configuration file. Handling TCP connections that are established could be configured there.

Ah, good to know. No, this is not a real bug. We probably at some point need the ability to pass down different parameters from Kubernetes to CRIU. But this is something for the far future.

You can also control CRIU options with a CRIU configuration file. Handling TCP connections that are established could be configured there.

Hi @adrianreber again, when I try to restore the checkpoint of the pod with TCP connection on a new pod, I encounter a new problem:

So I try to check the log file under /var/run/containers/storage/overlay-containers/order-service/userdata/restore.log but I only found those folders, no one with the name of the container to restore:

and the restoration for pods without TCP connections works fine.

@Tobeabellwether You can redirect the CRIU log file to another file using the CRIU configuration file: log-file /tmp/restore.log and have a look at that file.

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

/remove-lifecycle stale

/stage beta
/milestone v1.30
/label lead-opted-in

Hello @adrianreber 👋, v1.30 Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 9th February 2024.

This enhancement is targeting for stage beta for v1.30 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.30. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here).

Everything is done in #4288 and #4305. Please make sure that these PRs are merged before the enhancements freeze.

The status of this enhancement is marked as At risk for enhancement freeze currently. Please make sure your PRs are merged in time.

Hello 👋, v.130 Enhancements team here.

Unfortunately, this enhancement did not meet requirements for enhancements freeze.

#4288 is merged, but #4305 is still open. Please file an exception request to get this PR merged.

If you still wish to progress this enhancement in v1.30, please file an exception request. Thanks!

/milestone clear

Ah, good to know. No, this is not a real bug. We probably at some point need the ability to pass down different parameters from Kubernetes to CRIU. But this is something for the far future.
You can also control CRIU options with a CRIU configuration file. Handling TCP connections that are established could be configured there.

Hi @adrianreber again, when I try to restore the checkpoint of the pod with TCP connection on a new pod, I encounter a new problem:

So I try to check the log file under `/var/run/containers/storage/overlay-containers/order-service/userdata/restore.log` but I only found those folders, no one with the name of the container to restore: and the restoration for pods without TCP connections works fine.

I have the same problem, have you solved it?

Ah, good to know. No, this is not a real bug. We probably at some point need the ability to pass down different parameters from Kubernetes to CRIU. But this is something for the far future.
You can also control CRIU options with a CRIU configuration file. Handling TCP connections that are established could be configured there.

Hi @adrianreber again, when I try to restore the checkpoint of the pod with TCP connection on a new pod, I encounter a new problem:

So I try to check the log file under /var/run/containers/storage/overlay-containers/order-service/userdata/restore.log but I only found those folders, no one with the name of the container to restore:

and the restoration for pods without TCP connections works fine.

I have the same problem, have you solved it?

I've read CRIU's doc, I think checkpointing Container with data in and out is in general tricky and not safe, so I just always close its connection before I checkpointing.