[Dawei Wei (rzlink)]

Description

Fix in-place container restarts for Hyper-V isolated Windows pods (runhcs-wcow-hypervisor), e.g. with ContainerRestartRules / RestartAllContainers. Today a workload container restart can tear down the shared pod UVM, killing the sandbox and every other container in the pod. This is the failure reported in kubernetes/kubernetes#140187.

Root cause

A Windows guest holds the WaitForProcess response of a process until the host closes the process's stdio relay. A descendant process (for example the child of sh -c "...; sleep ...") can keep the relay open after the process itself has exited, while the host only closes the relay once the wait completes, so neither side makes progress. The SIGKILL that follows returns hrNotFound, which was logged and ignored, leaving the wait pending. The 30s SIGKILL watchdog in KillExec then closed the UVM, even when the task did not own it.

Fix

1. Gate the KillExec SIGKILL watchdog on ownsHost, so only a task that owns its UVM can close it. A workload container shares the pod UVM.
2. When a Windows guest reports a process missing while its wait is still pending, wait a grace period for the real exit, then close the host side of the process's stdio relay. The guest then delivers the real exit code; no exit is synthesized and the bridge protocol is unchanged. The grace period (1s) matches the default CopyAfterExitTimeout the host already allows stdio relays after a process exits, and the path is only taken on the abnormal hrNotFound path.

Linux guests (LCOW) are excluded: they drain and close the relay themselves before answering the wait, and reconnect if the host closes it first. Process-isolated containers don't use this code path.

Testing

• go test ./internal/gcs ./cmd/containerd-shim-runhcs-v1, including new tests that:
  • the real exit code is kept and stdio stays open when the real exit arrives within the grace period;
  • stdio is closed to release the real exit when the guest holds it;
  • a Linux guest's stdio relay is never closed.
• On a CAPZ Hyper-V cluster, ran the four RestartAllContainers e2e specs that fail today 10 times each: 40/40 passed (0/4 before). With ETW (ContainerPlatform) traces: no watchdog, no UVM close/terminate during the tests; every hrNotFound was followed by the stdio close after ~1s and the real exit (0xC000013A) within a few ms, which containerd reported as the container's exit status.
• 24 direct CRI stop probes on Hyper-V pods (sh -c with a child, sh -c with redirect / exec, direct process, grace 0/2): sandbox and sibling containers stayed running and the UVM was not closed in every case; real exit codes were reported.

[Hamza El-Saawy (helsaawy)]

Dawei Wei (@rzlink)
what is the context for restarting containers? that is not exactly a supported OCI/containerd scenario?
also, are you seeing that exec signal/kill takes longer than 30seconds?

[Dawei Wei (rzlink)]

Dawei Wei (@rzlink) what is the context for restarting containers? that is not exactly a supported OCI/containerd scenario? also, are you seeing that exec signal/kill takes longer than 30seconds?

Thanks. This is for Kubernetes RestartAllContainersOnContainerExits, tracked in kubernetes/kubernetes#140187.
Kubelet implements the in-place restart using standard CRI stop/remove/create/start operations while preserving the pod sandbox. The process-isolated Windows tests pass; only Hyper-V isolation fails.
The signal call itself does not take 30 seconds. It returns  hrNotFound  because the process is already gone, but the outstanding WaitForProcess remains pending. Consequently,  e.Wait()  never completes and the existing 30-second watchdog fires.
For a workload task, ht.host is the shared pod UVM and ht.ownsHost is false. The watchdog currently closes that UVM, terminating the sandbox and sibling containers. This change completes the stale wait and prevents non-owning workload tasks from closing the shared UVM.

[Dawei Wei (rzlink)]

I've reworked the fix based on further investigation and squashed it into a single commit; the description is updated.

To answer the earlier question: yes, the exec SIGKILL hangs well past 30s. The container runs sh -c "...; sleep ...". On stop, sh exits but its orphaned sleep child keeps the stdio relay open. The Windows guest holds the WaitForProcess response until the host closes the relay, and the host only closes it after the wait completes, so the wait never returns. The SIGKILL then gets hrNotFound, and after 30s the KillExec watchdog calls ht.host.Close() on the shared pod UVM. With only the ownsHost gate, the container stays hung indefinitely instead.

Changes from the previous revision:

• Removed the forced wait completion, the synthesized exit, the bridge tolerance for unmatched WaitForProcess responses, and the closeHost change.
• On hrNotFound with a pending wait (Windows guests only), wait 1s (the default CopyAfterExitTimeout), then close the host side of the stdio relay. The guest then returns the real exit code (0xC000013A) within a few ms.
• Kept the ownsHost gate on the watchdog.

Validated on a CAPZ Hyper-V cluster with ETW traces: the 4 RestartAllContainers e2e specs passed 40/40 (0/4 before), with no watchdog firing and no UVM close.

[Justin (jterry75)]

Dawei Wei (@rzlink) - I'm not sure I fully grasp this yet. So your goal is to still allow the signal to be sent to the init process, but not to allow the closure unless the target was the init process?

The logic here is a bit convoluted IMO. Isn't your goal that we would not tear down the init container at all?

[Dawei Wei (rzlink)]

Dawei Wei (@rzlink) - I'm not sure I fully grasp this yet. So your goal is to still allow the signal to be sent to the init process, but not to allow the closure unless the target was the init process?

The logic here is a bit convoluted IMO. Isn't your goal that we would not tear down the init container at all?

Justin (@jterry75) - Yes, if by "init container" you mean the pod sandbox (pause) task that owns the UVM: the goal is that stopping one workload container never tears down the sandbox / shared UVM. Today, when a workload container's exit isn't reported within 30s after SIGKILL, the watchdog closes that shared UVM and takes down the whole pod.

The ownsHost gate alone isn't enough. The underlying problem is a deadlock that the watchdog currently "breaks" by killing the UVM. With only the gate, nothing breaks it and the container hangs forever. The deadlock happens when the container runs  sh -c "...; sleep ..." :

1. SIGTERM (CtrlShutdown) kills sh, but its orphaned sleep child keeps the container's stdio relay open.
2. The Windows guest holds the WaitForProcess response until the host closes the stdio relay.
3. The host only closes the stdio relay after WaitForProcess returns.
4. Neither side makes progress, so the exit is never reported. The SIGKILL then gets hrNotFound (sh is already gone), and 30s later the watchdog fires.

The process.go change breaks the deadlock: on hrNotFound with a pending wait, it closes the host side of the stdio relay after 1s, the guest returns the real exit code ( 0xC000013A ), and the watchdog no longer fires.

[Dawei Wei (rzlink)]

Justin (@jterry75) - I also have a standalone repro that doesn't need Kubernetes: one script provisions a throwaway Azure VM with containerd, runs a two-container Hyper-V pod, stops one container the way kubelet does, and checks whether the other one survives. With the current shim the pod UVM dies; Happy to share it ([gist link] / ping me on Teams).