Repository navigation
fix: update liveness probe to avoid clash - #9412
Tim Wright (timmy-wright) wants to merge 8 commits into
Conversation
github-actions
Bot
added
the
components
Windows Unit Test Results 3 files 14 suites 52s ⏱️ Results for commit 99d7a8d. ♻️ This comment has been updated with latest results. |
There was a problem hiding this comment.
🔵 Needs a closer look
Cross-platform VHD build checks are still running, and the Windows cache change requires domain-owner validation.
Pull request overview
Updates cached CSI livenessprobe images to replace the Windows-breaking v2.19.0 image.
Changes:
- Advances the primary image to v2.20.0.
- Replaces v2.18.0 with the MCR-specific v2.19.0-6 tag.
- Applies both updates consistently across Linux multi-arch and Windows VHDs.
File summaries
| File | Description |
|---|---|
parts/common/components.json |
Updates cached livenessprobe image versions for Linux and Windows VHD builds. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
🟡 Changes recommended
The resulting component lists do not include the stated v2.20.0 update or retain v2.19.0 in the intended previous-version slots.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Balanced
There was a problem hiding this comment.
🔵 Needs a closer look
The active Renovate entries can re-propose the malformed image revisions that this mitigation avoids.
Review details
Suppressed comments (2)
Previously missed (1) — in code that hasn't changed since the last review.
parts/common/components.json:634
- 🟡 Medium Risk — 📦 Package Update: These are safety pins, but their registry
renovateTags remain active. The custom manager in.github/renovate.json:742-755will parse these entries, and theoss/v2/**versioning rule at lines 39-44 treats the numeric suffix as an updateable revision, so Renovate can immediately propose the known-bad-6/-11tags. Disable Renovate for these Windows-only pins until fixed images are published and validated.
This issue also appears on line 654 of the same file.
parts/common/components.json:658
- 🟡 Medium Risk — 📦 Package Update: These pins are still managed by Renovate. With the repository's suffix-aware
oss/v2/**rule, Renovate can proposev2.17.0-6andv2.16.0-11, which this PR identifies as malformed, reintroducing the Windows VHD build failure if that update is accepted. Use the repository's<DO_NOT_UPDATE>marker until validated republished revisions are available.
"latestVersion": "v2.17.0-5"
},
{
"renovateTag": "registry=https://mcr.microsoft.com, name=oss/v2/kubernetes-csi/csi-node-driver-registrar",
"latestVersion": "v2.16.0-10"
- Files reviewed: 1/1 changed files
- Comments generated: 0 new
- Review effort level: Balanced
🏷️ - Generated by Copilot
There was a problem hiding this comment.
🟡 Changes recommended
Automatic suffix stripping creates unintended aliases for unrelated cached Windows images.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 5/5 changed files
- Comments generated: 1
- Review effort level: Balanced
🏷️ - Generated by Copilot
There was a problem hiding this comment.
🟡 Changes recommended
Renovate remains enabled for all four tactical pins and can restore the known-malformed revisions.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (3)
parts/common/components.json:635
- 🟡 Medium Risk — 📦 Package Update: This pin remains Renovate-managed, so the custom container-image manager can advance
v2.18.0-10to the known-badv2.18.0-11and reintroduce the VHD build failure. Mark this entry as not updatable for the mitigation window, and restore its registry metadata after a corrected image is republished.
"latestVersion": "v2.18.0-10",
parts/common/components.json:656
- 🟡 Medium Risk — 📦 Package Update: The active Renovate metadata allows this safe registrar pin to be advanced to the known-malformed
v2.17.0-6revision. That would undo the mitigation and make Windows VHD builds fail again; disable Renovate for this entry until the fixed republish is available.
"latestVersion": "v2.17.0-5",
parts/common/components.json:661
- 🟡 Medium Risk — 📦 Package Update: Keeping this entry under Renovate permits the next registrar revision (
v2.16.0-11), which the PR identifies as carrying the same malformed Windows base layers. Disable automated updates for the tactical pin and re-enable them only after the corrected image is published.
"latestVersion": "v2.16.0-10",
- Files reviewed: 6/6 changed files
- Comments generated: 1
- Review effort level: Balanced
🔄 - Generated by Copilot
Changes cached containers or packages on windows VHDsPlease get a Windows SIG member to approve. The following dif file shows any additions or deletions from what will be cached on windows VHDs organised by VHD type.
diff --git a/vhd_files/2022-containerd-gen2.txt b/vhd_files/2022-containerd-gen2.txt
index 4e05767..93a9759 100644
--- a/vhd_files/2022-containerd-gen2.txt
+++ b/vhd_files/2022-containerd-gen2.txt
@@ -122,0 +123 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.16.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.16.0-10
@@ -123,0 +125 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.17.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.17.0-5
@@ -124,0 +127 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.18.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.18.0-10
@@ -125,0 +129 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.19.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.19.0-5
diff --git a/vhd_files/2022-containerd.txt b/vhd_files/2022-containerd.txt
index a8c76d7..360d4fc 100644
--- a/vhd_files/2022-containerd.txt
+++ b/vhd_files/2022-containerd.txt
@@ -122,0 +123 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.16.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.16.0-10
@@ -123,0 +125 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.17.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.17.0-5
@@ -124,0 +127 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.18.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.18.0-10
@@ -125,0 +129 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.19.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.19.0-5
diff --git a/vhd_files/2025-gen2-tl.txt b/vhd_files/2025-gen2-tl.txt
index e49ede0..ded7b62 100644
--- a/vhd_files/2025-gen2-tl.txt
+++ b/vhd_files/2025-gen2-tl.txt
@@ -53,0 +54 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.16.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.16.0-10
@@ -54,0 +56 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.17.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.17.0-5
@@ -55,0 +58 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.18.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.18.0-10
@@ -56,0 +60 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.19.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.19.0-5
diff --git a/vhd_files/2025-gen2.txt b/vhd_files/2025-gen2.txt
index eec1d89..4a097fe 100644
--- a/vhd_files/2025-gen2.txt
+++ b/vhd_files/2025-gen2.txt
@@ -53,0 +54 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.16.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.16.0-10
@@ -54,0 +56 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.17.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.17.0-5
@@ -55,0 +58 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.18.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.18.0-10
@@ -56,0 +60 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.19.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.19.0-5
diff --git a/vhd_files/2025.txt b/vhd_files/2025.txt
index 5997647..d01c375 100644
--- a/vhd_files/2025.txt
+++ b/vhd_files/2025.txt
@@ -53,0 +54 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.16.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.16.0-10
@@ -54,0 +56 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.17.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.17.0-5
@@ -55,0 +58 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.18.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.18.0-10
@@ -56,0 +60 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.19.0
+mcr.microsoft.com/oss/v2/kubernetes-csi/livenessprobe:v2.19.0-5 |
There was a problem hiding this comment.
🔵 Needs a closer look
The four tactical pins must opt out of Renovate to prevent automated updates from invalidating their fixed aliases.
Review details
Suppressed comments (4)
Previously missed (1) — in code that hasn't changed since the last review.
parts/common/components.json:636
- 🔴 High Risk — 📦 Package Update: This known-good revision is still monitored by Renovate, so a later automated update can change the source while leaving the
v2.18.0alias unchanged, defeating the mitigation. Mark the pinned entry<DO_NOT_UPDATE>perparts/linux/cloud-init/artifacts/README-COMPONENTS.md:171-172.
This issue also appears in the following locations of the same file:
- line 656
- line 661
parts/common/components.json:631
- 🔴 High Risk — 📦 Package Update: This is intended to be a known-good pin, but its recognized registry
renovateTagleaves it eligible for automated replacement while the alias remainsv2.19.0. That can silently point the normal tag at a different build revision and reintroduce the malformed-layer failure. Use<DO_NOT_UPDATE>as prescribed for pinned versions inparts/linux/cloud-init/artifacts/README-COMPONENTS.md:171-172.
"latestVersion": "v2.19.0-5",
"additionalTagsToApplyToContainer": ["v2.19.0"]
parts/common/components.json:657
- 🔴 High Risk — 📦 Package Update: Keeping the recognized registry
renovateTagmakes this tactical pin eligible for automated replacement even though the alias remainsv2.17.0. A future update could therefore cache an unverified revision under the normal tag; use<DO_NOT_UPDATE>as documented inparts/linux/cloud-init/artifacts/README-COMPONENTS.md:171-172.
"latestVersion": "v2.17.0-5",
"additionalTagsToApplyToContainer": ["v2.17.0"]
parts/common/components.json:662
- 🔴 High Risk — 📦 Package Update: This pinned registrar revision remains Renovate-managed, allowing automation to replace its source independently of the fixed
v2.16.0alias and undo the known-good mapping. SetrenovateTagto<DO_NOT_UPDATE>perparts/linux/cloud-init/artifacts/README-COMPONENTS.md:171-172.
"latestVersion": "v2.16.0-10",
"additionalTagsToApplyToContainer": ["v2.16.0"]
- Files reviewed: 6/6 changed files
- Comments generated: 0 new
- Review effort level: Balanced
AgentBaker Linux gate detectiveRun: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=180205822 TL;DR: AzureLinuxV3 CSE cached performance failed narrowly because AKS.CSE.ensureContainerd took 2.972s against the 2s threshold; CSE itself exited successfully. Likely cause / signature: $signature - threshold-only CSE timing breach, most consistent with runtime startup/package latency or a tight threshold rather than a broad provisioning failure. Confidence: Medium Recommended owner/action: Node Lifecycle/CSE owner should inspect ensureContainerd timing variance and either reduce latency or adjust the threshold/flake handling. Strongest alternative: A PR-caused Windows image regression is less likely because the changed commits are Windows-focused while the failure is isolated to AzureLinux CSE timing. Evidence: timeline failed RunAgentBakerE2E; failed test result; CSE timing report shows ensureContainerd 2.972s > 2s with ExitCode 0 and VM logs extracted. Wiki signature: cse-ensurecontainerd-duration-threshold |
Mutable Windows variants of the CSI livenessprobe and node-driver-registrar images currently resolve to revisions with an incompatible Windows base-layer archive layout. This causes Windows VHD builds to fail while containerd processes the base layer.
What this PR does / why we need it:
livenessprobe:v2.19.0tov2.19.0-5.livenessprobe:v2.18.0tov2.18.0-10.csi-node-driver-registrar:v2.17.0tov2.17.0-5.csi-node-driver-registrar:v2.16.0tov2.16.0-10.additionalTagsToApplyToContainerto Windows component versions so a pulled image can be cached under explicitly configured aliases.k8s.ionamespace after the build-specific image is pulled.The build-specific revisions remain the only images pulled from the registry. The additional tags point to those already-cached images:
v2.18.0-10v2.18.0v2.19.0-5v2.19.0v2.16.0-10v2.16.0v2.17.0-5v2.17.0This allows pods referencing the normal tag to use the known-good image cached in the VHD instead of downloading the broken mutable revision when local image reuse is permitted. Alias creation uses the existing Windows image selection, SKU filtering, and retry behavior. Additional tags are opt-in and are only applied to the
latestVersionentry that declares them.The mutable tags currently resolve as follows:
v2.18.0v2.18.0-11v2.18.0-10v2.19.0v2.19.0-6v2.19.0-5v2.16.0v2.16.0-11v2.16.0-10v2.17.0v2.17.0-6v2.17.0-5All affected revisions share the same malformed Windows root layers for each Windows version. Unlike the previous revisions, the affected root archives place the operating-system directories directly at the archive root:
Containerd imports Windows layers through
ociwclayer.ImportLayerFromTar. The importer preserves the tar entry names and, for a parentless layer, callshcsshim::ProcessBaseLayer. Because the affected archives do not create the requiredFiles/directory, base-layer processing fails withERROR_PATH_NOT_FOUND. The missingUtilityVM/payload is another consequence of the repackaging, but it is not the cause of this extraction failure.The failing Windows VHD build reproduced the livenessprobe failure on two containerd versions:
1.6.35-azure.1sha256:66545f5a12285a70850304d489f0c3065552f68bcdc3199a7c74bab49d60f44c2.0.4-azure.1sha256:b3a59411a9182db2a9656bce00537904c2f1b7d71cbc9782f5155263f1b6336cRetries failed against new snapshot directories with the same base-layer DiffID, confirming this was not a transient download or snapshot failure. The Windows Server 2019 variants have the same incompatible archive structure and are also affected, although that platform was not exercised by the cited build.
The node-driver-registrar mutable tags use the exact same malformed root-layer digests as the affected livenessprobe revisions:
sha256:06aaa22f843ee9448c25ab50b08937c2754b1d50591e94f898df0fef9da96c2bsha256:a20336801d45e01082ccf9e7a799da2901e008ffc51d7e170f685c8102ea2bfcsha256:4ec983932fc2c7e85f176f52ab1926d588a4323a2c808e9cc1b61c9add15eaafThe publishing regression was caused by the DALEC frontend applying build-time source filtering to Windows base images. Re-materializing a Windows base through
llb.Scratch()loses the special Windows layer semantics and produces the flattened archive. The upstream fix is project-dalec/dalec#1228, with the v0.22 backport in project-dalec/dalec#1229 and release v0.22.1.This PR is a tactical mitigation for Windows VHD builds until the affected CSI images are republished with the fixed DALEC frontend.
Which issue(s) this PR fixes:
None.