feat: embed distro-specific rendered script hotfixes in ANC

[Devinwong]

Impact of this PR

It changes the way we hotfix scripts, from "delivering the scripts in absvc", to "delivering only the ANC version pointer in absvc". Once this is merged and released in a weekly VHD, this becomes the script-hotfix delivery path for those VHDs. The legacy scripts_version disk-YAML application path remains compatible.

What this changes

This PR embeds fully rendered, distro-specific nodecustomdata YAML files in the ANC package instead of a JSON manifest plus separate payload files.

The hotfix generator selects changed write_files blocks from the traditional Linux nodecustomdata.yml path and renders them through AgentBaker's canonical Go-template functions. ANC selects the YAML matching the running distro, writes it to /opt/azure/containers/embedded-nodecustomdata.yml, and calls the existing applyNodeCustomData(path) function. The original applier is unchanged; there is no new transactional replacement engine. The YAML is retained for debugging without overwriting the legacy cloud-init nodecustomdata.yml.

Architecture

%%{init: {"themeVariables": {"fontSize": "18px"}}}%%
flowchart TB
    subgraph Legend[Legend]
        direction LR
        L1[New]:::new
        L2[Updated]:::updated
        L3[Existing]:::existing
    end

    A["Changed provisioning scripts"]:::existing --> B["hotfix_generate.py"]:::updated
    B --> C["Render Ubuntu and Azure Linux hotfix YAML"]:::new
    C --> D["Embed YAML in ANC package"]:::new
    D --> P["Publish ANC package to PMC"]:::existing
    E["LPS base-version map: primary selector"]:::existing --> G["Install selected ANC package"]:::existing
    F["ABsvc version pointer: fallback"]:::existing -.-> G
    P --> G
    G --> H["ANC selects distro YAML if present"]:::new
    H --> J["Existing applyNodeCustomData"]:::existing
    J --> L["Normal CSE provisioning"]:::existing
    H -. no payload .-> L
    J -. application error: log and continue .-> L

    classDef new fill:#dafbe1,stroke:#1a7f37,color:#0b1f0f,stroke-width:2px,font-size:18px
    classDef updated fill:#fff8c5,stroke:#9a6700,color:#24292f,stroke-width:2px,font-size:18px
    classDef existing fill:#ddf4ff,stroke:#0969da,color:#0b1f0f,stroke-width:2px,font-size:18px

Loading

Generated payload example

#cloud-config
write_files:
- path: /opt/azure/containers/provision_configs.sh
  permissions: "0744"
  encoding: gzip
  owner: root
  content: !!binary |
    H4sIAAAAAAAA/...

Each generated file is complete, template-free cloud-config. Conditional distro blocks are resolved by the same getBakerFuncMap and node-custom-data variables used by production rendering rather than reimplemented in Python.

Package selection and fallback

The LPS base-version map is the primary hotfix selector. ABsvc seeds the on-node configuration with a small single-version pointer so first-node provisioning still has a fallback when LPS is not ready. The new pointer carries no rendered script contents; those are stored in the corresponding PMC package, avoiding CustomData size growth.

The package version identifies the hotfix content for a VHD version base, so this path does not generate or depend on scripts_version. If package installation fails on a fresh node, provisioning can continue with the baked ANC. Embedded application errors are also fail-open, but do not undo script writes already completed.

Application behavior

• Reuses the original applyNodeCustomData parser, decoder, and sequential file writer.
• Retains /opt/azure/containers/embedded-nodecustomdata.yml for debugging, with mode 0600 on creation. It records the most recently written payload, including failed application attempts; the success log confirms completed application. Skipped applications leave any previously retained file untouched.
• Can create missing files and parent directories. Existing files use the original writer's permissions and symlink semantics.
• No staging, backups, transactional rollback, replace-only filtering, or applied/skipped counters. If an entry fails, earlier writes remain and provisioning continues.
• An absent distro YAML means no embedded script hotfix for that platform; other payload read errors are logged. No separate activation flag is needed.
• Ubuntu and standard Azure Linux are supported. Azure Linux retains the mariner payload filename; legacy ID=mariner is no longer supported. ACL, OS Guard, and Flatcar are explicitly skipped, including unsupported variants sharing the Azure Linux ID.
• The repository retains a README placeholder so go:embed builds without payloads; hotfix generation replaces it with rendered distro YAMLs.
• Distro selection does not reproduce non-distro template conditions such as custom-image exclusions. Hotfix authors must account for those applicability constraints separately.

Workflow behavior

The hotfix-generate workflow tracks the generated ANC YAML directory and hotfix version JSON. It no longer edits parts/linux/cloud-init/nodecustomdata.yml. Script selection is cumulative against the frozen VHD baseline. Script changes make the embedded ANC package a production change, causing the normal base-version patch bump and PMC package publication path. When no new script or ANC hotfix is detected, existing generated payloads and the version pointer are retained; retirement remains explicit.

Validation

• Canonical renderer and generator coverage from the earlier implementation.
• Focused Linux ANC tests for embedded selection/application, legacy disk application, provisioning ordering, fail-open, and dry-run behavior.
• Windows adapter and legacy application tests.
• Focused E2E fixture unit tests; scenario log assertion updated to the simplified success message.
• Changed ANC files pass golangci-lint.

The broader Windows provisioning tests encounter existing /bin/bash availability and temporary watcher cleanup failures; the corresponding targeted Linux run passes.

Standalone e2e validation

Before the application simplification, Ubuntu and Azure Linux 3 standalone runs passed: hotfix installed, scripts applied, node Ready. Those live runs have not been repeated for the simplified applier.

OS Guard's ordinary tdnf install failed before embedded application. OS Guard, Flatcar, and ACL are outside the supported embedded scope. Direct-HTTP delivery remains unvalidated.

[github-actions]

Windows Unit Test Results

  3 files   14 suites   53s ⏱️
415 tests 415 ✅ 0 💤 0 ❌
418 runs  418 ✅ 0 💤 0 ❌

Results for commit d7b3782.

♻️ This comment has been updated with latest results.

[Copilot]

🟡 Changes recommended

Critical stale-binary fallback and unsupported-script issues, plus payload completeness and durability concerns, remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Embeds distro-specific rendered Linux script hotfix payloads in ANC and applies them during provisioning.

Changes:

• Adds canonical AgentBaker rendering and hotfix generation.
• Adds transactional ANC payload validation and application.
• Adds unit, integration, E2E, documentation, and workflow coverage.

Required fixes:

• Critical — aks-node-controller/app.go:691: A failed package update can leave a previously staged hotfix binary selected, reapplying stale payloads instead of falling back to VHD scripts. Remove or validate stale binaries before launcher selection.
• Moderate — aks-node-controller/scripthotfix/applier.go:351: Apply file permissions before syncing, or sync again afterward, so executable modes are durable before commit.
• Moderate — hotfix/hotfix_generate.py:384: Track every requested key and fail generation when any mapped artifact lacks a selected write_files block.
• Critical — hotfix/hotfix_generate.py:83: initAKSCustomCloud lacks the canonical variable and write_files block required to generate its payload. Remove the mapping or add canonical support.

File summaries

File	Description
pkg/agent/nodecustomdata_render_test.go	Tests distro-aware rendering.
pkg/agent/baker.go	Exposes canonical rendering.
hotfix/render-nodecustomdata/main.go	Renders platform payloads.
hotfix/hotfix_generate.py	Generates ANC hotfix assets.
hotfix/hotfix_generate_test.py	Tests generation behavior.
e2e/vmss.go	Builds fixture-enabled ANC binaries.
e2e/vmss_test.go	Tests fixture generation.
e2e/types.go	Defines hotfix fixtures.
e2e/scenario_test.go	Adds Ubuntu hotfix E2E coverage.
aks-node-controller/scripthotfix/generated/rendered_nodecustomdata_ubuntu.yml	Adds the Ubuntu baseline payload.
aks-node-controller/scripthotfix/generated/rendered_nodecustomdata_mariner.yml	Adds the Mariner baseline payload.
aks-node-controller/scripthotfix/generated/rendered_nodecustomdata_flatcar.yml	Adds the Flatcar baseline payload.
aks-node-controller/scripthotfix/generated/rendered_nodecustomdata_azlosguard.yml	Adds the OS Guard baseline payload.
aks-node-controller/scripthotfix/generated/rendered_nodecustomdata_acl.yml	Adds the ACL baseline payload.
aks-node-controller/scripthotfix/generated/active	Marks the baseline payload inactive.
aks-node-controller/scripthotfix/applier.go	Validates and transactionally applies payloads.
aks-node-controller/scripthotfix/applier_test.go	Tests validation, application, and rollback.
aks-node-controller/README.md	Documents hotfix delivery.
aks-node-controller/app.go	Applies embedded payloads before provisioning.
aks-node-controller/app_test.go	Tests application and fail-open behavior.
.github/workflows/hotfix-generate.yml	Generates and commits hotfix artifacts.

Review details

• Files reviewed: 15/21 changed files
• Comments generated: 4
• Review effort level: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

It replaces a production provisioning-hotfix path, and the simplified end-to-end application flow has not been rerun live.

Review details

• Files reviewed: 20/23 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

The production provisioning path spans generation, package delivery, launcher selection, and script replacement, while the simplified live flow has not been rerun.

Review details

• Files reviewed: 20/21 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

It changes the production provisioning hotfix path and the simplified application flow has not received fresh live end-to-end validation.

Review details

• Files reviewed: 20/21 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Legacy hotfix fallback can be lost, and the exported renderer still panics on incomplete input.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 20/21 changed files
• Comments generated: 2
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Template-only hotfix changes can be silently omitted from generated ANC payloads.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 20/21 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

It changes production provisioning and hotfix delivery across supported VHDs, while the simplified application path has not received repeated live validation.

Review details

• Files reviewed: 20/21 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

It changes production node-provisioning and package-selection paths, while the simplified application flow has not received renewed live validation.

Review details

• Files reviewed: 20/21 changed files
• Comments generated: 0 new
• Review effort level: Balanced