Skip to content

fix: preserve sparse rootfs staging - #7920

Merged
lpcox merged 4 commits into
mainfrom
lpcox-sparse-rootfs-staging
Aug 31, 2026
Merged

fix: preserve sparse rootfs staging#7920
lpcox merged 4 commits into
mainfrom
lpcox-sparse-rootfs-staging

Conversation

@lpcox

@lpcox lpcox commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • copy the trusted rootfs snapshot with the preflight-resolved rsync --sparse
  • preserve sparse ext4 holes during writable guest preparation and per-run staging under /run
  • retain exclusive fs.copyFile(..., COPYFILE_EXCL) staging for binaries and attestation manifests
  • add focused coverage for sparse-copy routing, exact rsync arguments, and failure propagation
  • document the sparse-staging invariant

Root cause

The live-KVM run after #7894 copied a sparse ext4 rootfs through regular file-copy paths. The trusted snapshot and writable guest-preparation copies could materialize the rootfs holes before the final per-run staging step, multiplying the image's logical size into host storage and failing with ENOSPC before the VM could boot.

Security rationale

The fix reuses the already preflight-resolved, trusted rsync binary and passes --sparse with an argument terminator. Only rootfs copies use this path. Executables, kernels, supervisors, manifests, and bundles continue to use exclusive fs.copyFile staging, preserving the existing no-overwrite and trusted-snapshot protections.

Validation

  • npm test -- --runInBand src/cloud-hypervisor/preflight.test.ts src/cloud-hypervisor/manager.test.ts src/microvm/workspace.test.ts (59 tests)
  • npm run build
  • npm run lint
  • npm run lint:md
  • npm test -- --runInBand (5,374 tests)
  • bash -n scripts/ci/cloud-hypervisor-host-preflight.sh scripts/ci/cloud-hypervisor-live-smoke.sh guest/cloud-hypervisor/build-test-artifacts.sh guest/cloud-hypervisor/verify-test-artifacts.sh

Use trusted rsync for sparse rootfs copies. Avoid expanding ext4 holes into host disk exhaustion.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: b657421c-9b57-46c6-a32c-6012d90e5117
Copilot AI balanced review requested due to automatic review settings August 31, 2026 18:14
@lpcox lpcox added the cloud-hypervisor-kvm Trigger the Cloud Hypervisor live-KVM smoke/security suite label Aug 31, 2026
@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Documentation Preview

Documentation build failed for this PR. View logs.

Built from commit 1811ca3

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Security Guard has started processing this pull request

@lpcox lpcox changed the title fix(cloud-hypervisor): preserve sparse rootfs staging fix: preserve sparse rootfs staging Aug 31, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The production rootfs preparation path still performs an intermediate regular copy that can exhaust disk space before sparse staging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Balanced
Findings: 1 High severity · 1 Low severity

New issues introduced by this change (2)
Severity Finding
Low severity src/​cloud-hypervisor/​preflight.ts — The new helper's behavior is not exercised by the added tests: preflight mocks copySparseFile,…
High severity src/​cloud-hypervisor/​manager-start.ts — This sparse copy runs too late for the normal runtime path. The production backend always supplies…
What changed in this PR

Preserves sparse Cloud Hypervisor rootfs images during trusted snapshot and /run staging.

Changes:

  • Adds rsync-based sparse copying.
  • Wires sparse staging through preflight and manager dependencies.
  • Adds delegation tests and documents the invariant.
File Description
src/​cloud-hypervisor/​preflight.ts Adds sparse snapshot copying.
src/​cloud-hypervisor/​preflight.test.ts Tests preflight delegation.
src/​cloud-hypervisor/​manager.ts Registers the sparse copier.
src/​cloud-hypervisor/​manager.test.ts Tests manager delegation.
src/​cloud-hypervisor/​manager-types.ts Extends manager dependencies.
src/​cloud-hypervisor/​manager-start.ts Uses sparse /run staging.
src/​cloud-hypervisor/​diagnostics.ts Supports custom artifact copying.
docs/​cloud-hypervisor-foundation.md Documents sparse staging.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/cloud-hypervisor/preflight.ts
Comment thread src/cloud-hypervisor/manager-start.ts
@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric Base PR Delta
Lines 92.99% 92.97% 📉 -0.02%
Statements 91.72% 91.70% 📉 -0.02%
Functions 92.22% 92.18% 📉 -0.04%
Branches 85.31% 85.28% 📉 -0.03%
📁 Per-file Coverage Changes (5 files)
File Lines (Before → After) Statements (Before → After)
src/cloud-hypervisor/preflight.ts 85.8% → 83.9% (-1.90%) 85.2% → 83.3% (-1.83%)
src/cloud-hypervisor/manager-start.ts 96.0% → 96.0% (+0.05%) 96.2% → 96.2% (+0.05%)
src/cloud-hypervisor/manager.ts 88.6% → 88.7% (+0.09%) 87.4% → 87.5% (+0.10%)
src/cloud-hypervisor/diagnostics.ts 87.6% → 87.7% (+0.12%) 85.3% → 85.6% (+0.26%)
src/log-directory-setup.ts 96.2% → 100.0% (+3.78%) 96.3% → 100.0% (+3.71%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

@lpcox

lpcox commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator Author

@copilot address review feedback

Use the trusted sparse copier before guest customization.

Cover rsync arguments and failure behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

✅ Copilot review passed with no inline comments.

@lpcox Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Smoke Gemini reports failed. Facets need polishing...

💎 Faceted by Smoke Gemini

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

📡 OTel tracing validated by Smoke OTel Tracing

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

🔑 BYOK report filed by Smoke Copilot BYOK

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Smoke Claude passed

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by Smoke Claude for #7920

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • msfeed25.pkgs.visualstudio.com
  • registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "msfeed25.pkgs.visualstudio.com"
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

Tested by Smoke Chroot

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Docker Sbx has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Docker Sbx

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Copilot

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Contribution Check failed. Please review the logs for details.

Generated by Contribution Check for #7920

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Build Test Suite completed successfully!

Generated by Build Test Suite for #7920

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

🔌 Service connectivity validated by Smoke Services

@lpcox
lpcox deployed to aoai-model August 31, 2026 18:32 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Security Guard has started processing this pull request

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Docker Sbx has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Docker Sbx

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Contribution Check completed successfully!

Contribution check complete for PR #7920: no issues found against CONTRIBUTING.md. The PR includes tests for the new sparse-copy behavior, updates documentation, and the description is clear and references the related issue.

Generated by Contribution Check for #7920

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

🔌 Service connectivity validated by Smoke Services

@lpcox
lpcox deployed to aoai-model August 31, 2026 18:54 — with GitHub Actions Active
@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

🔑 BYOK report filed by Smoke Copilot BYOK

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

Tested by Smoke Chroot

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • msfeed25.pkgs.visualstudio.com
  • registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "msfeed25.pkgs.visualstudio.com"
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Build Test Suite completed successfully!

Generated by Build Test Suite for #7920

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Smoke Gemini reports failed. Facets need polishing...

💎 Faceted by Smoke Gemini

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Smoke Claude passed

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by Smoke Claude for #7920

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Security Guard has started processing this pull request

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot Network Isolation Egress@lpcox

EGRESS_RESULT allow=pass deny=pass

  • ✅ Allowed domain (github.com): allowed=200
  • ✅ Blocked domain (example.com): OK: example.com was blocked (CONNECT tunnel 403)

Overall status: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Docker Sbx@lpcox

Overall: PASS

📰 BREAKING: Report filed by Smoke Docker Sbx
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot Engine — PASS ✅

Overall: PASS

cc @lpcox

📰 BREAKING: Report filed by Smoke Copilot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Services Connectivity

  • Redis PING: ❌ (Temporary failure in name resolution)
  • PostgreSQL pg_isready: ❌ (no response)
  • PostgreSQL SELECT 1: ❌ (could not resolve host.docker.internal)

Overall: FAIL — DNS resolution for host.docker.internal is not available in the AWF sandbox.

🔌 Service connectivity validated by Smoke Services
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Copilot BYOK Smoke Test ✅

Test Results:

Status: PASS
Mode: Direct BYOK (COPILOT_PROVIDER_API_KEY via api-proxy sidecar)

@lpcox

🔑 BYOK report filed by Smoke Copilot BYOK
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Claude Engine Validation

Check Status
API ✅ PASS
gh CLI ✅ PASS
File ✅ PASS

Overall result: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by Smoke Claude for #7920 · claude · haiku45 · 56.1 AIC · ⊞ 4.5K ·
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Chroot Version Comparison Results

Runtime Host Version Chroot Version Match?
Python Python 3.12.14 Python 3.12.14 ✅ YES
Node.js v24.19.0 v2.97.0 ❌ NO
Go go1.22.12 go1.22.12 ✅ YES

Overall: FAILED — Node.js version mismatch between host and chroot environments (v24.19.0 vs v2.97.0). The smoke-chroot label was not added since not all tests passed.

Tested by Smoke Chroot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: API Proxy OTEL Tracing — Results

  • Scenario 1 (Module Loading):otel.js loaded successfully; isEnabled: true; exports startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled and helpers.
  • Scenario 2 (Test Suite): ✅ 68/68 tests passed (3 suites: otel, otel-serialization, otel-exporters).
  • Scenario 3 (Env Var Forwarding):env-passthrough.ts forwards GITHUB_AW_OTEL_TRACE_ID/GITHUB_AW_OTEL_PARENT_SPAN_ID; api-proxy-env-config.ts forwards GH_AW_OTLP_ENDPOINTS, OTEL_EXPORTER_OTLP_ENDPOINT, and trace context vars.
  • Scenario 4 (Token Tracker Integration):token-tracker-http.js contains the onUsage callback hook point.
  • Scenario 5 (OTEL Diagnostics): ✅ Spans were exported this run — otel.jsonl contains a gh-aw.agent.setup span with GenAI/workflow resource attributes.

Overall: All 5 scenarios passed. No unexpected failures detected.

📡 OTel tracing validated by Smoke OTel Tracing
Add label ready-for-aw to run again

@lpcox
lpcox deployed to aoai-model August 31, 2026 18:59 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

@lpcox
Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw)
feat: add durable identity-validated microVM cleanup: ❌
feat: isolate Cloud Hypervisor with per-run VMM identities: ❌
GitHub.com connectivity: ✅
Agent file write/read: ✅
BYOK inference path: ✅
Overall: FAIL

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

fix: preserve sparse rootfs staging
✅ MCP connectivity, ✅ GitHub.com connectivity, ✅ file write/read, ✅ direct BYOK inference
Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy 15 Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra
Overall: PASS
@lpcox

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke test: FAIL

PR titles:

  • fix(cloud-hypervisor): stabilize post-merge KVM checks
  • fix: preserve sparse rootfs staging

Checks: merged review ✅ | safeinputs-gh ❌ | playwright ✅ | file+cat ✅ | discussion comment ❌ | build ✅

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • msfeed25.pkgs.visualstudio.com
  • registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "msfeed25.pkgs.visualstudio.com"
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex
Add label ready-for-aw to run again

@lpcox
lpcox deployed to aoai-model August 31, 2026 19:00 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia 1/1 passed ✅ PASS
Bun hono 1/1 passed ✅ PASS
C++ fmt N/A ✅ PASS
C++ json N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world N/A (ran successfully) ✅ PASS
.NET json-parse N/A (ran successfully) ✅ PASS
Go color ok ✅ PASS
Go env ok ✅ PASS
Go uuid ok ✅ PASS
Java gson 1/1 passed ✅ PASS
Java caffeine 1/1 passed ✅ PASS
Node.js clsx All passed ✅ PASS
Node.js execa All passed ✅ PASS
Node.js p-limit All passed ✅ PASS
Rust fd 1/1 passed ✅ PASS
Rust zoxide 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — PASS

Notes:

  • Java (gson, caffeine) required overriding Maven's local repo path (-Dmaven.repo.local=/tmp/.../m2repo) because ~/.m2/repository was not writable in this environment; otherwise builds followed the documented proxy config and completed successfully.
  • All repositories cloned successfully; no CLONE_FAILED cases occurred.

Generated by Build Test Suite for #7920 · copilot · auto · 39.4 AIC · ⊞ 12K ·
Add label ready-for-aw to run again

@lpcox
lpcox merged commit 17c65d4 into main Aug 31, 2026
165 of 167 checks passed
@lpcox
lpcox deleted the lpcox-sparse-rootfs-staging branch August 31, 2026 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants