fix: stabilize post-merge KVM checks by lpcox · Pull Request #7922 · github/gh-aw-firewall

lpcox · 2026-08-31T18:49:51Z

Summary

retry virtiofsd sandbox worker acquisition when a transient setup worker exits during /proc identity capture or cgroup assignment
keep executable and same-process identity mismatches fail-closed
use a bounded production-scale API readiness timeout in the coverage-instrumented manager fixture

Context

PR #7896 was merged while its final coverage and live-KVM runs were still completing. The coverage run exposed a 1 ms test-only readiness race, and live-KVM run 33425529718 observed a virtiofsd child disappear between discovery and /proc/<pid>/stat capture.

Validation

npm test -- --runInBand — 335 suites, 5,428 tests passed
npm run build
npm run lint — zero errors
focused virtiofsd/manager coverage-instrumented test passed
bash -n scripts/ci/cloud-hypervisor-live-smoke.sh

Follow-up to #7896. Do not merge until live-KVM completes successfully.

Persist identity-validated Cloud Hypervisor cleanup records. Safely reap stale resources after abrupt owner death. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Capture executable inode identity from the live procfs object. Verify firewall rule absence after deletion. Cover cleanup races at base coverage levels. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 92d2c346-d396-40d3-8b74-1f1447a04871

Resolve Cloud Hypervisor lifecycle overlaps. Preserve durable cleanup and current runtime confinement. Assign the VMM cgroup before durable identity capture. This avoids a PID-reuse window. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 92d2c346-d396-40d3-8b74-1f1447a04871

Ensure VMM cgroup assignment precedes durable process identity capture. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 92d2c346-d396-40d3-8b74-1f1447a04871

Reconcile durable identity-validated cleanup with atomic network reservations, current virtiofsd worker confinement, and per-run firewall ownership. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 92d2c346-d396-40d3-8b74-1f1447a04871

…-cleanup

Count gh alongside Docker and the eleven Cloud Hypervisor runtime tools. This keeps preflight assertions aligned with manifest attestation verification. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 92d2c346-d396-40d3-8b74-1f1447a04871

Reuse the backend-owned attested snapshot during workflow startup and every boot retry. This prevents duplicate rootfs copies from exhausting the runner's /run filesystem. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 92d2c346-d396-40d3-8b74-1f1447a04871

Match the live virtiofsd sandbox state while retaining exact effective and permitted capability checks. Reject capabilities reacquirable across exec. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 92d2c346-d396-40d3-8b74-1f1447a04871

Merge current origin/main without rebasing and preserve durable cleanup across dedicated accounts, device ACLs, network reservations, and artifact snapshots. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 92d2c346-d396-40d3-8b74-1f1447a04871

Use a bounded production-scale API readiness budget so coverage instrumentation cannot race the mocked socket probe. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 92d2c346-d396-40d3-8b74-1f1447a04871

Acquire worker identity, cgroup membership, and same-process validation as one retryable operation when a setup worker exits during procfs inspection. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 92d2c346-d396-40d3-8b74-1f1447a04871

…-cleanup

github-actions · 2026-08-31T18:50:44Z

🚀 Security Guard has started processing this pull request

Copilot

Copilot review overview

🟡 Changes recommended

Broad ENOENT handling masks cgroup loss, and the cgroup-assignment retry path lacks coverage.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Balanced
Findings: 1 · 1

New issues introduced by this change (2)

Severity	Finding
	`src/cloud-hypervisor/virtiofsd-sandbox.ts` — The newly supported cgroup-assignment race is not exercised by the added test: that test throws…
	`src/cloud-hypervisor/virtiofsd-sandbox.ts` — `ENOENT` is handled here as if the candidate exited regardless of which operation failed. However,…

What changed in this PR

Stabilizes Cloud Hypervisor worker discovery and readiness tests against transient process and timing races.

Changes:

Retries transient virtiofsd worker exits during identity capture and cgroup assignment.
Adds worker-exit regression coverage.
Uses a realistic readiness timeout in the manager fixture.

File	Description
`src/cloud-hypervisor/virtiofsd-sandbox.ts`	Moves worker validation and assignment into the retry loop.
`src/cloud-hypervisor/virtiofsd.test.ts`	Tests worker exit during identity capture.
`src/cloud-hypervisor/manager.test.ts`	Increases the fixture readiness timeout.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-08-31T18:55:08Z

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric	Base	PR	Delta
Lines	92.95%	92.93%	📉 -0.02%
Statements	91.60%	91.57%	📉 -0.03%
Functions	92.07%	92.07%	➡️ +0.00%
Branches	85.37%	85.30%	📉 -0.07%

📁 Per-file Coverage Changes (2 files)

File	Lines (Before → After)	Statements (Before → After)
`src/cloud-hypervisor/virtiofsd-sandbox.ts`	100.0% → 95.9% (-4.09%)	99.3% → 93.6% (-5.62%)
`src/log-directory-setup.ts`	96.2% → 100.0% (+3.78%)	96.3% → 100.0% (+3.71%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Retry assignment only for ESRCH or when procfs proves the candidate exited. Preserve cgroup ENOENT while the worker remains alive. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 92d2c346-d396-40d3-8b74-1f1447a04871

github-actions · 2026-08-31T19:05:39Z

✅ Copilot review passed with no inline comments.

@lpcox Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

github-actions · 2026-08-31T19:13:59Z

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

📡 OTel tracing validated by Smoke OTel Tracing

github-actions · 2026-08-31T19:14:00Z

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

Tested by Smoke Chroot

github-actions · 2026-08-31T19:14:01Z

📰 VERDICT: Smoke Docker Sbx has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Docker Sbx

github-actions · 2026-08-31T19:14:02Z

🔌 Smoke Services — All services reachable! ✅

🔌 Service connectivity validated by Smoke Services

github-actions · 2026-08-31T19:14:03Z

✅ Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

github-actions · 2026-08-31T19:14:04Z

🚀 Security Guard has started processing this pull request

github-actions · 2026-08-31T19:14:04Z

🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation

github-actions · 2026-08-31T20:23:11Z

✅ Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

github-actions · 2026-08-31T20:25:51Z

Smoke Test: Copilot Engine — @lpcox

✅ GitHub MCP: PR fix: preserve sparse rootfs staging #7920 (fix: preserve sparse rootfs staging), Add add-llm-provider skill for onboarding new LLM providers/engines #7903 (Add add-llm-provider skill)
✅ GitHub.com connectivity: HTTP 200
✅ File write/read: verified

Overall: PASS

📰 BREAKING: Report filed by Smoke Copilot
_{Add label ready-for-aw to run again}

github-actions · 2026-08-31T20:26:18Z

Smoke Test Results: Copilot BYOK (Direct) Mode

✅ GitHub MCP Testing — connectivity verified
✅ GitHub.com Connectivity — HTTP 200
✅ File Write/Read Test — confirmed
✅ BYOK Inference Test — active (api-proxy → api.githubcopilot.com)

Status: PASS
Mode: Direct BYOK (COPILOT_PROVIDER_API_KEY via api-proxy sidecar)

🔑 BYOK report filed by Smoke Copilot BYOK
_{Add label ready-for-aw to run again}

github-actions · 2026-08-31T20:26:24Z

Smoke Test: Docker Sbx Validation

PR #7920: fix: preserve sparse rootfs staging
PR #7903: Add add-llm-provider skill for onboarding new LLM providers/engines

✅ GitHub MCP connectivity
✅ GitHub.com connectivity (HTTP 200)
✅ File write/read test

Overall: PASS

cc @lpcox

📰 BREAKING: Report filed by Smoke Docker Sbx
_{Add label ready-for-aw to run again}

github-actions · 2026-08-31T20:26:54Z

Chroot Version Comparison Results

Runtime	Host Version	Chroot Version	Match?
Python	Python 3.12.14	Python 3.12.14	✅ YES
Node.js	v24.19.0	v22.23.2	❌ NO
Go	go1.22.12	go1.22.12	✅ YES

Overall: FAILED — Node.js version differs between host and chroot environments. smoke-chroot label not applied.

Tested by Smoke Chroot
_{Add label ready-for-aw to run again}

github-actions · 2026-08-31T20:27:24Z

Smoke test: FAIL

Reviewed:

fix: preserve sparse rootfs staging
Add add-llm-provider skill for onboarding new LLM providers/engines
GitHub merged PR review ✅
Safe-input PR query ❌
Playwright ✅
File write + bash readback ✅
Discussion interaction ❌
Build ✅
Overall: FAIL

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

msfeed25.pkgs.visualstudio.com
registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "msfeed25.pkgs.visualstudio.com"
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex
_{Add label ready-for-aw to run again}

github-actions · 2026-08-31T20:27:36Z

EGRESS_RESULT allow=pass deny=pass

✅ Allowed domain (github.com) reachable: allowed=200
✅ Non-allowed domain (example.com) blocked: 403 proxy denial

Overall: PASS

cc @lpcox

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation
_{Add label ready-for-aw to run again}

github-actions · 2026-08-31T20:27:40Z

@lpcox Tests:

GitHub MCP: ❌ (got fix: stabilize post-merge KVM checks #7922, Upgrade workflows to gh-aw v0.87.10 #7885 vs expected fix: preserve sparse rootfs staging #7920, Add add-llm-provider skill for onboarding new LLM providers/engines #7903)
GitHub.com Connectivity: ✅
File I/O: ✅
BYOK Inference: ✅

Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw)
Overall: FAIL

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)
_{Add label ready-for-aw to run again}

github-actions · 2026-08-31T20:27:57Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	passed	✅ PASS
Node.js	execa	✅	passed	✅ PASS
Node.js	p-limit	✅	passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — PASS

Note (non-blocking): Java's Maven build initially failed with Could not create local repository at /home/runner/.m2/repository because ~/.m2 is owned by root in this sandbox (a pre-existing environment permission issue, unrelated to the firewall). Retried successfully using -Dmaven.repo.local=/tmp/gh-aw/agent/m2repo; the Squid HTTP/HTTPS proxy settings in ~/.m2/settings.xml worked correctly for dependency downloads once the repo path issue was worked around.

All 18 test projects across 8 language ecosystems built and passed their test suites successfully through the AWF firewall.

Generated by Build Test Suite for #7922 · copilot · auto · 46.3 AIC · ⊞ 12K · ◷
_{Add label ready-for-aw to run again}

github-actions · 2026-08-31T20:29:54Z

@lpcox
GitHub MCP Testing: ❌
GitHub.com Connectivity: ✅
File Write/Read Test: ✅
Direct BYOK Inference Test: ✅
Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra
Overall status: FAIL

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)
_{Add label ready-for-aw to run again}

github-actions · 2026-08-31T20:30:51Z

Smoke Test: Claude Engine Validation

Check	Status
API	✅ PASS
gh CLI	✅ PASS
File	✅ PASS

Overall result: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by Smoke Claude for #7922 · claude · haiku45 · 32.8 AIC · ⊞ 4.5K · ◷
_{Add label ready-for-aw to run again}

github-actions · 2026-08-31T20:31:13Z

Smoke Test Results:

Redis PING: ❌ (Temporary failure in name resolution)
PG pg_isready: ❌ (no response)
PG SELECT 1: ❌ (could not translate host name)

Overall: FAIL — host.docker.internal did not resolve in the sandbox.

🔌 Service connectivity validated by Smoke Services
_{Add label ready-for-aw to run again}

github-actions · 2026-08-31T20:32:10Z

📡 OTel Tracing Smoke Test Results

✅ Scenario 1 (Module Loading): otel.js loads successfully, isEnabled() = true, exports all expected functions (startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled, plus internals).
✅ Scenario 2 (Test Suite): 3 test suites, 68/68 tests passed (span creation, gen_ai token attributes, budget attributes, parent trace propagation, OTLP/JSON serialization, proxy-aware exporter, file exporter).
✅ Scenario 3 (Env Var Forwarding): env-passthrough.ts forwards GITHUB_AW_OTEL_TRACE_ID/GITHUB_AW_OTEL_PARENT_SPAN_ID; api-proxy-env-config.ts forwards GH_AW_OTLP_ENDPOINTS, OTEL_EXPORTER_OTLP_ENDPOINT, and trace context vars.
✅ Scenario 4 (Token Tracker Integration): token-tracker-http.js contains the onUsage callback hook point.
⚠️ Scenario 5 (OTEL Diagnostics): No otel.jsonl span file found in this local run environment (no live sandbox execution occurred), so live export could not be observed — expected given the validation context; unit tests confirm export logic works.

Overall: All testable scenarios pass. No regressions detected.

📡 OTel tracing validated by Smoke OTel Tracing
_{Add label ready-for-aw to run again}

github-actions · 2026-08-31T20:34:38Z

🚀 Security Guard has started processing this pull request

lpcox and others added 13 commits August 31, 2026 08:14

feat: add durable microvm cleanup

72e6b27

Persist identity-validated Cloud Hypervisor cleanup records. Safely reap stale resources after abrupt owner death. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

test: lock cgroup assignment ordering

939f0ee

Ensure VMM cgroup assignment precedes durable process identity capture. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 92d2c346-d396-40d3-8b74-1f1447a04871

Merge remote-tracking branch 'origin/main' into lpcox-durable-microvm…

bbb141a

…-cleanup

Merge remote-tracking branch 'origin/main' into lpcox-durable-microvm…

291fda9

…-cleanup

Copilot AI balanced review requested due to automatic review settings August 31, 2026 18:49

lpcox added the cloud-hypervisor-kvm Trigger the Cloud Hypervisor live-KVM smoke/security suite label Aug 31, 2026

Copilot started reviewing on behalf of lpcox August 31, 2026 18:50 View session

Copilot AI reviewed Aug 31, 2026

View reviewed changes

Comment thread src/cloud-hypervisor/virtiofsd-sandbox.ts Outdated

Comment thread src/cloud-hypervisor/virtiofsd-sandbox.ts Outdated

lpcox changed the title ~~fix(cloud-hypervisor): stabilize post-merge KVM checks~~ fix: stabilize post-merge KVM checks Aug 31, 2026

lpcox added the ready-for-aw label Aug 31, 2026

lpcox deployed to aoai-model August 31, 2026 20:23 — with GitHub Actions Active

github-actions Bot mentioned this pull request Aug 31, 2026

Extract shared tokenAuthHeaders helper to deduplicate auth-prefix header assembly in copilot.js #7875

Merged

github-actions Bot added the smoke-copilot label Aug 31, 2026

github-actions Bot added the smoke-copilot-byok label Aug 31, 2026

github-actions Bot added the smoke-docker-sbx label Aug 31, 2026

github-actions Bot added the build-test label Aug 31, 2026

github-actions Bot mentioned this pull request Aug 31, 2026

[aw] Smoke Codex reported incomplete result #7925

Closed

lpcox deployed to aoai-model August 31, 2026 20:28 — with GitHub Actions Active

github-actions Bot added the smoke-copilot-byok-aoai-entra label Aug 31, 2026

lpcox deployed to aoai-model August 31, 2026 20:30 — with GitHub Actions Active

github-actions Bot added the smoke-claude label Aug 31, 2026

lpcox added the cloud-hypervisor-kvm Trigger the Cloud Hypervisor live-KVM smoke/security suite label Aug 31, 2026

lpcox merged commit 3d20555 into main Aug 31, 2026
300 of 301 checks passed

lpcox deleted the lpcox-durable-microvm-cleanup branch August 31, 2026 20:39

lpcox mentioned this pull request Aug 31, 2026

fix: avoid self-matching KVM process probes #7927

Merged

github-actions Bot mentioned this pull request Aug 31, 2026

fix(ci): match staged KVM processes #7929

Merged

Uh oh!

Conversation

lpcox commented Aug 31, 2026

Summary

Context

Validation

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Coverage Regression Detected

Overall Coverage

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026

Smoke Test Results: Copilot BYOK (Direct) Mode

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026

Chroot Version Comparison Results

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026

🏗️ Build Test Suite Results

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026

Smoke Test: Claude Engine Validation

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

github-actions Bot commented Aug 31, 2026 •

edited

Loading

github-actions Bot commented Aug 31, 2026 •

edited

Loading

github-actions Bot commented Aug 31, 2026 •

edited

Loading

github-actions Bot commented Aug 31, 2026 •

edited

Loading

github-actions Bot commented Aug 31, 2026 •

edited

Loading

github-actions Bot commented Aug 31, 2026 •

edited

Loading

github-actions Bot commented Aug 31, 2026 •

edited

Loading

github-actions Bot commented Aug 31, 2026 •

edited

Loading