Skip to content

feat: reserve microVM network resources atomically - #7893

Merged
lpcox merged 1 commit into
mainfrom
lpcox-atomic-network-reservations
Aug 31, 2026
Merged

feat: reserve microVM network resources atomically#7893
lpcox merged 1 commit into
mainfrom
lpcox-atomic-network-reservations

Conversation

@lpcox

@lpcox lpcox commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a root-owned, flock-serialized reservation registry for per-run guest subnets, infrastructure addresses, and resource tokens
  • validate durable owner identity with boot ID, PID, process start time, and lease ID; reclaim stale records only after matching live namespaces, interfaces, routes, addresses, and firewall rules are gone
  • make network cleanup lease-owned and tag shared DOCKER-USER rules per reservation so concurrent runs cannot remove each other's resources
  • update Cloud Hypervisor diagnostics/docs and add separate-process live smoke coverage for the agent-microvm v0.9.0 concurrency gap

Validation

  • npm test -- --runInBand — 330 suites, 5,282 tests passed
  • npm run build
  • npm run lint -- --quiet
  • npx markdownlint-cli2 docs/cloud-hypervisor-foundation.md
  • bash -n scripts/ci/cloud-hypervisor-live-smoke.sh
  • focused reservation/network/manager/preflight suite — 83 tests passed

The KVM live smoke itself requires a GitHub-hosted Ubuntu x86_64 KVM runner; this change adds the real cross-process reservation exercise to that existing CI suite.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings August 31, 2026 14:41
@github-actions

Copy link
Copy Markdown
Contributor

Documentation Preview

Documentation build failed for this PR. View logs.

Built from commit 129686a

@lpcox lpcox changed the title Reserve microVM network resources atomically feat: reserve microVM network resources atomically Aug 31, 2026
@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric Base PR Delta
Lines 93.86% 93.14% 📉 -0.72%
Statements 92.67% 91.92% 📉 -0.75%
Functions 93.24% 92.27% 📉 -0.97%
Branches 86.14% 85.59% 📉 -0.55%
📁 Per-file Coverage Changes (4 files)
File Lines (Before → After) Statements (Before → After)
src/microvm/network-plan.ts 95.2% → 93.1% (-2.08%) 94.5% → 92.5% (-2.00%)
src/microvm/network-manager.ts 98.4% → 98.5% (+0.14%) 96.9% → 97.2% (+0.26%)
src/microvm/network-commands.ts 97.8% → 98.0% (+0.23%) 97.9% → 98.2% (+0.27%)
src/log-directory-setup.ts 96.2% → 100.0% (+3.78%) 96.3% → 100.0% (+3.71%)
✨ New Files (1 files)
  • src/microvm/network-reservation.ts: 52.0% lines

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Concurrent iptables locking and incomplete live-resource detection can still cause failures or unsafe reservation recovery.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Balanced
Findings: 1 High severity · 2 Medium severity

New issues introduced by this change (3)
Severity Finding
High severity src/​microvm/​network-reservation.ts — This ignores live host routes that ip route emits as a bare IPv4 (a /32) and typed routes such…
Medium severity src/​microvm/​network-reservation.ts — The PR's stale-recovery contract includes waiting for the reserved address to disappear, but this…
Medium severity src/​microvm/​network-commands.ts — The reservation lock is released before network setup/cleanup, so independent managers can execute…
What changed in this PR

Adds atomic, cross-process microVM network reservations and lease-owned cleanup to support concurrent Cloud Hypervisor runs safely.

Changes:

  • Adds a root-owned reservation registry for subnets, addresses, and resource tokens.
  • Integrates reservations and owned firewall rules into the network lifecycle.
  • Expands unit, live-smoke, and documentation coverage.
File Description
src/​microvm/​network.ts Exports reservation APIs and types.
src/​microvm/​network.test.ts Tests owned-rule and cleanup behavior.
src/​microvm/​network-types.ts Defines allocation and reservation types.
src/​microvm/​network-reservation.ts Implements locking, allocation, and recovery.
src/​microvm/​network-reservation.test.ts Tests concurrent reservations and recovery.
src/​microvm/​network-plan.ts Supports reserved network allocations.
src/​microvm/​network-manager.ts Releases reservations during cleanup.
src/​microvm/​network-commands.ts Adds token-owned firewall rules.
src/​cloud-hypervisor/​vm-config-builder.test.ts Updates plan fixture.
src/​cloud-hypervisor/​preflight.ts Requires trusted flock.
src/​cloud-hypervisor/​preflight.test.ts Updates preflight expectations.
src/​cloud-hypervisor/​manager.ts Wires reservation dependencies.
src/​cloud-hypervisor/​manager.test.ts Tests reservation propagation.
src/​cloud-hypervisor/​manager-types.ts Extends manager dependency contracts.
src/​cloud-hypervisor/​manager-start.ts Reserves networking before setup.
src/​cloud-hypervisor-runtime-backend.test.ts Updates host-tool fixture.
scripts/​ci/​cloud-hypervisor-live-smoke.sh Adds cross-process reservation smoke coverage.
docs/​cloud-hypervisor-foundation.md Documents reservation and cleanup behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +408 to +411
for (const line of routeOutput.stdout.split('\n')) {
const destination = line.trim().split(/\s+/)[0];
if (/^\d+\.\d+\.\d+\.\d+\/\d+$/.test(destination)) routes.push(destination);
}
Comment on lines +436 to +437
|| live.firewallTokens.has(record.resourceToken)
|| live.routes.some((route) => cidrsOverlap(route, record.guestSubnet));
Comment on lines 134 to +140
const checkResult = await this.execute(
'iptables',
['-t', 'filter', '-C', 'DOCKER-USER', '-i', bridgeName, '-o', bridgeName, '-j', 'ACCEPT'],
[
'-t', 'filter', '-C', 'DOCKER-USER',
'-i', bridgeName, '-o', bridgeName,
...ownership, '-j', 'ACCEPT',
],
@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Copilot

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Docker Sbx has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Docker Sbx

@lpcox
lpcox deployed to aoai-model August 31, 2026 16:37 — with GitHub Actions Active
@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Smoke Claude passed

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by Smoke Claude for #7893

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Smoke Gemini reports failed. Facets need polishing...

💎 Faceted by Smoke Gemini

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Contribution Check completed successfully!

PR #7893 follows the contribution guidelines in CONTRIBUTING.md based on the provided context: it includes tests for the new functionality, updates documentation for the feature, places new code under the appropriate src/ and scripts/ci/ locations, and the PR description is clear and references the related concurrency gap.

Generated by Contribution Check for #7893

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

Tested by Smoke Chroot

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

🔌 Service connectivity validated by Smoke Services

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

📡 OTel tracing validated by Smoke OTel Tracing

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

🔑 BYOK report filed by Smoke Copilot BYOK

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Security Guard has started processing this pull request

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • msfeed25.pkgs.visualstudio.com
  • registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "msfeed25.pkgs.visualstudio.com"
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Build Test Suite completed successfully!

Generated by Build Test Suite for #7893

@github-actions

Copy link
Copy Markdown
Contributor

EGRESS_RESULT allow=pass deny=pass

✅ Allowed domain (github.com) reachable: allowed=200
✅ Blocked domain (example.com) denied: proxy tunnel 403

Overall: PASS — network isolation egress enforcement working as expected.

cc @lpcox

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation
Add label ready-for-aw to run again

@github-actions github-actions Bot added the smoke-copilot-network-isolation Copilot network-isolation egress smoke test label Aug 31, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Claude Engine Validation

Check Status
API ✅ PASS
gh CLI ✅ PASS
File ✅ PASS

Overall result: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by Smoke Claude for #7893 · claude · haiku45 · 56.1 AIC · ⊞ 4.5K ·
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot BYOK (Direct) Mode

Test Result
GitHub MCP (list_pull_requests)
github.com Connectivity (HTTP 200)
File Write/Read
BYOK Inference Path

Status: PASS

Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY) via api-proxy → api.githubcopilot.com

🔑 BYOK report filed by Smoke Copilot BYOK
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test Results: FAIL ❌

  • Redis PING: ❌ Temporary failure in name resolution
  • PostgreSQL pg_isready: ❌ no response
  • PostgreSQL SELECT 1: ❌ could not translate host name "host.docker.internal"

Overall: FAILhost.docker.internal could not be resolved from within the AWF sandbox.

🔌 Service connectivity validated by Smoke Services
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: API Proxy OpenTelemetry Tracing

Scenario Result
1. OTEL module loading otel.js loaded successfully; exports startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled, plus internal exporters/helpers
2. Test suite ✅ 3 test suites, 68/68 tests passed (0 failed)
3. Env var forwarding ✅ Agent passthrough forwards GITHUB_AW_OTEL_TRACE_ID and GITHUB_AW_OTEL_PARENT_SPAN_ID (env-passthrough.ts); api-proxy config forwards GH_AW_OTLP_ENDPOINTS, OTEL_EXPORTER_OTLP_ENDPOINT, GITHUB_AW_OTEL_TRACE_ID, GITHUB_AW_OTEL_PARENT_SPAN_ID (api-proxy-env-config.ts)
4. Token tracker integration onUsage callback hook confirmed present in token-tracker-http.js
5. OTEL diagnostics ⚠️ No otel.jsonl exported in /tmp/gh-aw/sandbox/firewall/logs/api-proxy-logs/ for this run (only models.json, token-tracker-audit.jsonl, token-usage.jsonl present) — no LLM calls were routed through the api-proxy sidecar during this run, so no spans were generated. This is expected since the workflow's agent task didn't need to make proxied API calls beyond the primary session (which uses its own OTEL export path, seen separately at /tmp/gh-aw/otel.jsonl). Not an unexpected failure.

Overall: ✅ Pass — all core OTEL integration points (module, tests, env forwarding, token-tracker hook) verified working. Diagnostics scenario shows expected "no spans" state given no api-proxy-routed LLM traffic in this run, not a bug.

📡 OTel tracing validated by Smoke OTel Tracing
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Chroot Version Comparison

Runtime Host Version Chroot Version Match?
Python Python 3.12.14 Python 3.12.14 ✅ YES
Node.js v24.19.0 v22.23.2 ❌ NO
Go go1.22.12 go1.22.12 ✅ YES

Overall: FAILED — Node.js version mismatch between host and chroot environments (v24.19.0 vs v22.23.2). The smoke-chroot label was not added since not all tests passed.

Tested by Smoke Chroot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia 1/1 passed ✅ PASS
Bun hono 1/1 passed ✅ PASS
C++ fmt N/A ✅ PASS
C++ json N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world N/A (ran, output correct) ✅ PASS
.NET json-parse N/A (ran, output correct) ✅ PASS
Go color 1/1 passed ✅ PASS
Go env 1/1 passed ✅ PASS
Go uuid 1/1 passed ✅ PASS
Java gson 1/1 passed ✅ PASS
Java caffeine 1/1 passed ✅ PASS
Node.js clsx passed ✅ PASS
Node.js execa passed ✅ PASS
Node.js p-limit passed ✅ PASS
Rust fd 1/1 passed ✅ PASS
Rust zoxide 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — PASS

Notes:

  • All repository clones succeeded.
  • Java: ~/.m2/repository was not writable in this sandbox (unrelated to the firewall — a local filesystem permission restriction), so -Dmaven.repo.local=/tmp/gh-aw/agent/m2repo was used to point Maven's local repo elsewhere. All downloads/builds/tests then succeeded normally through the Squid proxy via ~/.m2/settings.xml.
  • No firewall-related failures were observed across any ecosystem.

Generated by Build Test Suite for #7893 · copilot · auto · 33.3 AIC · ⊞ 12K ·
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot Engine@lpcox

Overall: PASS

📰 BREAKING: Report filed by Smoke Copilot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Docker Sbx@lpcox

  • ✅ GitHub MCP connectivity (list_pull_requests)
  • ✅ github.com connectivity (HTTP 200)
  • ✅ File write/read test

PRs checked: #7892 fix: filter workflow commands from microVM output; #7889 feat: verify Cloud Hypervisor runtime confinement

Overall: PASS

📰 BREAKING: Report filed by Smoke Docker Sbx
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

@lpcox
fix: filter workflow commands from microVM output
feat: verify Cloud Hypervisor runtime confinement

❌ GitHub MCP Testing (expected #7892, #7889, got #7903, #7896)
✅ GitHub.com Connectivity
✅ File Write/Read Test
✅ BYOK Inference Test

Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw)

Overall: FAIL

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke test:

  • fix: filter workflow commands from microVM output
  • feat: verify virtiofsd sandbox confinement before VM boot
  • GitHub merged PR review: ✅
  • safeinputs-gh PR query: ❌
  • Playwright title: ✅
  • File write/read: ✅
  • Discussion query/comment: ❌
  • Build (npm ci && npm run build): ✅
    Overall: FAIL

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • msfeed25.pkgs.visualstudio.com
  • registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "msfeed25.pkgs.visualstudio.com"
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

@lpcox

fix: filter workflow commands from microVM output
feat: verify Cloud Hypervisor runtime confinement

MCP: ✅
GitHub.com: ✅
File IO: ✅
BYOK: ✅

Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra

OVERALL: PASS

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)
Add label ready-for-aw to run again

@lpcox
lpcox deployed to aoai-model August 31, 2026 16:50 — with GitHub Actions Active
@lpcox
lpcox merged commit dba82e2 into main Aug 31, 2026
168 of 171 checks passed
@lpcox
lpcox deleted the lpcox-atomic-network-reservations branch August 31, 2026 16:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants