From b36a5d35d4583eaae4e991ebaf98960bef7bb1e7 Mon Sep 17 00:00:00 2001 From: Claude Code Bot Date: Wed, 29 Apr 2026 15:54:37 -0700 Subject: [PATCH] docs(security): document execution environment constraints MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 2026-04-29 Q2 validation run of the recurring audit agent (routine trig_01JaKYSFQhPJoc3jADyQPBgM, issue dev-env#23) surfaced two environment-level constraints not previously documented: 1. The cloud sandbox may not have `gh` CLI. The agent falls back to MCP GitHub tools (code search + file reads), which can detect Patterns 1, 2, 3, 5, 7, 8, 9 from workflow content but cannot reach the workflow-permissions API (Pattern 6) or perform git reachability analysis (Pattern 4). 2. MCP write scope may be limited to smartwatermelon/dev-env. Issue creation works; cross-repo draft PR creation may fail. These are NOT methodology defects — the agent correctly reports them as PARTIAL coverage and skips PRs cleanly. The new "Execution environment notes" section documents: - What each constraint blocks (specific patterns and operations). - The local-verify workflow for Pattern 6 (matches the inaugural-audit one-liner that was used during the Q2 run validation). - When to consider switching environment_id (5+ skipped PRs/quarter, recurring PARTIAL on Pattern 6, or new patterns needing other API scope). Also appends the Q2 auto run to the audit history table. Q2 audit summary: 0 NEW, 0 RESOLVED, 1 UNCHANGED (the Tier-3 fetch-metadata@v3 finding already tracked in dev-env#19). Pattern 6 verified clean locally during Q2 run review (no regressions on the ralph-burndown / mac-server-setup flips from the inaugural audit). Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/github-actions-security-methodology.md | 86 ++++++++++++++++++++- 1 file changed, 85 insertions(+), 1 deletion(-) diff --git a/docs/github-actions-security-methodology.md b/docs/github-actions-security-methodology.md index 557649d..d2eb3e6 100644 --- a/docs/github-actions-security-methodology.md +++ b/docs/github-actions-security-methodology.md @@ -290,11 +290,95 @@ If the methodology needs a change the agent can't make on its own (a new attack pattern, a deprecated check), the agent files an issue labeled `audit-methodology-update`. Triage those as you would any backlog item. +## Execution environment notes + +The recurring agent runs in a sandboxed cloud environment that may not +have the same tooling assumed by this methodology. The 2026-04-29 Q2 +run surfaced two such constraints — these are environment-level, not +methodology-level, and the agent is correct to note them as partial +coverage rather than treat them as findings. + +### Constraint 1 — `gh` CLI may be unavailable + +When `gh` isn't present, the agent falls back to MCP GitHub tools (read +via `search_code` and direct file reads). That covers Patterns 1, 2, 3, +5, 7, 8, and 9 — they're all detectable from workflow file content +that's reachable via code search. It does NOT cover: + +- **Pattern 6** (`default_workflow_permissions`) — the + `/repos/{owner}/{repo}/actions/permissions/workflow` endpoint isn't + reachable through MCP code search. +- **Pattern 4** (imposter commits) — requires git reachability analysis + the cloud sandbox cannot perform. + +When the audit issue marks a pattern as **PARTIAL**, run this locally +to verify Pattern 6 across all in-scope repos: + +```bash +for repo in $( + gh repo list smartwatermelon --limit 200 \ + --json name,isArchived,isFork \ + --jq '.[] | select(.isArchived == false and .isFork == false) | "smartwatermelon/" + .name' + gh repo list nightowlstudiollc --limit 200 \ + --json name,isArchived,isFork \ + --jq '.[] | select(.isArchived == false and .isFork == false) | "nightowlstudiollc/" + .name' +); do + perms=$(gh api "repos/$repo/actions/permissions/workflow" \ + --jq '.default_workflow_permissions' 2>/dev/null) + [[ "$perms" == "write" ]] && echo "WRITE: $repo" +done +``` + +Anything that returns `WRITE` is a real finding; flip via: + +```bash +gh api -X PUT "/repos///actions/permissions/workflow" \ + -f default_workflow_permissions=read \ + -F can_approve_pull_request_reviews=true +``` + +Pattern 4 has no cheap human substitute; rely on SHA pinning hygiene +and (eventually) zizmor. See [`dev-env#19`](https://github.com/smartwatermelon/dev-env/issues/19). + +### Constraint 2 — MCP write scope may be narrow + +Issue creation typically works against `smartwatermelon/dev-env`. +Cross-repo `gh pr create --draft` may fail. The routine prompt +instructs the agent to skip cleanly with a `PR creation skipped for +repo X: ` line in the issue body — not to retry, escalate, or +abort the audit. + +When that line appears, apply the proposed remediation snippets from +the issue body manually. The audit report contains them inline for +exactly this purpose; manual application is typically a one-line edit +plus a regular branch + PR cycle. + +### When to upgrade the environment + +These constraints are tolerable as long as the audit's NEW-finding +count stays low. Re-evaluate the `environment_id` (in the routine +settings, not in this repo) if any of these hold: + +- A quarter produces 5+ NEW findings that would have warranted draft + PRs, and manual application becomes the bottleneck. +- Pattern 6 verification keeps reporting PARTIAL, and the local-verify + workflow above starts getting skipped. +- A new pattern emerges that requires API access the MCP scope can't + provide. + +The full inventory of available environments is in the routine +configuration UI. `GitHub swm` and `GitHub NOS` are likely scoped to +their respective orgs; `Default` is org-agnostic but read-leaning. +Switching environments may require the routine prompt to re-establish +its legitimacy anchor — the methodology doc reference covers this, but +test with a one-shot run before relying on a quarterly cron. + ## Audit history | Date | Repos | Findings (severity) | PRs landed | Issues filed | |------|-------|---------------------|------------|--------------| -| 2026-04-29 | 29 | 0 critical, 4 high, 10 medium | 9 | 5 | +| 2026-04-29 (Q2 auto, validation run) | 29 | 0 new (1 unchanged Low, tracked) | 0 | 1 ([dev-env#23](https://github.com/smartwatermelon/dev-env/issues/23)) | +| 2026-04-29 (inaugural manual) | 29 | 0 critical, 4 high, 10 medium | 9 | 5 | The 2026-04-29 inaugural audit was performed manually. The full report lives in the conversation that produced it; key outcomes: