Skip to content

ci(pr-verify): add the capture job that drives the affected app (S7 of #8651) #12232

New issue
New issue
Open
Task
Open
ci(pr-verify): add the capture job that drives the affected app (S7 of #8651)#12232
Task
Labels
host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.priority: mediumReal value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.

Description

@kyle-sexton
kyle-sexton
opened on Nov 2, 2026

Parent

Refs #8651. Phase S7 ("Phase 10: S7 pr-verify capture job") of the evidence-verification plan, durable copy at #11364 (comment); Brief at #11364 (comment). #11393 (S5c) waits on this job; before this issue existed it was blocked by #8651 itself, its own parent.

Relates to #7529 (Claude Code UI leg of the per-PR smoke).

Depends on

Outcome

pr-verify runs a capture job on in-repo PR heads. The job drives the affected app unattended, uploads the publishable captures as a retained artifact, and feeds them to the judge jobs as untrusted data. A missing or blocked capture never fails a verdict.

Plan (phase S7)

Satisfies: AC1, AC2, AC5, AC9, AC10
Size: Large. Files: .github/workflows/pr-verify.yml, .github/scripts/pr-verify/prompt.md, .github/scripts/pr-verify/workflow.test.mjs. Design: T8, sequence flow 2.

  • Probe step first. After setup-runner-isolation, a head-code step tries to read the runtime token and the credential from a hook. The job fails closed if either is readable.
  • Job capture. It uses the same if as the existing judge jobs, so fork heads never run. It runs the gates and uses a read-only token, .base plugins and --trusted-root .base, then calls /testing:run-e2e unattended app=<merge.verify-capture.app> base=<base-sha>. It uploads the publishable files as pr-capture-<pr>-<sha> with retention-days. When the key is unset, it skips with a note.
  • Gates first. The capture job's first steps are the kill switch, resolver, trusted-trigger and agree steps (check-intent's copies stay), so it captures nothing on a run that judges nothing. It sets timeout-minutes: 30. The outer session runs through ci-workflows' run-claude-cli with its own role, a tool allowlist and the drive maxEntryPoints cap (S4b). A capture timeout leaves capture: blocked and never stops the judge.
  • Judge jobs. Only check-intent gains needs: capture, with !cancelled(). check-intent-subscription and check-intent-fallback keep their success chain and download the same run's artifact by id. All three download into pr-verify-data/capture/. The prompt frames the capture as untrusted data and says a missing or blocked capture is never a FAIL reason. A trusted-text manifest line Evidence loop: 3/3 cap reached, last NEEDS WORK is a FAIL (AC7's park path then runs).
  • Last step. After the probe passes on a real PR, set merge.verify-capture.app: claude-code and show one in-repo PR with a judged capture (AC1). Update the header comment block in pr-verify.yml that describes the jobs (reviewed change: jobs now take an artifact).

Credential separation (Q10, Q11)

The superseding Q11 record (#11364 (comment)) is the Issue Triage session's reading of the owner's 2026-10-31 direction. It says:

  • The capture job may hold pr-verify's existing credential (Q10).
  • The PR's own code runs in a child process whose environment and readable files hold no token. The credential stays with the Claude Code driver and is never passed to, exported to, or written where the app under test can read it.
  • In-repo heads only.

If the build cannot keep the token out of the PR code's reach, it stops and brings that back to the owner as a new question. It does not ship without the separation.

Done when

  • node --test .github/scripts/pr-verify/workflow.test.mjs exits 0 with these cases:
    • the capture job's permissions contain no write;
    • the three judge jobs carry identical capture-download steps;
    • the capture if includes the head-repository check;
    • pr-verify.yml has no job-level always();
    • only check-intent has needs: capture;
    • a probe step exists, runs after setup-runner-isolation and before any head code, and has no continue-on-error.
  • The capture job's step that starts the app under test passes it no credential variable, and a test asserts that the app process's environment holds none of the pr-verify credential names.
  • actionlint passes on pr-verify.yml.
  • One in-repo PR shows a judged capture with merge.verify-capture.app: claude-code set (AC1).

Builder: the Automation Lanes session. This is host-run work: it edits a workflow file and needs a live PR run.

Activity

  1. kyle-sexton
    added
    priority: mediumReal value, no hard deadline; normal backlog flow.
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    on Nov 2, 2026
  2. kyle-sexton
    added the
    Task
    issue type on Nov 2, 2026
  3. kyle-sexton
    added
    work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 2, 2026
  4. kyle-sexton
    added a parent issue on Nov 2, 2026
  5. kyle-sexton
    marked this as blocked by 2 issues on Nov 2, 2026
  6. kyle-sexton
    marked this as blocking on Nov 2, 2026

  7. kyle-sexton commented on Nov 4, 2026

    @kyle-sexton
    kyle-sexton
    on Nov 4, 2026
    MemberAuthor

    Claim (lane 3, hub-assigned): partial build only. Lane 3 builds the parts that do not need #11386 on a local branch: the probe step, the capture job skeleton with its gates and timeout, the judge jobs' capture-download steps, the prompt framing, the workflow.test.mjs cases and actionlint. The live judged capture (done-when 4) waits on #11386. The body names the Automation Lanes session as builder; the hub has told that session lane 3 holds this partial build. No push until the hub confirms a partial PR is wanted.

  8. kyle-sexton commented on Nov 4, 2026

    @kyle-sexton
    kyle-sexton
    on Nov 4, 2026
    MemberAuthor

    Owner question Q1 (credential separation, under Q11). The build stopped on this issue's own rule: "If the build cannot keep the token out of the PR code's reach, it stops and brings that back to the owner as a new question." Nothing was committed.

    Problem. The merge gate trusts a verdict only because a run of pr-verify.yml uploaded it. In .github/workflows/pr-verify.yml:13-15: "Only steps of a run of this file on main can upload that run's artifacts, so the pr-merge lane counts a verdict only through its run record". And :21: "No pull request code runs here." babysit_verify_evidence.py keys on WORKFLOW_PATH = ".github/workflows/pr-verify.yml".

    A capture job inside that run puts PR code in the same run. Several node steps run later in that job while holding the job's runtime token: the capture artifact upload, and the post steps of checkout and setup-node, which run regardless. pr-run-activity-read.yml:2021-2024 documents the same exposure in the read lane: "Each actions/checkout post step runs git ... as a node step that holds the job's runtime token." PR code that outlives its step, or a file it plants, could then upload a forged pr-verify-<pr>-<sha>-pass.

    The issue's probe step checks only what can be read at one moment. It does not catch this.

    Options

    1. Recommended. Capture runs as its own workflow, triggered by the same workflow_run. The judge jobs download its artifact across runs by run id, with actions: read. PR code then never runs inside a pr-verify.yml run, so the verdict's trust basis stays as it is. Basis: pr-verify.yml:13-15,21 and babysit_verify_evidence.py WORKFLOW_PATH. Cost: the two runs must be sequenced, since the judges wait on the capture run or skip it.
    2. Keep one run, but have the judge jobs seal each verdict in a way PR code cannot reproduce. Every verdict reader would have to change.
    3. Keep one run and harden it: a process reaper, a root-owned externals directory, removing .git directories as the read lane does. You would accept the remaining risk of a forged pass verdict.

    Unblocks: all of #12232, and #11393 after it.

    Also found during the build, to fix whichever option wins:

    • The issue's /testing:run-e2e unattended ... call shape is wrong: the bare word is scenario text. The run needs the --unattended flag (plugins/testing/skills/run-e2e/SKILL.md:45-53).
    • The e2e-summary.json capture shape does not match the keys write-verdict.mjs reads, so a raw summary always records blocked.
    • The localLoop regex in write-verdict.mjs does not match Evidence loop: 3/3 cap reached, last NEEDS WORK.
    • resolve.mjs has no capture model route yet.
  9. kyle-sexton
    added
    status: needs-decisionAwaiting a human or maintainer judgment call.
    on Nov 4, 2026

  10. kyle-sexton commented on Nov 4, 2026

    @kyle-sexton
    kyle-sexton
    on Nov 4, 2026
    MemberAuthor

    Decision recorded. Q1 (issuecomment-6619061583): option 1. Capture runs as its own workflow, and pr-verify's judge jobs download its artifact across runs. PR code never runs inside a pr-verify.yml run. Owner, verbatim: "Go w/ whatever this suggests", with the standing direction s3: "Never weaken a high/critical security requirement or boundary to gain throughput." Options 2 and 3 either keep PR code in the run that holds the verdict's upload rights or accept a forged-pass risk. Lane 3 resumes the #11386-independent build under option 1, and also fixes the three plan defects listed in Q1.

  11. kyle-sexton
    removed
    status: needs-decisionAwaiting a human or maintainer judgment call.
    on Nov 4, 2026
  12. kyle-sexton
    added 2 commits that reference this issue on Nov 4, 2026

    fix(pr-verify): fence the capture driver's file tools to the head and…

    20aeac6

    chore(source-control): add a none fragment for the hold-floor test pins

    9b322bf

  13. kyle-sexton commented on Nov 4, 2026

    @kyle-sexton
    kyle-sexton
    on Nov 4, 2026
    MemberAuthor

    Status. #13053 merged 2026-11-04T05:02:46Z. The capture workflow is on main and stays off while merge.verify-capture.app is unset; its first run after the merge ended success.

    This issue stays open for done-when 4, one in-repo PR showing a judged capture with the app set, which waits on #11386. The PR's Remaining section lists the follow-ups to fix before the app is set.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.priority: mediumReal value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.

    Type

    Task

    Fields

    No fields configured for Task.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions