Skip to content

perf(pr-verify): read the key's credit history only for a head it judges - #12799

Merged
kyle-sexton merged 8 commits into
mainfrom
perf/pr-verify-skip-judged-head
Nov 3, 2026
Merged

kyle-sexton merged 8 commits into
mainfrom
perf/pr-verify-skip-judged-head

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

No related issue: GITHUB_TOKEN volume cut #1 (.work volume-cuts 2026-11-03)

Summary

pr-verify ran its credit gate (check-intent-credit) before the resolver's no-op exit. Every non-skipped run paid for the gate's reads and a VM, including the runs that then stop at the resolver because the head is already judged, the PR is a draft, or there is no single PR. This change folds the gate into check-intent after the resolver, so a no-op run spends nothing on it.

This PR was first meant to remove 8 duplicate full judgments seen in the 05:47-06:47Z hour on 2026-11-03. Their run logs show none of them was a duplicate. The judged-head skip (resolve-pr.mjs judged()) is unchanged.

Head (as the time join counted it) Run What it actually was
52a312c (3 judgments) 41587739299, 41587743734, 41587746198 Three different PRs: 12730, 12640, 12738 (one verdict artifact each). The time join put them on one head.
a778997 41589688163 PR 12756 at e3a17c8 (join error)
2a6e5ad 41588831301 PR 12704 at 9a365b4 (join error)
2a6e5ad 41591273474 The only judgment of PR 12747. The key hit a credit stop and the subscription leg hit the core rate limit (403 ... until 06:49:01Z), so no verdict was written.
c146b08 41591390022 PR 12740 (join error)
c146b08 41591342510, then 41591453992 PR 12714. The first run lost its verdict to the same rate limit, so the second judgment was needed (fail-i12649).
03bb2a3 41590798290 PR 12712. An intended re-judge: the log shows resolve-pr: proceed=true reason=body-edited, after a fail-i12658.
5e7556c 41587683570 Not a judgment. The resolver logged reason=already-judged after 8 calls, and no model ran.

Fix

The check-intent-credit job is removed. Its gate and report steps now run in check-intent after "Hand the reads on":

  • the check-key-credit read, with id: credit;
  • "Report the skipped key leg".

They run only when steps.agree.outcome == 'success', which means the resolver and the trigger gate both passed.

The gate reads the credit-stop marker that ci-workflows#973 added (merge commit bc62beb). On the old gate, no-op runs would have filled its newest-3 run window (MAX_JOB_READS = 3) and hidden a credit stop. The new gate does not scan runs:

  • check-key-credit is pinned at bc62beb. It makes one call, GET /repos/{repo}/actions/artifacts?name=<marker>. Its marker output is key-credit-stop-pr-verify.yml-check-intent.
  • After a judge step's credit-stop handoff, check-intent runs three new steps: "Write the key's credit-stop marker" (the run id), "Scan the key's credit-stop marker for credentials" (scan-upload) and "Upload the key's credit-stop marker". The upload is named key-credit-stop-pr-verify.yml-check-intent, the gate's name, written out so its own 1-day row in .github/artifact-retention.json claims it (a computed name would fall to the job's pr-verify-* row, 90 days). The write step runs only while steps.credit.outputs.marker equals that name, so a gate that names another marker uploads none. All three steps are continue-on-error. This follows the key job in pr-review-cli.yml at bc62beb.
  • The write step keeps this workflow's handoff-reason == 'credit-stop' trigger, not upstream's lane-class. That is what the old record step used, and a workflow test bans lane-class == 'credit-stop' in any if.
  • Both "Record the key's credit stop" steps are gone: the carried one and the key leg's own. The window now counts from the marker's upload.
  • The marker upload needs no new permission. check-intent already uploads its verdict, and its token keeps actions: read.
  • pr-verify's other ci-workflows actions move to the same pin (one-pin test). Between 2476d5d and bc62beb, run-claude-cli changed only one output description, and apply-claude-managed-settings did not change. bc62beb is added to callers.test.mjs's checked launchers, and replay.mjs RUN_CLAUDE_CLI_PIN moves to it.

When the key is skipped:

  • The model resolve, credential check, managed settings, runner isolation and prompt steps do not run, so no judge runs either.
  • "Fail without a verdict" does not fire.
  • The new skip-key job output hands the judgment to check-intent-subscription. That job's if and its 13 fallback-reason lines read needs.check-intent.outputs.skip-key.
  • The subscription job now takes check-intent's carried reads. Before, check-intent was skipped and the subscription job read everything again.

GITHUB_TOKEN (counter A) call change:

  • Per no-op run (stopped at the resolver): the gate makes no calls and the run uses one fewer VM. The old gate cost 2 to 4 calls: 1 run list plus up to 3 job lists. Job 125768540875 logged cost=4, and ci-workflows#973 measured 2.
  • Per judged run: the gate makes 1 call (the marker listing), down from 2 to 4.
  • Per run where the gate skips the key: the subscription job takes the carried reads instead of repeating check-intent's reads (resolve, trigger gate, trusted text, upstream jobs). Saving not measured; the logged credit-stop runs carried their reads, at 3 calls on the subscription leg.
  • Per hour: the evidence hour had about 55 no-op runs. That is about 110 to 220 fewer A calls per hour from no-op runs, plus 1 to 3 fewer per judged run, and about 55 fewer VMs per hour.
  • Counter B (owner token): no change.

The old gate missed a credit stop because of its newest-3 window. At 06:41:40 it logged "no run of the newest 3 settles it", while run 41590798290 had recorded a credit stop at 06:39:33. The marker read has no such window, which fixes this.

Verification

  • .github/scripts/pr-verify/workflow.test.mjs: 39 pass. The tests check that:

    • the gate runs after the resolver, the trigger gate, agree and the reads hand-on;
    • check-key-credit appears once and is pinned at bc62beb;
    • a skipped key gates the setup steps and "Fail without a verdict";
    • the skip-key output and the subscription gate behave as described.
  • The marker test checks that:

    • no "Record the key's credit stop" step or origin-run reference remains;
    • the marker name is key-credit-stop-pr-verify.yml- plus the gate step's key-job (gate.cjs markerName at bc62beb), and equals key-credit-stop-pr-verify.yml-check-intent;
    • the write step's if is steps.credit.outputs.marker == '<that name>' plus the 13-step handoff-reason == 'credit-stop' chain;
    • the upload's if is the scan step's if in parentheses, then its clean output;
    • write, scan and upload run in that order, after every judge step;
    • the upload's name is that literal, and the scan and upload use the same path;
    • retention is 1, with if-no-files-found: error;
    • all three steps are continue-on-error;
    • the subscription and fallback jobs have none of the three steps.
  • The artifact-upload test now allows the key job a second upload, the marker. The subscription parity test excludes the three marker steps.

  • .github/scripts/maintenance-reconcile-lost-runs/reconcile.test.mjs: 151 pass. The carried-record row is gone from the credit-skip fixture. reconcile.mjs reads no credit step names.

  • Every node suite that reads pr-verify.yml passes, after merging main at 7cfb456a2d:

    Suite Tests
    check-trusted-trigger 398
    log-api-budget 67
    resolve-config callers 234
    resolve-config models 143
    reconcile 151
    carry-run-cache 22
    enable-auto-merge 205
    merge-holds 65
    feedback-dispatch 208
    fix-ci-context 53
    workflow-gates 49
    chunk-diff 14
    parity 20
    resolve-pr 145
    workflow 39
    write-verdict 39
    lane-transcripts pull 27
  • babysit-prs pytest (review evidence, verify evidence, verify index, verify rules sync): 753 passed, at the first commit. It was not re-run for the marker change, which touches no Python or verdict names.

  • scripts/check-artifact-retention.sh --check: 82 upload steps match the table (57 families), after merging main at aef714a. scripts/check-artifact-retention.test.sh: 32 pass, 0 fail.

  • After the retention-row fix: workflow.test.mjs and resolve-config callers.test.mjs, 273 pass, 0 fail.

  • actionlint (shellcheck on PATH) on pr-verify.yml, with the repository config's cache-mode ignore: clean.

Related

  • Evidence: the volume-cuts measurement of 2026-11-03, ranks 1 and 2 (a local, uncommitted record).
  • Depends on ci-workflows#973 (merged as bc62beb): the marker contract is in .github/actions/check-key-credit/gate.cjs and action.yml there.
  • pr-review-main.yml's repin and marker upload are a separate change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VcupomBPg7kuobwLoNpxEW

check-intent-credit ran before the resolver's no-op exit, so every
non-skipped pr-verify run paid its run-list and three job-list reads
(api-budget cost=4, run 41591342510) and a VM, including the ~55/h runs
that then skip at the resolver (already judged, draft, not one PR).

Fold the gate, its carried record and the skipped-leg report into
check-intent after the resolver, the trigger gate and the reads hand-on.
A skipped key runs no pre-judge setup or judge, writes no "no verdict"
failure, and hands on through a skip-key job output, so the
subscription job now takes the carried reads instead of reading again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VcupomBPg7kuobwLoNpxEW
@kyle-sexton kyle-sexton added the do-not-merge Hard merge gate: do not merge while applied. label Nov 3, 2026
kyle-sexton and others added 4 commits November 3, 2026 09:49
ci-workflows#973 (bc62beb) makes check-key-credit read one exact-name
artifact instead of run history. check-intent now writes, scans and
uploads the gate's `marker` artifact (one-day retention) after a judge
step's credit-stop handoff, and the step that carried a record forward
is gone. pr-verify's ci-workflows pins move to bc62beb; run-claude-cli
changed only in a description there, so its lane.cjs still passes
--permission-prompts none (callers.test.mjs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pr-verify.yml now pins run-claude-cli and apply-claude-managed-settings
at bc62beb. Between 2476d5d and bc62beb, run-claude-cli changed only one
output description in action.yml and apply-claude-managed-settings did
not change, so cliArgs, LEG_ENV, CLI_VERSION and MANAGED_SETTINGS still
hold; parity.test.mjs fails until the constant moves with the pin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton kyle-sexton removed the do-not-merge Hard merge gate: do not merge while applied. label Nov 3, 2026
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 3, 2026 15:08
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T15:10:02.298219Z b301732 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 4bb6a73e3927ab24e2aa89ecf7c992992f054744: read 7 of 7 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #12799, perf(pr-verify): read the key's credit history only for a head it judges

No findings. I read every hunk of all 7 changed files: the changelog fragment, callers.test.mjs, artifact-retention.json, reconcile.test.mjs, probe/replay.mjs, pr-verify/workflow.test.mjs and .github/workflows/pr-verify.yml. pr-verify.yml is on the trusted-path list, so I also read the check-intent job at head from its if through the first judge step, and the gate on check-intent-subscription.

What I checked at head

  • Gate placement. Read the key's credit history (id: credit, pr-verify.yml#L300-L308) comes after the resolver, Check the trigger, agree and Hand the reads on, and runs only when steps.agree.outcome == 'success'. A run that stops at the resolver never reaches it. check-intent keeps actions: read, which the artifact listing needs.
  • When the gate skips the key. The skip turns off five steps: the model resolve, the credential check, the managed settings, runner isolation and the prompt write. credential.present stays empty, so no judge step runs. Fail without a verdict excludes skip-key == 'true', so the job ends green and hands on through the new skip-key output.
  • Subscription gate. The if at L2023-L2033 has balanced parentheses. It now needs needs.check-intent.result == 'success' on all three routes. So a failed key job still skips this job, and so does a key job skipped by the trigger gate. All 13 fallback-reason lines now read needs.check-intent.outputs.skip-key.
  • Marker steps. Write, scan and upload run in that order after every judge step. All three are continue-on-error. The upload requires scan-upload's clean == 'true', and the scan and the upload use the same path. If the gate fails and leaves marker empty, no marker is written. That fails open: later runs try the key, which is what the design intends.
  • Retention row. The literal name key-credit-stop-pr-verify.yml-check-intent has its own 1-day row. It does not match the pr-verify-* glob, so that row does not claim it.
  • Reconciler fixture. check-intent now ends success with its judge steps skipped, and the subscription leg decides held versus lost. This matches the updated test.
  • wake-pr-merge. It reads only check-intent's verdict and pr-number. On a skipped key the verdict is empty, so nothing is woken from the key leg. That is the same result as when the key job was skipped.

Not verified

I had no shell, so I ran no tests, no actionlint and no check-artifact-retention.sh. Every pass count in the PR body is the author's claim and remains unverified.

I could not read ci-workflows at bc62beb, so these points are also the author's claims:

  • the check-key-credit marker output and the markerName format;
  • that the gate caps KEY_CREDIT_WINDOW_HOURS at 24 or fewer, which the 1-day retention assumes;
  • the lane.cjs:404-405 permission-prompts line behind the SHA added to callers.test.mjs;
  • "run-claude-cli changed only one output description".

Closing issue

The PR closes no issue ("No related issue"), so there are no acceptance criteria to check.

Prior findings

The lane's earlier review, at b301732, reported 0 findings. This round adds the artifact-retention.json row and the marker steps, which I reviewed above. No prior finding repeats.

Spend

Spend goes down: fewer GITHUB_TOKEN reads, and one fewer VM per run that stops at the resolver. Model, turn and billing settings are unchanged.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 4bb6a73e3927ab24e2aa89ecf7c992992f054744: read 7 of 7 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #12799 (pr-verify credit gate folded into check-intent)

I found no security issues in this change.

critical-findings: 0

I read every hunk of all 7 changed files. .github/workflows/pr-verify.yml is a trusted path, so I read its whole change and what it gates. Since the last review at b301732, the change is a merge of main plus the new 1-day row for the marker in .github/artifact-retention.json.

What I checked

  • Token scope did not grow. check-intent and check-intent-subscription both still have only actions: read beside their read scopes (pr-verify.yml#L103-L106, #L2037-L2040). The removed check-intent-credit job only had actions: read. The marker upload uses the runner's artifact token, so no new permission is needed.
  • The gate still runs after the hardening steps. The credit read (id: credit, #L301) runs only when steps.agree.outcome == 'success'. That is after the kill switch, the resolver, the trigger gate and "Hand the reads on". When the key is skipped, these steps are skipped too: model resolve, credential check, managed settings, runner isolation and prompt (#L328-L368). Every judge step still needs steps.credential.outputs.present == 'true', so no judge runs on the key.
  • What reaches the subscription leg. The skip-key output is only compared to 'true' in the subscription job's if and in the fallback-reason literals, so it cannot inject anything. The job still needs needs.check-intent.result == 'success' (#L2033). That means a failed key job, including one with a permission denial, still leaves it skipped. The job also runs its own kill switch, resolver and trigger gate. The carried reads come through the same CARRIED and takeReads path the handoff case already used, so no new untrusted text reaches a lane that holds credentials.
  • "Fail without a verdict". It now skips only when skip-key == 'true', and that same value starts the subscription job. That job's own missing-verdict failure still fires, so the merge gate's need for a verdict is unchanged.
  • The marker artifact. It holds only $GITHUB_RUN_ID, is scanned by scan-upload before upload, and is kept for 1 day (the retention row is in artifact-retention.json). No secret goes into it.
  • Pin bump to bc62beb (run-claude-cli, apply-claude-managed-settings, check-key-credit). All three are full-SHA pins in the owner's own ci-workflows repository, so this does not widen any trust control. Pin hygiene is zizmor's lane. The callers.test.mjs addition is a test allowlist and loosens nothing at runtime.
  • Test files and the changeset (callers.test.mjs, reconcile.test.mjs, workflow.test.mjs, replay.mjs, the .changes entry) have no security surface.

Observation, not counted as a finding. The gate looks up the marker by artifact name across the whole repository. Any workflow run in the repository could upload an artifact with that name. The worst a fake marker can do is skip the key leg for up to the window, and the subscription leg then judges the same head under the same gates. That is no leak, no compromise and no charge, and under the owner's threat model only the owner's own actors can push workflows here.

Spend: none added. The change removes the gate's VM and its 2 to 4 calls from runs that stop at the resolver; it adds no model or credential use.

Embedded text in the PR context: none found.

kyle-sexton and others added 2 commits November 3, 2026 10:17
The marker upload took a computed name, so check-artifact-retention
assigned it to the job's computed-name family, pr-verify-* (90 days),
and its 1-day retention differed. It now uploads the gate's name,
key-credit-stop-pr-verify.yml-check-intent (check-key-credit gate.cjs
markerName at bc62beb), under its own 1-day row; the write step runs
only while the gate's marker output equals that name, so a gate that
names another marker uploads none.

The upload's if: now repeats the scan step's if: in parentheses before
the clean output, the shape the checker takes as gated by that scan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit dddfc1b Nov 3, 2026
87 checks passed
@kyle-sexton
kyle-sexton deleted the perf/pr-verify-skip-judged-head branch November 3, 2026 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant