From 43873e4c60b70cbc93148c09156393d9f1d32cdd Mon Sep 17 00:00:00 2001 From: mdsmithaustin <15272014+mdsmithaustin@users.noreply.github.com> Date: Tue, 20 Oct 2026 07:22:48 -0600 Subject: [PATCH 1/3] fix(upstream-sync): require committed live results before completion --- PORTING.md | 6 +- .../results/2026-10-20-sync-88d47514.md | 44 ++++++ tools/test_upstream_sync.py | 145 ++++++++++++++++++ tools/upstream-sync.sh | 61 ++++++++ 4 files changed, 254 insertions(+), 2 deletions(-) create mode 100644 evals/promises/results/2026-10-20-sync-88d47514.md diff --git a/PORTING.md b/PORTING.md index c1959068..2b564ff8 100644 --- a/PORTING.md +++ b/PORTING.md @@ -211,12 +211,14 @@ This section is the only sync procedure. A runner gives it three inputs: the rec The manual-dispatch GitHub Action in `.github/workflows/upstream-sync.yml` is one runner. It needs the `CLAUDE_CODE_OAUTH_TOKEN` repository secret. Its checkout has `.no-mistakes.yaml` but no initialized gate, so it prepares the sync and reports publication as blocked. It does not push a branch or open a PR with its workflow token. A maintainer resumes the sync in an initialized checkout and publishes through the configured gate. The Action checkout is temporary; its local commits are not a durable handoff. Host-specific schedules, machine paths, and maintainer-specific tooling are intentionally not documented in this repository. -Run `tools/upstream-sync.sh` for the local scheduled runner. It requires Git, `gh`, Python 3, and authenticated Claude Code. It checks upstream and open sync PRs before invoking Claude. Each attempt starts with a fresh clone; attempts that invoke Claude get a unique branch and retain their clone. `PSTACK_SYNC_STATE_DIR` overrides the default `~/.local/state/pstack-sync` directory. The directory contains `sync.log` and an `attempt-*` directory for each run. An attempt that invokes Claude retains `repo/`, `prompt.txt`, `upstream-sha`, and `claude.log`. +Run `tools/upstream-sync.sh` for the local scheduled runner. It requires Git, `gh`, Python 3, and authenticated Claude Code. It checks upstream and open sync PRs before invoking Claude. Each attempt starts with a fresh clone; attempts that invoke Claude get a unique branch and retain their clone. The runner pins the clone's initial commit before Claude runs and saves it in `base-sha`. `PSTACK_SYNC_STATE_DIR` overrides the default `~/.local/state/pstack-sync` directory. The directory contains `sync.log` and an `attempt-*` directory for each run. An attempt that invokes Claude retains `repo/`, `base-sha`, `prompt.txt`, `upstream-sha`, and `claude.log`. The local runner records the user's explicit authorization to bypass no-mistakes only to publish that attempt's sync branch and create its draft PR. This exception applies only to that runner's publication in step 10. All local checks remain required. It does not authorize marking the PR ready, merging, pushing to `main`, or force-pushing. The Action and other publication routes keep their existing gate requirements. Claude's exit code alone does not establish success. The runner requires clean committed work, the intended upstream SHA in the commit, and a live open draft PR to `main`. The PR must belong to this repository and match the attempt's branch and commit SHA. A failure after Claude starts retains the clone and exits nonzero. Unchanged-upstream skips, open-PR skips, and blocks caused by a prior unfinished attempt remove only the new untouched clone and retain the attempt directory. An unfinished attempt blocks another Claude run after the unchanged-upstream and open-PR checks. To recover, inspect `sync.log`, resume the retained `repo/` using `prompt.txt`, and resolve its work. Then move the whole resolved attempt outside the state directory before retrying. Keep that copy until its work is safe. Successful attempts remain retained with a `complete` file containing the verified PR URL. Harmless skips also have a `complete` file that records their reason. +Before checking the PR, the runner independently reads the committed promise ledgers at the pinned initial commit and candidate head. Missing or malformed ledger data blocks completion. It takes live owner paths from both ledgers and compares changed paths without rename detection. A changed or deleted owner still triggers the check after its promise is removed. A changed live owner requires an added or updated `.md` file under `evals/promises/results/` with nonempty committed contents. An unchanged report, a mode-only change, or a deleted report does not qualify. When no live owner changes, the runner requires no new report. A missing-report refusal names the changed promises and owners, retains the work, and leaves the attempt incomplete. This check proves that the sync committed a report. Reviewers still assess the report's truth, case coverage, and upstream prose fidelity. The check does not establish that every case passed. + 1. Add the `upstream` remote `https://github.com/cursor/plugins.git` if the checkout lacks it, and run `git fetch upstream main`. Without a runner, the recorded commit is the SHA in `.github/upstream-sha` (originally the import commit), and the latest commit is `git rev-list -1 upstream/main -- pstack`. When the two match, there is nothing to sync. 2. Study `git log ..upstream/main -- pstack/` and `git diff ..upstream/main -- pstack/`. Map upstream `pstack/skills/` to `skills/`, except `tdd` to `poteto-tdd` and `teach` to `poteto-teach`. Map upstream docs to this repo's root files where they exist. 3. Apply each incoming hunk with the substitution map. Merge it into the port's file and never overwrite the file, so the port additions survive: GitNexus nudges, pstack-harness pointers, `agents/openai.yaml` files, and renamed skill references. Apply the map's `disable-model-invocation` row to incoming frontmatter and its sibling-skill-load row to every incoming load of another skill. A passage that quotes what the user types goes in `USER_FACING` in `tools/skill-references.py`. Every SKILL.md frontmatter `name:` must equal its directory name. Every port addition and every "on sync" rule elsewhere in this file binds this step. @@ -234,7 +236,7 @@ Claude's exit code alone does not establish success. The runner requires clean c - `tools/gate-check.sh test`. It needs Bun. `tools/gate-check.sh` picks its own Python, and `PSTACK_GATE_PYTHON` overrides it, as `README.md` describes. On a host with mise shims on `PATH`, put the real `bun` and `rg` ahead of the shims, and say in the PR body which tools ran. #166 tracks one pinned entry point per test tier. - `python3 evals/promises/ledger.py audit --upstream-ref upstream/main`. An "unclassified upstream unit" is a guide sentence that upstream added or reworded. An unclassified or stale port unit is guide text that this sync changed, such as a count or a link from step 5. Classify it in `evals/promises/ledger.json` per `evals/promises/README.md`, and bind each new promise to a quote, a test, or a case. Port the sentence into `docs/guide` with the substitution map, unless you classify it `cursor-only` with its reason or `substituted` with the port units that replace it. A "quote no longer in" line names a guide promise whose skill text the sync rewrote. Keep the promise true, change the guide and the ledger together, or record the change for human judgment in the PR body. A quote recorded for human judgment can leave other checks failing that `main` passes. These are the lint tier's `promise ledger` check and the `AuditTest` cases in `evals/promises/test_ledger.py`, such as `test_the_committed_ledger_audits_clean`. These local failures may remain only when the complete listed set of held quotes explains every remaining cause. List each quote and each affected check in the PR body. Any unrelated failure blocks this local exception. Recording the hold waives neither authoritative CI nor no-mistakes Lint. Keep gated publication pending when the configured gate cannot accept the gap through its supported authority. Do not bypass or weaken checks. When you reword a guide sentence that ports an upstream unit, point that unit's `ported_as` at the new unit. List every ledger change in the PR body under "Guide promises". - `skill-ci check` -8. After a sync that changes a live promise's owner files, rerun that promise's cases on all four harnesses with `evals/promises/live.py run`. When the runner lacks a harness, list the cases that did not run in the PR body as needing judgment. Render the matrix with `live.py report`, save it as `evals/promises/results/.md`, and compare it with the newest earlier file there, if one exists. Read that file's "Comparison baseline" section first. `2026-10-03.md` measures the tree before its own fixes, so compare against its "Fixed in this stack" rows, and run a full screen on the current tree before treating any other row as a baseline. +8. After a sync that changes a live promise's owner files, rerun that promise's cases on all four harnesses with `evals/promises/live.py run`. When the runner lacks a harness, list the cases that did not run in the report and PR body as needing judgment. Render the matrix with `live.py report` and save it as `evals/promises/results/.md`. Record the final skills revision evaluated, the matrix, the baseline comparison, and explicit judgments for missing cases or harnesses. Save and commit the report even when some cases cannot run. Compare it with the newest earlier file there, if one exists, or state that no earlier baseline exists. Read that file's "Comparison baseline" section first. `2026-10-03.md` measures the tree before its own fixes, so compare against its "Fixed in this stack" rows, and run a full screen on the current tree before treating any other row as a baseline. 9. Write the latest upstream commit to `.github/upstream-sha`. 10. Create the runner-provided branch, or `sync/upstream-` without one, run `git add -A`, and commit everything with a message that summarizes the upstream commits ported. A ledger classification from step 7 goes in the same commit. For `tools/upstream-sync.sh`, use the runner-provided branch and the scheduled draft-only publication exception described above. For every other route, resolve publication through **Gated publish** in `skills/poteto-mode/playbooks/opening-a-pr.md`. Except for that local scheduled runner, a runner route never bypasses a configured gate. For those other routes, that section owns publication and PR readiness in a gated checkout. When their configured gate is missing, uninitialized, or broken, stop before publication and report the upstream commits, local branch and head, checks, and required gated handoff. The Action follows this stop path; a workflow token is not a gate exemption. On the scheduled runner's authorized route or a supported ungated path, publish a draft PR to `main` through the runner's route. Without a route, push the branch with `git push -u origin `, then run `gh pr create --draft --repo github.com/mdsmithaustin/pstack`. The explicit host and repository keep `gh` from selecting cursor/plugins through the `upstream` remote. Every published outcome requires a PR, because the `main` ruleset grants no bypass. Never push to `main` and never force-push. 11. Prepare the PR body, or include the same information in the report when publication is blocked. List each upstream commit, the transformation applied to it, the checks that ran, and anything that needs human judgment. When you are unsure whether a change is portable, include it with a "needs judgment" note instead of dropping it. When the only change is the `.github/upstream-sha` bump, say that nothing else was portable. Report only CI checks that actually ran; local checks do not establish that the Docker suites or other CI-only checks passed. diff --git a/evals/promises/results/2026-10-20-sync-88d47514.md b/evals/promises/results/2026-10-20-sync-88d47514.md new file mode 100644 index 00000000..d19d0a4a --- /dev/null +++ b/evals/promises/results/2026-10-20-sync-88d47514.md @@ -0,0 +1,44 @@ +# Sharpen sync verification, 2026-10-20 + +## Examined revision + +[Port PR #278](https://github.com/mdsmithaustin/pstack/pull/278) ports upstream [#729](https://github.com/cursor/plugins/pull/729), commit `88d47514b82e7c44f00f2b2a72ff6dd12158038c`. The tested portable skills revision is `bf97cceece38422b281db97a403723d050bddd1c`. + +This receipt records the checks that ran for that revision. It does not claim that the port has merged or that another revision has been tested. The results were initially retained outside Git and are committed here to close that reporting gap. + +## Changed live owner + +The port changes `skills/poteto-help/SKILL.md`, an owner of `poteto-help-finds-need-and-routes`. Both bound cases ran on every supported harness against the committed skills revision above. No case or harness was unavailable. + +| Case | Claude Code | Codex | Hermes | Grok | +|---|---|---|---|---| +| `poteto-help-run` | PASS | PASS | PASS | PASS | +| `poteto-help-unclear-run` | PASS | PASS | PASS | PASS | + +The first case checks branch-review routing and a copyable prompt. The second checks a clarifying question for an unclear request. These are routing results. They do not establish `/sharpen` compliance on all four harnesses. + +The matrix came from `evals/promises/live.py run` with both cases, each harness, and `--skills-at bf97cceece38422b281db97a403723d050bddd1c`. `live.py report` returned eight PASS verdicts for the promise. Nonzero command exit or a non-PASS applicable verdict is a failure signal. + +## Direct sharpen scenarios + +The no-mistakes Test phase also ran three native Codex scenarios against the same skill revision. Each passed: + +- A prompt-rewrite request returned the four framing answers and a rewritten prompt. +- A request containing an instruction to write `onboarding.md` left that file unwritten. Its rewritten prompt left the database, language, and hosting choices open. +- A rewrite-only help request received a `/sharpen` recommendation and a copyable prompt. + +The gate retained actual replies, grades, project-state checks, an HTML matrix, and a repeatable scenario driver. Inspect them through `no-mistakes axi logs --run 01M5CNXT0E1AT5BF19ADNK351A --step test --full`. A nonzero exit or unavailable evidence is a verification gap. The recorded Test verdict is `go`, with no findings. These direct scenarios ran on Codex only. + +## Comparison baseline + +The previous sync receipt is [2026-10-19-sync-d457567.md](2026-10-19-sync-d457567.md). Its help-routing row passed twice on Codex and twice on Hermes, with no Claude Code or Grok result. This rerun also passed both cases on Codex and Hermes and added passing results on Claude Code and Grok. No regression appeared in those two cases. + +The previous receipt's comparison section says the 2026-10-03 full screen measured the tree before its own fixes. Only its fixed rows are a valid baseline. This focused rerun does not close unrelated evaluation debt listed in earlier receipts. + +## Other verification + +The upstream-aware ledger audit returned zero problems. `mise run gate-lint`, `mise run gate-test`, type checks, and `skill-ci check` exited zero. The Python suites ran 248 tools tests, 822 promise tests, and 185 setup tests. Bun reported 541 passed, one skipped, and zero failed. Each check fails on nonzero exit; the suite summaries must also show executed tests. + +A different-agent documentation review passed on the six upstream prose changes and portable metadata. The reviewed diff fingerprint is SHA-256 `185a20e1bcb9ca30ba1f93788499a88403c7b6ca538eef5f4a14edf247af2228`. + +All seven GitHub checks passed on the PR head above. The no-mistakes run `01M5CNXT0E1AT5BF19ADNK351A` returned `checks-passed`. The PR remains a draft and unmerged at the time of this receipt. diff --git a/tools/test_upstream_sync.py b/tools/test_upstream_sync.py index fee1022b..24949d43 100644 --- a/tools/test_upstream_sync.py +++ b/tools/test_upstream_sync.py @@ -1,6 +1,7 @@ from __future__ import annotations import argparse +import json import os import subprocess import sys @@ -51,6 +52,19 @@ def setUp(self) -> None: (self.origin / ".github").mkdir() (self.origin / ".github" / "upstream-sha").write_text(self.recorded + "\n") (self.origin / "PORTING.md").write_text("Fixture porting contract.\n") + ledger = self.origin / "evals" / "promises" / "ledger.json" + ledger.parent.mkdir(parents=True) + ledger.write_text(json.dumps({"promises": { + "architect-checkpoint-opt-in": { + "check": "live", "owners": ["skills/architect/SKILL.md"], + }, + }})) + owner = self.origin / "skills" / "architect" / "SKILL.md" + owner.parent.mkdir(parents=True) + owner.write_text("Original live promise owner.\n") + report = self.origin / "evals" / "promises" / "results" / "earlier.md" + report.parent.mkdir() + report.write_text("Earlier live results.\n") self.git("add", ".", cwd=self.origin) self.git("commit", "-qm", "Initial port", cwd=self.origin) self.git("config", "--global", f"url.{self.origin}.insteadOf", @@ -123,8 +137,19 @@ def setUp(self) -> None: subprocess.run(["git", "checkout", "-qb", branch], check=True) if mode != "stale_recorded": Path(".github/upstream-sha").write_text(os.environ["PSTACK_TEST_LATEST"] + "\\n") + for name, contents in json.loads(os.environ.get("PSTACK_TEST_FILES", "{}")).items(): + path = Path(name) + if contents is None: + path.unlink() + else: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(contents) + if os.environ.get("PSTACK_TEST_CHMOD_REPORT"): + Path("evals/promises/results/earlier.md").chmod(0o755) subprocess.run(["git", "add", "-A"], check=True) subprocess.run(["git", "commit", "-qm", "Sync fixture"], check=True) + if os.environ.get("PSTACK_TEST_MOVE_BASE_REF"): + subprocess.run(["git", "branch", "-f", "main", "HEAD"], check=True) subprocess.run(["git", "push", "-qu", "origin", branch], check=True) head = subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip() pulls = [{ @@ -183,8 +208,128 @@ def test_published_draft_matches_committed_sha_and_keeps_attempt(self) -> None: self.assertEqual((clone.parent / "complete").read_text(), "https://github.com/mdsmithaustin/pstack/pull/123\n") self.assertEqual((clone.parent / "upstream-sha").read_text(), self.latest + "\n") + self.assertEqual((clone.parent / "base-sha").read_text(), + self.git("rev-parse", "HEAD", cwd=self.origin) + "\n") + self.assertEqual((clone / "skills/architect/SKILL.md").read_text(), + "Original live promise owner.\n") self.assertEqual((clone.parent / "claude.log").read_text(), "Draft PR published.\n") + def test_changed_live_owner_requires_committed_results(self) -> None: + result = self.run_sync( + PSTACK_TEST_MODE="published", + PSTACK_TEST_FILES=json.dumps({"skills/architect/SKILL.md": "Changed owner.\n"}), + PSTACK_TEST_MOVE_BASE_REF="1", + ) + self.assertNotEqual(result.returncode, 0) + clone = Path((self.root / "agent-cwd").read_text()) + self.assertTrue(clone.is_dir()) + self.assertEqual((clone / "skills/architect/SKILL.md").read_text(), "Changed owner.\n") + self.assertFalse((clone.parent / "complete").exists()) + self.assertEqual((clone.parent / "base-sha").read_text(), + self.git("rev-parse", "HEAD", cwd=self.origin) + "\n") + self.assertEqual(self.git("rev-parse", "main", cwd=clone), + self.git("rev-parse", "HEAD", cwd=clone)) + log = (self.state / "sync.log").read_text() + self.assertIn("architect-checkpoint-opt-in", log) + self.assertIn("skills/architect/SKILL.md", log) + self.assertIn("committed results Markdown", log) + retry = self.run_sync(PSTACK_TEST_MODE="published") + self.assertNotEqual(retry.returncode, 0) + self.assertEqual((self.root / "agent-calls").read_text(), "called\n") + + def test_added_or_updated_committed_results_allow_live_owner_change(self) -> None: + for report in ("new.md", "earlier.md"): + with self.subTest(report=report): + result = self.run_sync( + PSTACK_TEST_MODE="published", PSTACK_SYNC_STATE_DIR=str(self.root / report), + PSTACK_TEST_FILES=json.dumps({ + "skills/architect/SKILL.md": "Changed owner.\n", + f"evals/promises/results/{report}": "Current results.\n", + }), + ) + self.assertEqual(result.returncode, 0) + clone = Path((self.root / "agent-cwd").read_text()) + self.assertTrue((clone.parent / "complete").is_file()) + + def test_invalid_results_do_not_satisfy_live_evidence(self) -> None: + reports = { + "empty": {"evals/promises/results/new.md": " \n\t"}, + "unrelated": {"docs/results.md": "Current results.\n", + "evals/promises/results/new.txt": "Current results.\n"}, + "deleted": {"evals/promises/results/earlier.md": None}, + "mode_only": {}, + } + for case, files in reports.items(): + with self.subTest(case=case): + result = self.run_sync( + PSTACK_TEST_MODE="published", PSTACK_SYNC_STATE_DIR=str(self.root / case), + PSTACK_TEST_CHMOD_REPORT="1" if case == "mode_only" else "", + PSTACK_TEST_FILES=json.dumps({"skills/architect/SKILL.md": "Changed owner.\n", **files}), + ) + self.assertNotEqual(result.returncode, 0) + clone = Path((self.root / "agent-cwd").read_text()) + self.assertFalse((clone.parent / "complete").exists()) + self.assertIn("committed results Markdown", (clone.parent.parent / "sync.log").read_text()) + + def test_added_and_deleted_live_owners_require_results(self) -> None: + cases = { + "added": { + "evals/promises/ledger.json": json.dumps({"promises": { + "new-live-promise": {"check": "live", "owners": ["skills/new/SKILL.md"]}, + }}), + "skills/new/SKILL.md": "New live owner.\n", + }, + "deleted": { + "evals/promises/ledger.json": json.dumps({"promises": {}}), + "skills/architect/SKILL.md": None, + }, + } + for case, files in cases.items(): + with self.subTest(case=case): + result = self.run_sync( + PSTACK_TEST_MODE="published", PSTACK_SYNC_STATE_DIR=str(self.root / case), + PSTACK_TEST_FILES=json.dumps(files), + ) + self.assertNotEqual(result.returncode, 0) + clone = Path((self.root / "agent-cwd").read_text()) + self.assertFalse((clone.parent / "complete").exists()) + self.assertIn("committed results Markdown", (clone.parent.parent / "sync.log").read_text()) + + def test_missing_or_malformed_committed_ledger_refuses_completion(self) -> None: + for case, contents in (("missing", None), ("malformed", "not JSON"), + ("wrong_shape", '{"promises": []}')): + with self.subTest(case=case): + result = self.run_sync( + PSTACK_TEST_MODE="published", PSTACK_SYNC_STATE_DIR=str(self.root / case), + PSTACK_TEST_FILES=json.dumps({"evals/promises/ledger.json": contents}), + ) + self.assertNotEqual(result.returncode, 0) + clone = Path((self.root / "agent-cwd").read_text()) + self.assertFalse((clone.parent / "complete").exists()) + self.assertIn("cannot verify committed promise ledger", + (clone.parent.parent / "sync.log").read_text()) + + def test_invalid_base_ledger_cannot_be_repaired_to_bypass_guard(self) -> None: + ledger = self.origin / "evals" / "promises" / "ledger.json" + contents = ledger.read_text() + for case in ("missing", "malformed"): + with self.subTest(case=case): + if case == "missing": + ledger.unlink() + else: + ledger.write_text("not JSON") + self.git("add", "-A", cwd=self.origin) + self.git("commit", "-qm", "Invalid baseline ledger", cwd=self.origin) + result = self.run_sync( + PSTACK_TEST_MODE="published", PSTACK_SYNC_STATE_DIR=str(self.root / case), + PSTACK_TEST_FILES=json.dumps({"evals/promises/ledger.json": contents}), + ) + self.assertNotEqual(result.returncode, 0) + clone = Path((self.root / "agent-cwd").read_text()) + self.assertFalse((clone.parent / "complete").exists()) + self.assertIn("cannot verify committed promise ledger", + (clone.parent.parent / "sync.log").read_text()) + def test_incomplete_sync_is_rejected_even_when_agent_says_published(self) -> None: for mode in ("stale_recorded", "dirty", "wrong_branch", "commit_only"): with self.subTest(mode=mode): diff --git a/tools/upstream-sync.sh b/tools/upstream-sync.sh index 5ec3bcfc..1fd056e9 100755 --- a/tools/upstream-sync.sh +++ b/tools/upstream-sync.sh @@ -17,6 +17,8 @@ repo="$attempt/repo" git clone --quiet "https://github.com/$REPO.git" "$repo" cd "$repo" +base=$(git rev-parse --verify 'HEAD^{commit}') +printf '%s\n' "$base" >"$attempt/base-sha" git remote add upstream https://github.com/cursor/plugins.git git fetch upstream main --quiet @@ -98,6 +100,65 @@ if [ -n "$status" ]; then exit 1 fi +python3 - "$base" "$head" <<'PY' +import json +import subprocess +import sys + +base, head = sys.argv[1:] +live_owners = {} +for revision in (base, head): + try: + ledger = json.loads(subprocess.check_output( + ["git", "show", f"{revision}:evals/promises/ledger.json"], + )) + promises = ledger["promises"] + if not isinstance(promises, dict): + raise ValueError("promises must be a mapping") + for promise_id, promise in promises.items(): + if not isinstance(promise, dict) or not isinstance(promise.get("check"), str): + raise ValueError(f"invalid promise {promise_id}") + if promise["check"] != "live": + continue + owners = promise["owners"] + if not isinstance(owners, list) or not owners or any( + not isinstance(owner, str) or not owner for owner in owners + ): + raise ValueError(f"invalid live owners for {promise_id}") + for owner in owners: + live_owners.setdefault(owner, set()).add(promise_id) + except (subprocess.CalledProcessError, ValueError, KeyError, TypeError) as error: + sys.exit(f"cannot verify committed promise ledger at {revision}: {error}") + +changed = set(subprocess.check_output([ + "git", "diff", "--no-renames", "--name-only", "-z", base, head, +]).decode().split("\0")) +changed_owners = sorted(changed.intersection(live_owners)) +if changed_owners: + reports = subprocess.check_output([ + "git", "diff", "--no-renames", "--diff-filter=AM", "--name-only", "-z", + base, head, "--", "evals/promises/results/", + ]).decode().split("\0") + for report in reports: + if not report.endswith(".md"): + continue + previous_blob = subprocess.run( + ["git", "rev-parse", "--verify", f"{base}:{report}"], capture_output=True, + ) + current_blob = subprocess.check_output([ + "git", "rev-parse", "--verify", f"{head}:{report}", + ]) + if previous_blob.returncode == 0 and previous_blob.stdout == current_blob: + continue + if subprocess.check_output(["git", "show", f"{head}:{report}"]).strip(): + break + else: + for owner in changed_owners: + print(f"changed live promise owner {owner}: {', '.join(sorted(live_owners[owner]))}", + file=sys.stderr) + sys.exit("sync lacks added or modified nonempty committed results Markdown under evals/promises/results/") +PY + gh api --method GET "repos/$REPO/pulls" -f state=open -f "head=${REPO%/*}:$branch" -f base=main >"$attempt/pull-requests.json" python3 - "$attempt/pull-requests.json" "$REPO" "$branch" "$head" >"$attempt/verified-pr" <<'PY' import json From 19a3eaef9e23db635ed2890ba0e9e3e92754ca24 Mon Sep 17 00:00:00 2001 From: mdsmithaustin <15272014+mdsmithaustin@users.noreply.github.com> Date: Tue, 20 Oct 2026 12:54:09 -0600 Subject: [PATCH 2/3] no-mistakes(review): Correct receipt baseline and cover static-only sync completion --- .../results/2026-10-20-sync-88d47514.md | 4 ++-- tools/test_upstream_sync.py | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/evals/promises/results/2026-10-20-sync-88d47514.md b/evals/promises/results/2026-10-20-sync-88d47514.md index d19d0a4a..fa61ed09 100644 --- a/evals/promises/results/2026-10-20-sync-88d47514.md +++ b/evals/promises/results/2026-10-20-sync-88d47514.md @@ -31,9 +31,9 @@ The gate retained actual replies, grades, project-state checks, an HTML matrix, ## Comparison baseline -The previous sync receipt is [2026-10-19-sync-d457567.md](2026-10-19-sync-d457567.md). Its help-routing row passed twice on Codex and twice on Hermes, with no Claude Code or Grok result. This rerun also passed both cases on Codex and Hermes and added passing results on Claude Code and Grok. No regression appeared in those two cases. +The previous sync receipt is [2026-10-20-sync-7a9a3ddb.md](2026-10-20-sync-7a9a3ddb.md). Its `poteto-help-finds-need-and-routes` row records PASS (2) on all four harnesses. This rerun matches that row with both cases passing on every harness. No regression appeared in those two cases. -The previous receipt's comparison section says the 2026-10-03 full screen measured the tree before its own fixes. Only its fixed rows are a valid baseline. This focused rerun does not close unrelated evaluation debt listed in earlier receipts. +The previous receipt's comparison section establishes that wave as the baseline for the next sync and lists its known failures and unobserved behavior. This focused rerun does not close unrelated evaluation debt listed in earlier receipts. ## Other verification diff --git a/tools/test_upstream_sync.py b/tools/test_upstream_sync.py index 24949d43..92e3b5a7 100644 --- a/tools/test_upstream_sync.py +++ b/tools/test_upstream_sync.py @@ -58,10 +58,16 @@ def setUp(self) -> None: "architect-checkpoint-opt-in": { "check": "live", "owners": ["skills/architect/SKILL.md"], }, + "static-contract": { + "check": "static", "owners": ["skills/static-check/SKILL.md"], + }, }})) owner = self.origin / "skills" / "architect" / "SKILL.md" owner.parent.mkdir(parents=True) owner.write_text("Original live promise owner.\n") + static_owner = self.origin / "skills" / "static-check" / "SKILL.md" + static_owner.parent.mkdir(parents=True) + static_owner.write_text("Original static promise owner.\n") report = self.origin / "evals" / "promises" / "results" / "earlier.md" report.parent.mkdir() report.write_text("Earlier live results.\n") @@ -214,6 +220,18 @@ def test_published_draft_matches_committed_sha_and_keeps_attempt(self) -> None: "Original live promise owner.\n") self.assertEqual((clone.parent / "claude.log").read_text(), "Draft PR published.\n") + def test_changed_static_owner_completes_without_new_results(self) -> None: + result = self.run_sync( + PSTACK_TEST_MODE="published", + PSTACK_TEST_FILES=json.dumps({"skills/static-check/SKILL.md": "Changed static owner.\n"}), + ) + self.assertEqual(result.returncode, 0, result.stderr) + clone = Path((self.root / "agent-cwd").read_text()) + self.assertEqual(self.git("show", "HEAD:skills/static-check/SKILL.md", cwd=clone), + "Changed static owner.") + self.assertEqual((clone.parent / "complete").read_text(), + "https://github.com/mdsmithaustin/pstack/pull/123\n") + def test_changed_live_owner_requires_committed_results(self) -> None: result = self.run_sync( PSTACK_TEST_MODE="published", From 20d292d0396285eba8b7bb7387156ea710cf2159 Mon Sep 17 00:00:00 2001 From: mdsmithaustin <15272014+mdsmithaustin@users.noreply.github.com> Date: Tue, 20 Oct 2026 14:23:32 -0600 Subject: [PATCH 3/3] no-mistakes(review): Clarify historical sharpen evidence and report content requirements --- PORTING.md | 2 +- evals/promises/results/2026-10-20-sync-88d47514.md | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/PORTING.md b/PORTING.md index 2b564ff8..04ea7d7a 100644 --- a/PORTING.md +++ b/PORTING.md @@ -236,7 +236,7 @@ Before checking the PR, the runner independently reads the committed promise led - `tools/gate-check.sh test`. It needs Bun. `tools/gate-check.sh` picks its own Python, and `PSTACK_GATE_PYTHON` overrides it, as `README.md` describes. On a host with mise shims on `PATH`, put the real `bun` and `rg` ahead of the shims, and say in the PR body which tools ran. #166 tracks one pinned entry point per test tier. - `python3 evals/promises/ledger.py audit --upstream-ref upstream/main`. An "unclassified upstream unit" is a guide sentence that upstream added or reworded. An unclassified or stale port unit is guide text that this sync changed, such as a count or a link from step 5. Classify it in `evals/promises/ledger.json` per `evals/promises/README.md`, and bind each new promise to a quote, a test, or a case. Port the sentence into `docs/guide` with the substitution map, unless you classify it `cursor-only` with its reason or `substituted` with the port units that replace it. A "quote no longer in" line names a guide promise whose skill text the sync rewrote. Keep the promise true, change the guide and the ledger together, or record the change for human judgment in the PR body. A quote recorded for human judgment can leave other checks failing that `main` passes. These are the lint tier's `promise ledger` check and the `AuditTest` cases in `evals/promises/test_ledger.py`, such as `test_the_committed_ledger_audits_clean`. These local failures may remain only when the complete listed set of held quotes explains every remaining cause. List each quote and each affected check in the PR body. Any unrelated failure blocks this local exception. Recording the hold waives neither authoritative CI nor no-mistakes Lint. Keep gated publication pending when the configured gate cannot accept the gap through its supported authority. Do not bypass or weaken checks. When you reword a guide sentence that ports an upstream unit, point that unit's `ported_as` at the new unit. List every ledger change in the PR body under "Guide promises". - `skill-ci check` -8. After a sync that changes a live promise's owner files, rerun that promise's cases on all four harnesses with `evals/promises/live.py run`. When the runner lacks a harness, list the cases that did not run in the report and PR body as needing judgment. Render the matrix with `live.py report` and save it as `evals/promises/results/.md`. Record the final skills revision evaluated, the matrix, the baseline comparison, and explicit judgments for missing cases or harnesses. Save and commit the report even when some cases cannot run. Compare it with the newest earlier file there, if one exists, or state that no earlier baseline exists. Read that file's "Comparison baseline" section first. `2026-10-03.md` measures the tree before its own fixes, so compare against its "Fixed in this stack" rows, and run a full screen on the current tree before treating any other row as a baseline. +8. After a sync that changes a live promise's owner files, rerun that promise's cases on all four harnesses with `evals/promises/live.py run`. When the runner lacks a harness, list the cases that did not run in the report and PR body as needing judgment. Render the matrix with `live.py report` and save it as `evals/promises/results/.md`. Record the final skills revision evaluated, the matrix, the baseline comparison, and explicit judgments for missing cases or harnesses. These are report contents, not a fixed format; the local runner checks report presence only, not format or meaning. Save and commit the report even when some cases cannot run. Compare it with the newest earlier file there, if one exists, or state that no earlier baseline exists. Read that file's "Comparison baseline" section first. `2026-10-03.md` measures the tree before its own fixes, so compare against its "Fixed in this stack" rows, and run a full screen on the current tree before treating any other row as a baseline. 9. Write the latest upstream commit to `.github/upstream-sha`. 10. Create the runner-provided branch, or `sync/upstream-` without one, run `git add -A`, and commit everything with a message that summarizes the upstream commits ported. A ledger classification from step 7 goes in the same commit. For `tools/upstream-sync.sh`, use the runner-provided branch and the scheduled draft-only publication exception described above. For every other route, resolve publication through **Gated publish** in `skills/poteto-mode/playbooks/opening-a-pr.md`. Except for that local scheduled runner, a runner route never bypasses a configured gate. For those other routes, that section owns publication and PR readiness in a gated checkout. When their configured gate is missing, uninitialized, or broken, stop before publication and report the upstream commits, local branch and head, checks, and required gated handoff. The Action follows this stop path; a workflow token is not a gate exemption. On the scheduled runner's authorized route or a supported ungated path, publish a draft PR to `main` through the runner's route. Without a route, push the branch with `git push -u origin `, then run `gh pr create --draft --repo github.com/mdsmithaustin/pstack`. The explicit host and repository keep `gh` from selecting cursor/plugins through the `upstream` remote. Every published outcome requires a PR, because the `main` ruleset grants no bypass. Never push to `main` and never force-push. 11. Prepare the PR body, or include the same information in the report when publication is blocked. List each upstream commit, the transformation applied to it, the checks that ran, and anything that needs human judgment. When you are unsure whether a change is portable, include it with a "needs judgment" note instead of dropping it. When the only change is the `.github/upstream-sha` bump, say that nothing else was portable. Report only CI checks that actually ran; local checks do not establish that the Docker suites or other CI-only checks passed. diff --git a/evals/promises/results/2026-10-20-sync-88d47514.md b/evals/promises/results/2026-10-20-sync-88d47514.md index fa61ed09..df22764e 100644 --- a/evals/promises/results/2026-10-20-sync-88d47514.md +++ b/evals/promises/results/2026-10-20-sync-88d47514.md @@ -6,6 +6,8 @@ This receipt records the checks that ran for that revision. It does not claim that the port has merged or that another revision has been tested. The results were initially retained outside Git and are committed here to close that reporting gap. +The eight canonical help grades and three Codex scenarios at `bf97cceece38422b281db97a403723d050bddd1c` are historical evidence from before the later local sharpen boundary fixes. The [final sharpen validation receipt](2026-10-20T1935-sharpen-729-final.md) is the current sharpen validation record, tested at `815e15cbfbaba27eade02a1b1f91270b0588a63b` after both fixes. The outcomes and PR status below describe the original run. + ## Changed live owner The port changes `skills/poteto-help/SKILL.md`, an owner of `poteto-help-finds-need-and-routes`. Both bound cases ran on every supported harness against the committed skills revision above. No case or harness was unavailable.