Skip to content

fix(pstack): add TLA+ default and no-verifier fallback to create-formal-verification-skill - #721

Merged
cursor[bot] merged 1 commit into
mainfrom
lauren/formal-verification-tla-fit-wording-9247
Oct 19, 2026
Merged

cursor[bot] merged 1 commit into
mainfrom
lauren/formal-verification-tla-fit-wording-9247

Conversation

@poteto

@poteto poteto commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Why

create-formal-verification-skill gave no fallback for a language with no formal verifier. In a preregistered eval on a JavaScript library with no verifier, 6 of 9 runs with the old paragraph proved their claim only on a hand-written copy of the function. With the new paragraph, all 9 runs checked the shipped code.

What changed

  • The event-order sentence now makes TLA+ the default for that question in any language. The old sentence that recommended TLA+ for concurrent and distributed designs is gone.
  • When no verifier runs on the language, a new sentence tells the agent to say so and to test the real code with generated inputs, not a hand model of the function.
  • generate-pstack-harness-trees.mjs regenerated the claude-code harness copy.

Scope

Blast Radius

Every run of /create-formal-verification-skill now defaults to TLA+ for event-order questions, and tests the real code with generated inputs when no verifier runs on the language. The eval found no regression on a Java connection pool or a C parser.

Verification

  • The eval ran Claude, GPT, and Gemini on five open-source repos, with at most 3 runs per model and repo for each paragraph. One blind judge model gave each run one pass for each of four criteria that the run met. In both scoring views, both changes met their preregistered gates, with no regression and no harm on the other repos. Pooled, the new paragraph scores 3.62 passes per run against about 3.2, and it also ranks higher for Claude and for Gemini. About half of the pooled gain is in honest statements of what the check covers, which neither change mentions. Outside the JavaScript library, the old paragraph already met the target criteria in almost every run, so the eval cannot show a gain for the TLA+ change.
  • The open item is that the two scoring views disagree on whether GPT ranks the new paragraph above the old. Both views put GPT near a tie. The primary view gives the new paragraph 3.78 passes per run against 3.75, and the sensitivity view gives the new paragraph 3.67 against 3.79. The sensitivity view also scores finished runs that the preregistered rule excluded only for a failed gh call or for seeing the name of an empty sandbox folder. The difference comes from three GPT runs with the new paragraph that only the sensitivity view scores. Two of them listed worker threads, which the JavaScript circuit breaker that they checked does not have, and the eval found no link from that text to the new wording. The primary view also has no scored GPT run on the C parser.
  • node scripts/validate-plugins.mjs, node scripts/generate-pstack-harness-trees.mjs --check, and node scripts/check-pstack-docs.mjs pass.
Open in Web Open in Cursor 

Note

Low Risk
Documentation-only change to agent skill guidance; no runtime or product code paths.

Reviewed by Cursor Bugbot for commit 5bd8f4a. Bugbot is set up for automated code reviews on this repo. Configure here.

…al-verification-skill

The event-order sentence names TLA+ as the default in any language, and
the sentence that recommended TLA+ for concurrent and distributed
designs is gone. When no verifier runs on the language, the agent says
so and tests the real code with generated inputs, not a hand model of
the function.

The new paragraph is the exact text that a preregistered eval tested.
@poteto
poteto marked this pull request as ready for review October 19, 2026 01:49
@cursor
cursor Bot merged commit 6b78e9f into main Oct 19, 2026
2 checks passed
cursor Bot pushed a commit that referenced this pull request Oct 19, 2026
…ation-skill (#724)

* docs(pstack): match guide page 6 to create-formal-verification-skill's tool choice

#721 made TLA+ the default for event-order questions in any language, and
added a fallback: when no verifier runs on the language, the agent says so
and tests the real code with generated inputs. The guide's tool-choice
paragraph still described the old wording. Claude Code harness copy
regenerated.

Co-authored-by: lauren <poteto@users.noreply.github.com>

* chore(pstack): bump to 0.15.28

Co-authored-by: lauren <poteto@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
caelaxie added a commit to pstack-ports/devin-pstack that referenced this pull request Oct 20, 2026
<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
synced to
[`d457567`](cursor/plugins@d457567).
later commits on upstream `main` through
[`ddf824a`](cursor/plugins@ddf824a)
do not touch `pstack/`. version is `0.15.28` in `plugin.json`,
`.devin-plugin/plugin.json`, and the readme marker.

ported:
-
[`6b78e9f`](cursor/plugins@6b78e9f)
([#721](cursor/plugins#721))
create-formal-verification-skill: tla+ is the default for event-order
questions in any language, and when no verifier runs on the language the
agent says so and tests the real code with generated inputs.
-
[`7b353f7`](cursor/plugins@7b353f7)
([#716](cursor/plugins#716)) add
maintain-formal-verification-skill. create-formal-verification-skill
records covered code and the sync commit. poteto-help routing table,
guide pages 6 and 9, and recipes mention it. slash commands keep the
`/pstack:` prefix. `disable-model-invocation: true` stays `triggers:
[user]`.
-
[`d457567`](cursor/plugins@d457567)
([#724](cursor/plugins#724)) guide page 6
matches the create-formal-verification-skill tool choice, and the
version bumps to `0.15.28`. the same bump is in
`.devin-plugin/plugin.json` and the readme marker, which is where this
repo records the version.

skipped:
- `harness/pstack/claude-code/` copies in
[#721](cursor/plugins#721),
[#716](cursor/plugins#716), and
[#724](cursor/plugins#724). this port does not
carry the generated claude code tree.
- [#724](cursor/plugins#724)
`pstack/.cursor-plugin/plugin.json`. cursor packaging. the version bump
lands in `.devin-plugin/plugin.json` instead.
- [#716](cursor/plugins#716) upstream readme
skill-table row for `/maintain-formal-verification-skill`. this port's
readme is a pointer, not a copy of the upstream catalog. the version
marker is bumped. the skill is indexed in poteto-help, the guide, and
recipes.

<!-- CURSOR_AGENT_PR_BODY_END -->

<div><a
href="https://cursor.com/agents/bc-d0ddf2c8-7c38-5482-a7c0-b374846fcae6?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a>&nbsp;<a
href="https://cursor.com/background-agent?bcId=bc-d0ddf2c8-7c38-5482-a7c0-b374846fcae6&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a>&nbsp;</div>

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Kaitian Xie <caelaxie@users.noreply.github.com>
mstephenholl added a commit to mstephenholl/pstack-claude-codex that referenced this pull request Oct 20, 2026
Upstream: cursor/plugins@a66a8931c2dd

Closes #24. `merge-upstream.py` stopped on a conflict that needed a
person, so this pull request finishes that merge by hand. Upstream has
moved past the issue's `88d4751` since then, but the newer commits touch
nothing the port keeps.

## From upstream

- **`sharpen`**: back to upstream's original prompt (cursor#729). It asks four
expert questions and returns only the rewritten prompt.
- **`create-formal-verification-skill`**: TLA+ as the default for
ordering questions in any language, generated inputs when no verifier
supports the language (cursor#721), and a recorded coverage list and sync
commit in the generated skill. Its hand-off now points users to
`/maintain-formal-verification-skill`.
- **`maintain-formal-verification-skill`**: new (cursor#716). It brings a
project's formal verification back in line after code changes that
skipped the check, and says whether each failure is drift in the
verification or a bug in the code.

## Conflict resolved

- **`setup-pstack/SKILL.md`**: upstream's only change was a new step
offering a formal verification skill (cursor#715). The rest of upstream's side
of the conflict was its own Cursor version of this skill, which the port
replaces. I kept the port's version and added the offer as step 7,
worded the port's way: read
`~/.agents/pstack/skills/create-formal-verification-skill/SKILL.md` and
follow it, instead of invoking the skill wherever Cursor installed
pstack.

## Ported

- **`maintain-formal-verification-skill`** (separate commit): it finds
and updates a project-local verify skill, so it gets the
`pstack-harness` pointer, whose rules say where to search for project
skills in Claude Code and Codex. It is user-only, so it gets the Codex
policy file. It is added to `SKILLS` in `port/install.sh`. The README
now lists each verification skill next to its maintain skill, so
"Keeping that skill accurate" sits under the skill it refers to again.

## Proof

- 252 unit tests and `install_test.sh` pass, and `check.py` reports 0
problems across 54 pstack skills.
- Not run: `port/smoke/smoke.py`. It tests the installed copy in
`~/.agents/pstack` and needs Codex, which isn't on this machine. Its
scenarios don't cover `sharpen`, `setup-pstack`, or either formal
verification skill.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants