Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/gap-cli-block-per-law.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@tanstack/oracles': minor
---

A failing law no longer blocks the gap report for its whole file. In `oracle-gaps`, a test of an oracle file that fails with no mutant active judges nothing: it is left out per test, as a nondeterministic test is, and the passing tests of its file still judge the mutants the file owns. A failing law keeps its reach and its work: its calls count toward its file's call budget, and its file's mutant runs leave it out through a selection that reproduces the rest. A test of a `describe` whose `beforeAll` or `afterAll` throws is a failing law too. An owner file that fails outside its tests (it does not load, or a file-level hook throws) judges nothing, and another owner file's passing tests still judge the mutants they share. A mutant that no deterministic passing owner test reached is `blocked` by its owners' failing laws that reached it, and by owner files that judge nothing; its directive names them. The baseline prints one `oracle-gaps: blocked` line per failing law, named by its file and test names (or one per broken file), `results.json` lists each failing file's `laws`, and the report stays provisional.
19 changes: 14 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,10 +48,19 @@ agent.
The report, on stdout and in `summary.md`, leads with the baseline: each
test file's result with its test counts (ordinary files too), and for a
failing file its failing test, its error, the law and counterexample of a
law violation, and a `replay:` command. An oracle file that fails with no
mutant active blocks only the mutants it owns: each that no passing owner
kills is a `blocked` gap, a `blocked` line names the file and its failing
test, the rest are judged, and the report says it is provisional. With
law violation, and a `replay:` command. A test of an oracle file that
fails with no mutant active (a failing law) judges nothing: it is left
out, as a nondeterministic test is, and the passing tests of its file
still judge the mutants the file owns; it keeps its reach and its work. A
test of a `describe` whose `beforeAll` or `afterAll` throws is a failing
law too. An owner file that fails outside its tests (it does not load, or
a file-level hook throws) is broken, and judges nothing. A mutant that no
deterministic passing owner test reached, held by a failing law that
reached it or by an owner file that judges nothing, and that nothing
kills, is a `blocked` gap whose directive names what it waits for. A
`blocked` line names each failing law by its file and test names (or a
broken file by its path), `results.json` lists each failing file's
`laws`, and the report says it is provisional. With
`--only <id,...>`, the baseline runs only the owners of the selected
mutants. A `crashed` gap is not a kill: production threw where no law
judged it, and the error is on the gap's Detail line in `summary.md` and
Expand Down Expand Up @@ -107,7 +116,7 @@ agent.
| `config` | `oracle-gaps: config vitest.config.ts`, or `oracle-gaps: config none, Vitest's default discovery` |
| `baseline` | `oracle-gaps: baseline pass tests/a.test.ts: 3 tests`, `oracle-gaps: baseline fail tests/b.test.ts: 1 of 2 tests failed` |
| `baseline` (decisive, before any mutant) | `oracle-gaps: baseline all 4 oracle files pass`, or `oracle-gaps: baseline 1 of 4 oracle files failed (provisional)` |
| `blocked` | `oracle-gaps: blocked <file> fails with no mutant active at <test>, so the mutants it owns wait for it: …` |
| `blocked` | `oracle-gaps: blocked <file> > <test> fails with no mutant active, so it judges nothing: …`, one per failing law |
| `gap` | `oracle-gaps: gap <kind> L<line> <id> <change>` |
| `progress` | `oracle-gaps: progress judged 25 of 120 mutants: 7 gaps so far, 18 not gaps` |
| `busy` | `oracle-gaps: busy <mutant id or baseline> in <test file> for <n>s` |
Expand Down
35 changes: 19 additions & 16 deletions packages/oracles/docs/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,22 +59,25 @@ term for one concept. Do not rotate synonyms for style.

## Gaps

| Term | Meaning |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| owns | The production files or functions an oracle answers for, listed in the `owns` option of `oracleProperty`. The gap check judges owned code only. In conversation, an oracle owns a public behavior; its `owns` list names the functions that produce it. |
| gap check | The loop that runs the oracle files against production and against mutants of the owned code. It reports every gap with one instruction. |
| mutant | A small deliberate change to owned production code. Every mutant is deliberately wrong, so do not qualify the word. The original defect is a common mutant to check. |
| kill | A mutant that a law rejects through a registered `oracleProperty` route. A plain test in an oracle file is an unregistered kill, and it does not count. |
| survivor | A mutant that no law rejects. Every survivor is a gap. |
| gap | Owned behavior that the oracles do not judge. It closes only when a law kills it or the code is removed. |
| exit | Any reasoning that puts a gap out of reach: calling a mutant equivalent or unobservable, or accepting a gap. None is allowed. |
| framework bug | A gap that no kill or removal can close. The agent stops work on that gap and reports its key and evidence. |
| exclusion control | ORC-004's control: a nearby invalid history that the grammar rejects. It is required, and distinct from a quiet exclusion, which is a finding. |
| dead code | Code whose removal no legal history can observe. Remove it. |
| invalid mutant | A mutant that no oracle could possibly detect. The framework must not generate one, so meeting one is a framework bug to report. The author never decides that a mutant is invalid, and the label does not close a gap. |
| unowned | Production code that no oracle owns. The gap check reports it as a warning, not as judged. |
| RED | The extended oracle failing on unchanged production. It shows that the oracle can see the defect. |
| GREEN | The same oracle passing after the change. |
| Term | Meaning |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| owns | The production files or functions an oracle answers for, listed in the `owns` option of `oracleProperty`. The gap check judges owned code only. In conversation, an oracle owns a public behavior; its `owns` list names the functions that produce it. |
| gap check | The loop that runs the oracle files against production and against mutants of the owned code. It reports every gap with one instruction. |
| mutant | A small deliberate change to owned production code. Every mutant is deliberately wrong, so do not qualify the word. The original defect is a common mutant to check. |
| kill | A mutant that a law rejects through a registered `oracleProperty` route. A plain test in an oracle file is an unregistered kill, and it does not count. |
| survivor | A mutant that no law rejects. Every survivor is a gap. |
| gap | Owned behavior that the oracles do not judge. It closes only when a law kills it or the code is removed. |
| exit | Any reasoning that puts a gap out of reach: calling a mutant equivalent or unobservable, or accepting a gap. None is allowed. |
| framework bug | A gap that no kill or removal can close. The agent stops work on that gap and reports its key and evidence. |
| exclusion control | ORC-004's control: a nearby invalid history that the grammar rejects. It is required, and distinct from a quiet exclusion, which is a finding. |
| dead code | Code whose removal no legal history can observe. Remove it. |
| invalid mutant | A mutant that no oracle could possibly detect. The framework must not generate one, so meeting one is a framework bug to report. The author never decides that a mutant is invalid, and the label does not close a gap. |
| failing law | A test of an oracle file that fails with no mutant active, or a test of a `describe` whose `beforeAll` or `afterAll` fails. It judges nothing: the gap check leaves it out per test, as it leaves out a nondeterministic test, and keeps its reach. The file's passing tests still judge. |
| broken owner file | An owner file that fails outside its tests: it does not load, or a file-level hook fails. It judges nothing, and another owner file's passing tests still judge the mutants they share. |
| blocked | A mutant that no deterministic passing owner test reached, held by its owners' failing laws that reached it or by owner files that judge nothing. Its directive names them, and the report is provisional. |
| unowned | Production code that no oracle owns. The gap check reports it as a warning, not as judged. |
| RED | The extended oracle failing on unchanged production. It shows that the oracle can see the defect. |
| GREEN | The same oracle passing after the change. |

There is no "equivalent mutant" and no "unobservable effect". A survivor
shows a missing history, observation, or law, or dead code. An effect that
Expand Down
Loading
Loading