Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
ecbae84
test(oracles): law constructed independence in the framework self-oracle
KyleAMathews Nov 16, 2026
2b3b484
fix(oracles): draw a campaign's generated histories apart from its co…
KyleAMathews Nov 16, 2026
d25a52a
Merge remote-tracking branch 'origin/main' into fix/constructed-strea…
KyleAMathews Nov 16, 2026
bdd7917
test(oracles): keep the self-oracle's split-verdict and appended hist…
KyleAMathews Nov 16, 2026
7427921
Merge remote-tracking branch 'origin/main' into fix/constructed-strea…
KyleAMathews Nov 16, 2026
d01a346
test(oracles): each constructed history runs its N bases from its own…
KyleAMathews Nov 16, 2026
6564421
feat(oracles): each constructed history runs N bases from its own stream
KyleAMathews Nov 16, 2026
b91168a
test(oracle-gaps): a constructed history enters authored agreement-on…
KyleAMathews Nov 16, 2026
378b5e7
test(oracles): one base per constructed history in the two end-to-end…
KyleAMathews Nov 16, 2026
774d915
test(oracle-gaps): three bases per constructed history in the static …
KyleAMathews Nov 16, 2026
ba8ec15
docs(oracles): cite the register's N-bases entry as the self-oracle's…
KyleAMathews Nov 16, 2026
26e8c60
Merge remote-tracking branch 'origin/main' into fix/constructed-strea…
KyleAMathews Nov 16, 2026
04cf1b6
test(oracles): a costly property's constructed unit runs base 0 plus …
KyleAMathews Nov 17, 2026
3afff1e
feat(oracles): drawnBases runs further bases from the run's random seed
KyleAMathews Nov 17, 2026
c4d0a1b
Merge remote-tracking branch 'origin/main' into fix/constructed-strea…
KyleAMathews Nov 17, 2026
5df0fc0
test(oracles): a drawn base's failure names the run's seed and replay…
KyleAMathews Nov 17, 2026
c9eac4c
test(oracles): a constructed history may declare its own drawnBases
KyleAMathews Nov 17, 2026
63dbc34
test(oracles): a constructed history enters constructed-own-drawn-bases
KyleAMathews Nov 17, 2026
c6b5a3a
feat(oracles): a constructed history's own drawnBases; reshard the co…
KyleAMathews Nov 17, 2026
00bc920
Merge remote-tracking branch 'origin/main' into fix/constructed-strea…
KyleAMathews Nov 17, 2026
7315bad
test(eslint-plugin-oracles): two bases per constructed history in the…
KyleAMathews Nov 17, 2026
59b5c44
test(oracle-gaps): two bases per constructed history in the static or…
KyleAMathews Nov 17, 2026
95b2ee6
test(oracles): drawn bases vary from run to run
KyleAMathews Nov 18, 2026
3a1094f
test(oracles): a drawn base's report directs base replay and filing
KyleAMathews Nov 18, 2026
2e3752b
feat(oracles): a drawn base's report directs base replay and filing
KyleAMathews Nov 18, 2026
8e29e10
ci: split the two thinnest shards
KyleAMathews Nov 18, 2026
f763593
ci: pack jobs to about 9 minutes
KyleAMathews Nov 18, 2026
b470a88
Merge remote-tracking branch 'origin/main' into fix/constructed-strea…
KyleAMathews Nov 18, 2026
3ea7bcc
ci: seven gap e2e jobs after #131's daemon witnesses
KyleAMathews Nov 18, 2026
d7e0b78
ci: eight gap e2e jobs, each with a minute of margin
KyleAMathews Nov 18, 2026
3450b60
Merge remote-tracking branch 'origin/main' into fix/constructed-strea…
KyleAMathews Nov 18, 2026
460b7b3
test(oracle-gaps): one fixed base and no drawn base in the e2e oracle
KyleAMathews Nov 18, 2026
deb5997
Merge remote-tracking branch 'origin/main' into fix/constructed-strea…
KyleAMathews Nov 18, 2026
cf2bca8
ci: seven self-oracle jobs, each with a minute of margin
KyleAMathews Nov 18, 2026
e463ea7
ci: eight self-oracle jobs; no two determinism shards share one
KyleAMathews Nov 18, 2026
fefd9e3
ci: the classify group runs apart from the first static shard again
KyleAMathews Nov 18, 2026
616dbcc
ci: the first half of the random-seed campaign units runs alone
KyleAMathews Nov 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .changeset/constructed-stream-independent.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
'@tanstack/oracles': minor
---

A campaign's generated histories no longer depend on its constructed histories, and each constructed history runs several bases from a stream of its own. The `generated` unit draws only from the property's arbitrary. Constructed history k builds from each of the first `bases` draws of the arbitrary's sample at the campaign's seed plus k, and every base must enter the states it names. The new `histories.bases` option sets that number. It must be a whole number of at least 1 (default 5), or the property fails to load. So appending a constructed history keeps every fixed seed's generated histories and every other constructed history's bases. Before, half of the generated unit's runs were constructed histories, weighted by how many there were, so appending one moved every run's draw, and each constructed unit ran one history.

A costly property can keep few fixed bases and still vary its constructed histories from run to run with the new `histories.drawnBases` option (a whole number of at least 0, default 0). After its fixed bases, each constructed unit runs that many more, drawn from the run's random seed: the seed the random units run, which the `ORACLE_SEED` line logs. That line now comes from the first unit that uses the seed, which may be a fixed-seed constructed unit. A failure on a drawn base says which base it was, prints its full history as the counterexample, and prints a replay line with the run's seed, at which the random unit of the same name runs that base at the same index.

This shifts each fixed seed once, for properties that have at least one constructed history. Their `generated` unit now runs other histories, and a seed-and-path replay line printed before this change for that unit no longer replays the same history. A constructed unit keeps its first base, and its first replay path index still selects it. Replay lines for properties without constructed histories stay valid: their `generated` unit already drew from the arbitrary alone. Each constructed unit now runs `bases` histories, so a campaign takes longer; set `bases` lower for a property whose constructed histories are slow.
15 changes: 9 additions & 6 deletions .github/workflows/pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,12 +60,12 @@ jobs:
- run: pnpm run build
- run: pnpm --filter @tanstack/eslint-plugin-oracles run test
test-oracle-framework:
name: Framework self-oracle (${{ matrix.shard }}/8)
name: Framework self-oracle (${{ matrix.shard }}/9)
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4, 5, 6, 7, 8]
shard: [1, 2, 3, 4, 5, 6, 7, 8, 9]
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
Expand All @@ -74,8 +74,8 @@ jobs:
- name: Setup Tools
uses: tanstack/config/.github/setup@e4b48f16568324f76f467aa4c2aac2f05db632c3
# The oracleProperty self-oracle spawns real Vitest runs, so the
# default test run leaves it out; it runs here on its own, in eight
# shards.
# default test run leaves it out; it runs here on its own, in nine
# shards, packed to about 9 minutes each.
- run: pnpm exec vitest --run --config vitest.oracle-framework.config.mjs -t "$(node scripts/pr-shards.mjs pattern oracle-framework "$SHARD")"
working-directory: packages/oracles
env:
Expand All @@ -97,6 +97,8 @@ jobs:
# The gap report's own oracles also run real Vitest runs on fixture
# packages (test:oracle-gaps:fast): the static oracle in two shards,
# and the classify, probe-form, and mutant-validity oracles in a third.
# Packed with the first static shard, that job ran 8m10s and 9m08s on
# #122's CI, within a minute of the limit, so it runs apart.
- run: |
if [ "$SHARD" = 3 ]; then
exec pnpm exec vitest --run --config oracle-gaps/vitest.config.mjs gap-report-classify gap-report-probe-form gap-report-mutant-validity gap-report-path-spelling
Expand Down Expand Up @@ -200,6 +202,7 @@ jobs:
- oracle-framework 6
- oracle-framework 7
- oracle-framework 8
- oracle-framework 9
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
Expand Down Expand Up @@ -281,13 +284,13 @@ jobs:
- if: always()
run: pnpm --filter @tanstack/eslint-plugin-oracles run test
windows-gap-oracles:
name: Windows gap report static oracle (${{ matrix.shard }}/6)
name: Windows gap report static oracle (${{ matrix.shard }}/2)
runs-on: windows-latest
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4, 5, 6]
shard: [1, 2]
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -596,6 +596,9 @@ oracleProperty({
histories: {
arbitrary: historyArbitrary,
runs: 40,
// Two bases per constructed history: at the framework's default of five
// this file took the Windows unit job past 10 minutes (#122).
bases: 2,
fixedSeeds: [2261111],
states: STATES,
constructed,
Expand Down Expand Up @@ -1283,6 +1286,9 @@ oracleProperty({
histories: {
arbitrary: agreeHistories,
runs: 40,
// Two bases per constructed history: at the framework's default of five
// this file took the Windows unit job past 10 minutes (#122).
bases: 2,
fixedSeeds: [2261112],
states: AGREE_STATES,
constructed: agreeConstructed,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2157,6 +2157,9 @@ oracleProperty({
histories: {
arbitrary: historyArbitrary,
runs: 40,
// Two bases per constructed history: at the framework's default of five
// this file took the Windows unit job past 10 minutes (#122).
bases: 2,
fixedSeeds: [2261113],
states: STATES,
constructed,
Expand Down
12 changes: 8 additions & 4 deletions packages/oracles/docs/oracle-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -275,10 +275,14 @@ No requirement here can be met by declaring something out of reach.
on, and a history MUST be constructed to enter each one.
- **Framework:** Enforced. Every state in `states` must be named by a
constructed history. Every campaign runs each constructed history, and one
that does not enter its named state fails. Half of the generated histories
are also constructed, so the histories after each state are explored. A
state that random histories reach only by chance goes in `measured`, which
is reported but not required.
that does not enter its named state, on any of its bases, fails. Each
constructed history runs `bases` bases (5 by default) from a stream of its
own, and the generated histories are drawn apart from the constructed ones.
So at a fixed seed, appending a constructed history keeps the generated
histories and every other constructed history's bases, and the histories
after each required state are explored, not visited once. A state that random histories
reach only by chance goes in `measured`, which is reported but not
required.
- **Author:** Reach is a predicate over the plan or trace, or a production
probe. A literal example is not reach. A `fixed` history is a regression,
not reach. A law that needs a state entered at least N times in each
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3230,6 +3230,7 @@ const required = [
`a survivor whose work fell`,
`extra-work named past an unregistered rise`,
`a rise exactly at the declared bound`,
`authored agreement-only-kill`,
]

// One constructed history per outcome class: one mutant judged by one
Expand Down Expand Up @@ -4000,6 +4001,18 @@ const constructed = [
baseline: `load`,
},
),
// An authored mutant that only the reach agreement rejects: a G11
// failure-gap class, so a constructed history enters it (review of #122;
// the fixed seed's generated histories stopped reaching it once the
// constructed histories left the generated unit).
single(
`an authored mutant killed by reach agreement alone`,
[`authored agreement-only-kill`],
{
effect: { kind: `verdict`, law: `reach agreement` },
authored: [{}],
},
),
]

oracleProperty({
Expand All @@ -4016,7 +4029,6 @@ oracleProperty({
measured: [
...[
`timeout`,
`agreement-only-kill`,
`unregistered-kill`,
`crashed`,
`framework-error`,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6665,6 +6665,13 @@ oracleProperty({
`a check that ran more than one batch`,
],
runs: 2,
// One fixed base per constructed history and none drawn: the per-run
// randomness comes from the random-seed jobs that every pull request runs
// (#90), which build each constructed history from a fresh sample (register
// M, 2026-11-18, "Gap e2e: 1 fixed base plus random-seed jobs"). Each
// history runs the CLI end to end, often for minutes: five fixed bases
// took a fixed-seed shard to 18 to 30 minutes on an Arclet (#122).
bases: 1,
fixedSeeds: [20261105],
// A filed counterexample, replayed by its history (register M, "Replay by
// history"): the fourth base of `a stale authored mutant and a
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1482,6 +1482,11 @@ oracleProperty({
arbitrary: argumentHistories,
states: required,
runs: 40,
// Two bases per constructed history: each runs the CLI on a fixture
// package, and each base adds about 4.5 minutes to the larger fixed-seed
// shard on CI (3m00s at one base on main, 11m58s at three), so two keep it
// under the 10-minute limit (#122).
bases: 2,
fixedSeeds: [20261105],
constructed: withDeclaration([
{
Expand Down
Loading
Loading