Repository navigation
feat(oracle-gaps): block per law: a failing law judges nothing, and its file's passing laws still judge - #176
Open
KyleAMathews wants to merge 14 commits into
Open
KyleAMathews wants to merge 14 commits into
KyleAMathews wants to merge 14 commits into
Conversation
…ng laws still judge Extends the gap report's end-to-end and classify oracles with the maintainer's decision "Block per law" (register M, 2026-11-18), which replaces v21.2 R11's file-level block: a test that fails with no mutant active judges nothing and is left out per test, as a nondeterministic test is, so the passing tests of its file still judge the mutants the file owns. A mutant is blocked only when no owner test passes, and the baseline's blocked lines and the blocked directive name each failing law. - e2e model: a drawn field `failingLaw` (a, b, both, or none), appended last; the model judges by passing owner tests and blocks only a mutant with none; law 14 owes one blocked line per failing law. - New required states "a failing law left its file's other laws judging" and "blocked by a failing law", entered by the constructed history "a failing law beside its file's passing laws" (appended last), and the first by the brokenOwner declaration fault. "blocked by brokenOwner" is retired: its failing owner always leaves passing tests in its file. - classify model: failing laws judge nothing; blockersOf over owners' passing tests; the same constructed history and states. RED on the parent: main blocks the whole file. Co-authored-by: Isaac <no-reply@databricks.com>
…wner test judges Block per law (register M, 2026-11-18): a test that fails with no mutant active judges nothing, so it is left out per test, as a nondeterministic test is, and its file's passing tests still judge the mutants the file owns. Replaces v21.2 R11's file-level block. - passOf drops a failing law's records and gives each failing file's laws (its failing tests' keys, or the file when it failed outside its tests) and the files with a passing test. - baselineOf's leftOut joins the nondeterministic tests and the failing laws: their verdicts neither kill nor stop a run. - blockersOf blocks a mutant only when no owner test passes; the blocked directive names its owners' failing laws. - A file that failed outside its tests is still left out whole. - The baseline prints one blocked line per failing law, and the report stays provisional. Co-authored-by: Isaac <no-reply@databricks.com>
…ivor beside it A third mutant of the first file survives its passing law, so a runner that blocked the whole file, or counted the failing law's verdict as a kill, disagrees with the model in the constructed history itself, not only in generated ones. Co-authored-by: Isaac <no-reply@databricks.com>
…st judges The README, the setup and explore skills, and a changeset describe the per-law block (register M, 2026-11-18, "Block per law"). Co-authored-by: Isaac <no-reply@databricks.com>
…files to the check runAcross returned only the pass's units, rows and failed files, so the check read `failing` and `passing` from undefined and every run ended in an internal error. Co-authored-by: Isaac <no-reply@databricks.com>
…e2e model The model had begun to count an owner file's tests of no plan (the console test, the failing owner's passing pair) as reaching code that ran at load. The generated fixed seed showed that production does not (module data owned beside a console-only file is indeterminate on main and here), so the model keeps main's rule, over the plan's tests. Those tests still keep their file's mutants from blocking, as passing owner tests. Co-authored-by: Isaac <no-reply@databricks.com>
🦋 Changeset detectedLatest commit: 7e105ae The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
Co-authored-by: Isaac <no-reply@databricks.com>
Its build prepended a function, an owner and a test, so a drawn base of three functions or five tests gave a history the arbitrary cannot take back (random seed -1273878131 in CI). The judged function and its test of file b now take the base's first function's and last test's places. Co-authored-by: Isaac <no-reply@databricks.com>
main's two constructed histories from #164 come first in the list, and the failing-law history follows them. Co-authored-by: Isaac <no-reply@databricks.com>
…broken owner file judges nothing Review of #176 (F1 to F6) and the maintainer's ruling "A broken owner file judges nothing" (register M, 2026-11-19), oracle-first. - e2e grammar: three drawn fields, appended last: the failing law's position (start or end of its file), whether it calls the first generated function (none, once, or 110,000 times, past the call floor), and whether a describe's beforeAll or afterAll throws around it. - e2e model: an operator mutant that no deterministic passing owner test reached is blocked by its owners' failing laws that reached it, and by owner files that judge nothing (no load, or every test failing); one that nothing reached stays unreached. A describe's failing hook makes its laws fail, and the file fails outside its tests. - Three constructed histories, appended last, with three required states: a heavy failing law at the start of its file beside a passing law (F1, F2, F3); a failing beforeAll around a law; a failing afterAll around a law that calls production (F2, F4). - classify model: a failing law keeps its reach (targets, owners line) and judges nothing; the same blocked rule; its constructed history gains a mutant that only the failing law reaches. - Both oracles' limits name what the evidence does not cover (F6). Co-authored-by: Isaac <no-reply@databricks.com>
…n owner file judges nothing Review of #176 (F1 to F5, F7) and register M, 2026-11-19, "A broken owner file judges nothing". - passOf keeps a failing law's records, so its reach, its calls (the call budget) and its I/O count as a test's do; only its judging is left out (F1). A failing law is never listed as nondeterministic, and its work, infections and operands judge nothing. - blockersOf blocks an operator mutant that no deterministic passing owner test reached, by its owners' failing laws that reached it and by owner files that judge nothing at all (broken, or every test failing): a mutant that only a failing law reached is blocked, not unreached (F2, F5). - A file's selection leaves its failing laws out, checked by reproduces; the file runs whole, its laws ignored, otherwise (F3). - A describe whose beforeAll or afterAll throws fails its own tests as failing laws (F4). - The baseline's laws name each failing law; results.json gives them per failing file, and the failedOutsideTests flag is gone (F7). blockedLines is one template. Co-authored-by: Isaac <no-reply@databricks.com>
…e glossary and README Review of #176, F7 (finding 9, glossary-first naming) and the changeset. Co-authored-by: Isaac <no-reply@databricks.com>
…ile with a plan test The e2e model let code that ran at load reach every failing law of an owner file that runs any test. Main's rule for code run at load, which the model keeps for passing tests, counts a file's tests of the plan, and production records no load reach for a file without one: CI's random seed -555480425 of "a failing owner under its parent's config" judged module data owned by file b, whose only tests are the failing owner's suite, unreached, which the model owed blocked. The blocked rule now follows the same rule. Co-authored-by: Isaac <no-reply@databricks.com>
…lan test A random base of the brokenOwner fault (seed 101) judges module data by a test of no plan, which the model does not count (issue #186). It is stated in the e2e oracle's limits with its trace, for the maintainer's ruling. Co-authored-by: Isaac <no-reply@databricks.com>
commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Kody (Kent's agent) reported that one failing law hides the gap report for its whole file. This PR builds two maintainer decisions into the
oracle-gapsCLI, oracle-first:The second round fixes the review of 597c8bd (findings F1 to F7).
The rules now are:
Failing law. A test of an oracle file that fails with no mutant active judges nothing. The CLI leaves it out per test, as it already leaves out a nondeterministic test, and the file's passing tests still judge the mutants the file owns.
A failing law keeps its records. Its reach, its calls toward the call budget, and its I/O still count; only its judging is left out (F1).
Mutant runs. A file's mutant runs leave its failing laws out through a selection that
reproduceschecks against the unmutated pass (F3).Describe hooks. A test of a
describewhosebeforeAllorafterAllthrows is a failing law too (F4).Broken owner file. An owner file that fails outside its tests (it does not load, or a file-level hook throws) judges nothing. Another owner file's passing tests still judge the mutants they share (F5).
blocked. An operator mutant that no deterministic passing owner test reached isblockedby:A mutant that only a failing law reached is
blocked, notunreached(F2). A mutant that nothing reached staysunreached. An authored mutant isblockedonly when no owner file has a passing test.Report. The baseline prints one
blockedline per failing law (or one per broken file).results.jsonlists each failing file'slaws, and the report stays provisional. ThefailedOutsideTestsfield is gone (F7).Changes
Production (
packages/oracles/oracle-gaps/):classify.mjs:passOfkeeps every record. It returnsfailing(each failing file's failing test keys, including the tests of a describe whose hook failed, or the file itself when it is broken) andpassing(the files with a passing test that is not a failing law).baselineOf(first, second, failing)never lists a failing law as nondeterministic. ItsleftOutset holds both kinds, and neither one's work, infections or operands judge a mutant.blockersOf(owners, pass, targets, baseline)decidesblockedby reach for operator mutants, and by file for authored mutants.check.mjs:unit()records the errors of each describe hook and the tests under it (hooks).selectionOfleaves failing laws out.laws.runAcrosspassesfailingandpassingon.stream.mjs:blockedLinesis one template, driven bylaws.report.mjs: the provisional text, and the blocked directive's names.Oracles:
gap-report-model-oracle.mjs): four drawn fields, appended last under "Randomness wins" (M-25, R54):failingLaw: in file a, b, both, or neither;failingLawAt: at the start or the end of its file;failingLawCalls: calls the first generated function not at all, once, or 110,000 times, past the call floor;failingLawHook: a describe'sbeforeAllorafterAllthrows around it.blockedrule follows reach, as above. A describe's failing hook makes its laws fail, and the file fails outside its tests. Constructed histories and the filed fixed history set the four fields to neutral values.brokenOwnerdeclaration-fault history: its build gives file b a function and a passing law that kills it, within the grammar's bounds. "blocked by brokenOwner" is retired, because its failing owner always leaves passing tests in its file.limitsstrings list what the evidence does not cover (F6).constructed-order.json,history-fingerprint.json,e2e-shards.mjs, and Windows shard 7 inpr-shards.mjs.Docs: glossary entries "failing law", "broken owner file" and "blocked" (finding 9); the README; the
oracles-setupandoracles-exploreskills; and a changeset (minor).Evidence
Law:
blockedline per law.Reach (GREEN
perlaw-green-5, fixed seed). Each cell is how many runs of that history reached the state:generatedunit reaches none of the new states. Earlier (fd32d67) its reach equalled main's 30928ee, with no state lost or gained. Random seeds draw a failing law in 3 of 7 histories.RED:
perlaw-red-2:f0:31:if-true: timeout, owed killed(F1).the stream's decisive baseline lines are ["…all 2 oracle files pass"], owed ["…1 of 2 oracle files failed (provisional)"](F4).m2: unreached, owed blocked(F2) and the owners line.GREEN (84b4fa5 production; 7f8e810 model correction for load reach, verified by
perlaw-replay3-1, 7 of 7):perlaw-green-5: 6 of 6 targeted e2e units pass (the four failing-law histories, brokenOwner,generated).perlaw-fast-3: the fast suite passes 33899 of 33899. Both shard checks and the fingerprint canary pass.Mutants (scratch branches; e2e on Arca
perlaw2-mut-*; classify locally):blockersOfreturns every failing owner's laws): 5 of 5 e2e units fail (f0 if-false: blocked, owed branch-never-taken; brokenOwnerw_flaky: blocked, owed indeterminate). Classify fails 12, among them the new history:m0: blocked, owed unreached.leftOutwithout the laws): 4 of 5 e2e units fail (killed, owed branch-never-taken;delete-call: not-distinguished, owed blocked; brokenOwnercrashed). Classify fails 8.timeout, owed killed(F1). The afterAll history fails withdelete-call: unreached, owed blocked(F2). Classify fails 8.Limits (also in the oracles'
limits):beforeEachorafterEachthat throws fails every test, and the report names one failing law per test, not the hook (finding 8). It is not generated.CI (7e105ae)
57 checks pass and 1 is skipped. The only failures are Windows fixed-seed 2/10 and 5/10, which fail the same way on main and do not block (#79, D56). The longest job took 8m56s.
D_ARR arithmetic-swap: unreached, owed blockedin "a failing owner under its parent's config". The e2e model had let code that ran at load reach a failing law in a file with no plan test. 7f8e810 applies main's load rule there too. CI's seed -555480425 and seeds 101, 202 and 303 then pass every failing-law unit, with one exception: seed 101 of brokenOwner.D_OBJ field-drop: not-distinguished, owed indeterminate): the model does not cover load-time reach of an owner file that has no plan test. Production judges that code by a test that is not in the plan. Before this PR, blocking the whole file hid the case. It is filed as Gap e2e: load-time reach of an owner file with no plan test is not modeled (exposed by per-law blocking) #186 for the maintainer's ruling and stated with its trace in the e2elimits. This random unit can fail on such a seed until that ruling.perlaw-177-main-1,-2); fix(oracle-gaps): a stopped run removes its session directory (#150) #164 probably fixed it, and the issue is annotated.Production weight
Minified (esbuild), main 6d05f05 compared with this branch, for
check,classify,reportandstream: 59,860 to 61,398 bytes (+1,538); gzip 23,980 to 24,540 (+560).classify.mjsblockersOf, the hook-scoped laws inpassOf,leftOutin work, infection and operandscheck.mjshooksinunit(), the selection without laws, baselinelawsstream.mjsblockedLinesis now one template, down from +284 in round 1report.mjsKeeping the records removed the row filter (the reviewer's suggestion). The reach rule needs a scan only for code that ran at load. I found no state or branch to remove in return: the per-law and broken-file paths are both required by the rulings.
v21.2 conformance
classify.mjspassOf,baselineOf,blockersOf;check.mjsselectionOf.blockedby them.classify.mjsblockersOf.leftOut, and its work counts toward the call budget, so it cannot spin a copy (F1, M3).unreachedand notimeoutfrom a failing law's work.blockedline per law, before any mutant. Low: a file-levelbeforeEachfailure prints one line per test (a stated limit).This pull request and its description were written by Isaac.