Repository navigation
Compare satask answers against the ask oracle in CI - #7
Merged
Merged
Conversation
Upstream satask parity rejects intentional improvements: the sathandlers work raises the validation pass rate from 40/104 to 79/104 by matching full ask, yet introduces six divergences from upstream satask (including three ValueError answers for inconsistent assumptions), so the comparison gate goes red. Add an --oracle mode to benchmarks.compare_satask that uses pinned SymPy's ask as the correctness reference. Being stronger or weaker than ask is reported but accepted, and ValueError for inconsistent assumptions is accepted; only a definite answer contradicting ask, or an unexpected exception, fails the run. CI switches to this mode; the baseline satask comparison remains available via the existing options.
This was referenced Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The upstream-
sataskparity check in CI rejects intentional improvements. The sathandlers work raises the validation pass rate from 40/104 to 79/104 by matching fullask, but introduces six divergences from upstreamsatask(three of themValueErroranswers for inconsistent assumptions), sobenchmarks.compare_satask --include-early-returngoes from 0 to 6 mismatches and the CI job fails.What
--oraclemode tobenchmarks.compare_sataskthat uses pinned SymPy's fullaskas the correctness reference.mismatch(definite contradiction withask) andcurrent-error(unexpected exception) fail the run.stronger/weakerthanaskis reported but accepted;ValueErrorfor inconsistent assumptions is accepted.python -m benchmarks.compare_satask --oracle --include-early-return; the baseline-satask parity mode remains available via the existing options.Verification
pytest reasoning/tests-> 149 passed, 1 xfailed (this branch); the sathandlers branch with this comparison file -> 128 agree, 3 stronger, 3 inconsistent, 0 failures.--oracle --include-early-returnonmain: 134 agree in both modes.mypystrict clean; known-facts check up to date; old-assumptions scan unchanged (2 pre-existing LRA findings).