test(rewrite): systematic coverage matrix for assertion rewriting - #14813
Draft
RonnyPfannschmidt wants to merge 1 commit into
Draft
test(rewrite): systematic coverage matrix for assertion rewriting#14813RonnyPfannschmidt wants to merge 1 commit into
RonnyPfannschmidt wants to merge 1 commit into
Conversation
RonnyPfannschmidt
force-pushed
the
ronny/assert-rewrite-coverage-harness
branch
from
July 31, 2026 19:01
dc66953 to
2c2d51e
Compare
This was referenced Jul 31, 2026
The rewriter is tested through its failure messages, one regression test per issue. That makes it hard to say whether a change improved anything: there is no place that records what the rewriter cannot do yet. Add a matrix over expression types with four axes -- introspection depth, semantic equivalence, single evaluation, and evaluation order -- and record every known gap as a strict xfail tagged with a group name. A change that closes one names the group and flips its markers; strictness means it cannot forget. The evaluation-order axis is new. Comparing pass/fail is not enough: an operand read after a later walrus operator rebound its name runs exactly once and can still fail the assertion, having compared the wrong value. assert_evaluation_order() compares what check() returns, because the fragile operand is a bare name -- wrapping it in a call to observe it would hoist it and hide the bug. Behaviour is unchanged; this only describes it.
RonnyPfannschmidt
force-pushed
the
ronny/assert-rewrite-coverage-harness
branch
from
July 31, 2026 20:12
2c2d51e to
ff0f7c0
Compare
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The assertion rewriter is tested through its failure messages, one regression test per issue. That makes it hard to say whether a change improved anything: there is no place that records what the rewriter cannot do yet.
This adds a matrix over expression types with four axes — introspection depth, semantic equivalence, single evaluation, and evaluation order — and records every known gap as a
strict=Truexfail tagged with a group name. A change that closes one names the group and flips its markers; strictness means it cannot forget, and it cannot silently regress a neighbouring group either.On
mainthat is 24 xfails across ten groups, listed in the module docstring.The evaluation-order axis is new
Comparing pass/fail is not enough. An operand read after a later walrus operator rebound its name runs exactly once and can still fail the assertion, having compared the wrong value — so neither the semantic-equivalence axis nor the single-evaluation axis sees it.
assert_evaluation_order()compares whatcheck()returns from a plain and a rewritten run. It cannot wrap the fragile operand to observe it: a bare name is exactly the case the rewriter leaves unhoisted, and putting a call around it would hoist it and hide the bug.Most operands are safe today because
generic_visithoists them into a temporary. The passing tests in that class are guards, and they are the point: they keep that true as new visitors take expression types away fromgeneric_visit.No behaviour change
This PR only describes the rewriter. It is the base of a stack that closes the groups one at a time:
visit_Subscript→ 11visit_IfExp→ 8Draft until the ones above it are reviewed; each is a separate PR based on
main, so their diffs include this one.