Skip to content

test(rewrite): systematic coverage matrix for assertion rewriting - #14813

Draft
RonnyPfannschmidt wants to merge 1 commit into
pytest-dev:mainfrom
RonnyPfannschmidt:ronny/assert-rewrite-coverage-harness
Draft

test(rewrite): systematic coverage matrix for assertion rewriting#14813
RonnyPfannschmidt wants to merge 1 commit into
pytest-dev:mainfrom
RonnyPfannschmidt:ronny/assert-rewrite-coverage-harness

Conversation

@RonnyPfannschmidt

Copy link
Copy Markdown
Member

The assertion rewriter is tested through its failure messages, one regression test per issue. That makes it hard to say whether a change improved anything: there is no place that records what the rewriter cannot do yet.

This adds a matrix over expression types with four axes — introspection depth, semantic equivalence, single evaluation, and evaluation order — and records every known gap as a strict=True xfail tagged with a group name. A change that closes one names the group and flips its markers; strictness means it cannot forget, and it cannot silently regress a neighbouring group either.

On main that is 24 xfails across ten groups, listed in the module docstring.

The evaluation-order axis is new

Comparing pass/fail is not enough. An operand read after a later walrus operator rebound its name runs exactly once and can still fail the assertion, having compared the wrong value — so neither the semantic-equivalence axis nor the single-evaluation axis sees it.

assert_evaluation_order() compares what check() returns from a plain and a rewritten run. It cannot wrap the fragile operand to observe it: a bare name is exactly the case the rewriter leaves unhoisted, and putting a call around it would hoist it and hide the bug.

Most operands are safe today because generic_visit hoists them into a temporary. The passing tests in that class are guards, and they are the point: they keep that true as new visitors take expression types away from generic_visit.

No behaviour change

This PR only describes the rewriter. It is the base of a stack that closes the groups one at a time:

Draft until the ones above it are reviewed; each is a separate PR based on main, so their diffs include this one.

The rewriter is tested through its failure messages, one regression test
per issue.  That makes it hard to say whether a change improved anything:
there is no place that records what the rewriter cannot do yet.

Add a matrix over expression types with four axes -- introspection depth,
semantic equivalence, single evaluation, and evaluation order -- and
record every known gap as a strict xfail tagged with a group name.  A
change that closes one names the group and flips its markers; strictness
means it cannot forget.

The evaluation-order axis is new.  Comparing pass/fail is not enough: an
operand read after a later walrus operator rebound its name runs exactly
once and can still fail the assertion, having compared the wrong value.
assert_evaluation_order() compares what check() returns, because the
fragile operand is a bare name -- wrapping it in a call to observe it
would hoist it and hide the bug.

Behaviour is unchanged; this only describes it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided (automation) changelog entry is part of PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant