Add sp run report — the verdict, then who caused it - #7
Merged
Conversation
A run's verdict answers one question: did the output match the approved file. Reviewers need a second one, and it is the one that decides whether a branch is blocked -- did *this change* cause it. Answering that today means finding a baseline by hand, running `run compare` against it, and knowing which baseline was the right choice. `sp run report <run>` does both. It prints the verdict as-is, resolves the runs worth comparing against -- the newest on the target branch, and the newest that predates this run -- and reports each failure against them. A failure present in both is not this change's doing; one the reference passed is where to start. The second reference is a proxy for the branch point, not ancestry: the API exposes no commit graph, so ordering by time is the best available and the label says "before this run" rather than "ancestor". `--against` names a baseline explicitly when the caller knows better. Table output caps each list at REPORT_LIST_LIMIT rows and says how many were held back; JSON is never truncated, so a script loses nothing.
A count is not something a reviewer can act on. "69 do not match" leaves them
where they started: opening the web UI to find out which 69, and then working
out which of those the reference already had.
The report now names them, one row per failure, with a column per reference:
test sample code vs 9494 vs 9484
11 Hoarders _AETVHD_2012... OUTPUT_DIFF fails there too NEW HERE
16 The Time That Remains... MISSING_OUT fails there too fails there too
Which failures are this change's is then a column to read rather than a set
difference to work out. The per-reference lists this replaces said the same
thing three times over and still made the reader join them by eye.
Codes come from classify_sample: the raw sample rows carry the ingredients --
exit codes, output states -- but not the verdict.
cfsmp3
force-pushed
the
feat/run-report
branch
from
August 17, 2026 05:50
5789c53 to
f7a1f4e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A run's verdict answers one question: did the output match the approved file. A reviewer needs a second one, and it is the one that decides whether a branch is blocked — did this change cause it?
Answering that today takes three steps: find a baseline by hand,
sp run compareagainst it, and know which baseline was the right choice. That last part is where it goes wrong. Comparing ccextractor PR #2325 against the two obvious candidates gives opposite answers:Same PR, same failures. The choice of reference is the answer, so a command that makes it explicit beats a habit that leaves it implicit.
What
Prints the verdict, resolves the references, and reports the failures against each:
JSON stays the default and is the full payload:
{run, verdict, references[]}, each reference carryingcounts,new,fixed, and any coveragewarnings.Honest about what it does not know
The second reference is a proxy for "where the branch was cut from", not ancestry — the API exposes no commit graph, so runs are ordered by time. That holds for a branch cut from the target and stops holding for one rebased weeks later. The label therefore says "the newest run before this one" rather than "ancestor", and
--againstlets a caller name the baseline when they know better.pick_references' docstring spells out the limit rather than leaving it for someone to discover.A run with no reference reports
references: []and says so in words — having nothing to compare against must never read as good news, which is the same rulenot_rerunfollows inrun compare.Testing
5 new tests, 223 total. The two that matter:
isort, pycodestyle, pydocstyle, mypy clean. Exercised against production runs 9492/9494/9484 in both output modes.
Note
pycodestyle'signore = E701insetup.cfgreplaces its default ignore list rather than adding to it, so W503 and W504 are both active — every line break near a binary operator fails, whichever side it lands on. The multi-line conditions here are written as early-continueloops for that reason. Worth knowing before anyone writes a wrapped boolean and wonders why CI disagrees with their editor.