Skip to content

Add sp run report — the verdict, then who caused it - #7

Merged
cfsmp3 merged 2 commits into
masterfrom
feat/run-report
Aug 17, 2026
Merged

Add sp run report — the verdict, then who caused it#7
cfsmp3 merged 2 commits into
masterfrom
feat/run-report

Conversation

@cfsmp3

@cfsmp3 cfsmp3 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Why

A run's verdict answers one question: did the output match the approved file. A reviewer needs a second one, and it is the one that decides whether a branch is blocked — did this change cause it?

Answering that today takes three steps: find a baseline by hand, sp run compare against it, and know which baseline was the right choice. That last part is where it goes wrong. Comparing ccextractor PR #2325 against the two obvious candidates gives opposite answers:

vs run 9494 (the newest master run)     -> new 0
vs run 9484 (the newest before this one) -> new 45

Same PR, same failures. The choice of reference is the answer, so a command that makes it explicit beats a habit that leaves it implicit.

What

sp run report <run_id> [--against RUN_ID]... [--branch master] [--scan N]

Prints the verdict, resolves the references, and reports the failures against each:

run 9492  linux  aee96932  status=fail
  69 of 237 tests do not match the approved output

  vs the newest run on the target branch — run 9494 (3af3fc22)
      new 0   changed 0   still_failing 69   fixed 0

  vs the newest run before this one — run 9484 (128175ea)
      new 45   changed 0   still_failing 24   fixed 1
  ...
  45 of 69 failures are new relative to the newest run before this one. Those are this change's.

JSON stays the default and is the full payload: {run, verdict, references[]}, each reference carrying counts, new, fixed, and any coverage warnings.

Honest about what it does not know

The second reference is a proxy for "where the branch was cut from", not ancestry — the API exposes no commit graph, so runs are ordered by time. That holds for a branch cut from the target and stops holding for one rebased weeks later. The label therefore says "the newest run before this one" rather than "ancestor", and --against lets a caller name the baseline when they know better. pick_references' docstring spells out the limit rather than leaving it for someone to discover.

A run with no reference reports references: [] and says so in words — having nothing to compare against must never read as good news, which is the same rule not_rerun follows in run compare.

Testing

5 new tests, 223 total. The two that matter:

  • a failure the reference also has is not reported as new — the mistake the platform itself used to make, reporting 70 tests as broken by a branch that had not touched them
  • table output caps at 15 rows per reference and says how many were held back, while JSON returns all 39 — truncation is for reading, and a script must lose nothing

isort, pycodestyle, pydocstyle, mypy clean. Exercised against production runs 9492/9494/9484 in both output modes.

Note

pycodestyle's ignore = E701 in setup.cfg replaces its default ignore list rather than adding to it, so W503 and W504 are both active — every line break near a binary operator fails, whichever side it lands on. The multi-line conditions here are written as early-continue loops for that reason. Worth knowing before anyone writes a wrapped boolean and wonders why CI disagrees with their editor.

cfsmp3 added 2 commits August 16, 2026 22:50
A run's verdict answers one question: did the output match the approved file.
Reviewers need a second one, and it is the one that decides whether a branch is
blocked -- did *this change* cause it. Answering that today means finding a
baseline by hand, running `run compare` against it, and knowing which baseline
was the right choice.

`sp run report <run>` does both. It prints the verdict as-is, resolves the runs
worth comparing against -- the newest on the target branch, and the newest that
predates this run -- and reports each failure against them. A failure present in
both is not this change's doing; one the reference passed is where to start.

The second reference is a proxy for the branch point, not ancestry: the API
exposes no commit graph, so ordering by time is the best available and the label
says "before this run" rather than "ancestor". `--against` names a baseline
explicitly when the caller knows better.

Table output caps each list at REPORT_LIST_LIMIT rows and says how many were
held back; JSON is never truncated, so a script loses nothing.
A count is not something a reviewer can act on. "69 do not match" leaves them
where they started: opening the web UI to find out which 69, and then working
out which of those the reference already had.

The report now names them, one row per failure, with a column per reference:

    test  sample                    code         vs 9494          vs 9484
    11    Hoarders _AETVHD_2012...  OUTPUT_DIFF  fails there too  NEW HERE
    16    The Time That Remains...  MISSING_OUT  fails there too  fails there too

Which failures are this change's is then a column to read rather than a set
difference to work out. The per-reference lists this replaces said the same
thing three times over and still made the reader join them by eye.

Codes come from classify_sample: the raw sample rows carry the ingredients --
exit codes, output states -- but not the verdict.
@cfsmp3
cfsmp3 merged commit 6c526eb into master Aug 17, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant