Skip to content

fix(verify): fingerprint the failure identity, not the pytest run totals and progress lines (AGT-4681) - #822

Merged
unohee merged 1 commit into
mainfrom
fix/agt-4681-failure-identity-fingerprint
Oct 3, 2026
Merged

unohee merged 1 commit into
mainfrom
fix/agt-4681-failure-identity-fingerprint

Conversation

@unohee

@unohee unohee commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • AGT-4680 made the summary duration ignore the (H:MM:SS) suffix, but the failure fingerprint still hashes the whole normalised transcript. Two runs of the same failure on base and head still hashed differently when the number of tests that ran differed (a worker added tests, or main gained tests the worktree lacks), because the N passed, M skipped totals and the -q progress lines (....ss.. [ 11%]) were part of the text.
  • normalizeFailureOutput now masks the passed/skipped/deselected/xfailed/xpassed/warnings counts as <N> and drops pytest progress lines. failed and error counts, the FAILED/ERROR lines and the traceback blocks stay, so a different failure still differs.

Evidence

  • Probe before the change: duration-only difference equal=true, 4234 passed vs 4237 passed equal=false, progress line one character longer equal=false.
  • cgf-portal main fails tests/test_b5_repository.py::test_the_empty_result_has_the_same_shape_as_a_real_one on a clean checkout; workers on unrelated issues were told tests failed for it.

Tests

  • fingerprintStability.test.ts: passed-count difference, progress-line difference and decorated summary all normalise equal; a different FAILED node id, a different failed count and an extra error all still differ. Two existing assertions that pinned the old normalised text were updated (<N> skipped, 1 error, 1 failed, <N> passed, <N> skipped).
  • Mutations: removing the progress-line replace fails the progress test; removing the totals mask fails four tests.
  • src/verify and deterministicTester suites pass. Six sandbox tests in runner.test.ts timed out at 30 s while the machine ran at load average 190 and pass when rerun with a longer timeout; they are unrelated to this change.

@unohee
unohee merged commit 81bc196 into main Oct 3, 2026
7 checks passed
@unohee
unohee deleted the fix/agt-4681-failure-identity-fingerprint branch October 3, 2026 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant