fix(verify): fingerprint the failure identity, not the pytest run totals and progress lines (AGT-4681) - #822
Merged
Conversation
…als and progress lines (AGT-4681)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
(H:MM:SS)suffix, but the failure fingerprint still hashes the whole normalised transcript. Two runs of the same failure on base and head still hashed differently when the number of tests that ran differed (a worker added tests, or main gained tests the worktree lacks), because theN passed, M skippedtotals and the-qprogress lines (....ss.. [ 11%]) were part of the text.normalizeFailureOutputnow masks the passed/skipped/deselected/xfailed/xpassed/warnings counts as<N>and drops pytest progress lines.failedanderrorcounts, the FAILED/ERROR lines and the traceback blocks stay, so a different failure still differs.Evidence
4234 passedvs4237 passedequal=false, progress line one character longer equal=false.tests/test_b5_repository.py::test_the_empty_result_has_the_same_shape_as_a_real_oneon a clean checkout; workers on unrelated issues were toldtests failedfor it.Tests
fingerprintStability.test.ts: passed-count difference, progress-line difference and decorated summary all normalise equal; a different FAILED node id, a different failed count and an extra error all still differ. Two existing assertions that pinned the old normalised text were updated (<N> skipped, 1 error,1 failed, <N> passed, <N> skipped).src/verifyanddeterministicTestersuites pass. Six sandbox tests inrunner.test.tstimed out at 30 s while the machine ran at load average 190 and pass when rerun with a longer timeout; they are unrelated to this change.