Add nightly E2E failure-mode report - #8312
Open
isaacroldan wants to merge 1 commit into
Open
Conversation
Classifies failed E2E shard jobs from the PR workflow into named failure modes and appends them to a durable dataset on the e2e-failure-data branch. Job logs expire after 90 days and run history after ~8 months; the classified data is what survives. Because the E2E job is continue-on-error, runs with failed shards still conclude green, so the scanner walks every run's job list instead of filtering for failed runs. Patterns live in bin/e2e-failure-modes.json, ordered so job-fatal signatures win over cleanup noise as a job's primary mode. Unmatched failures are recorded as unclassified with their first error line, so new modes surface with enough context to be named. The nightly run scans a 2-day window (idempotent, deduped by job ID); workflow_dispatch takes a days_back input for backfills. The summary lands in the job summary, and in Slack when the optional E2E_REPORT_SLACK_WEBHOOK secret is configured. Validated with a real 8-day backfill: 308 runs scanned, 87 failed shards classified, zero unclassified. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
github-actions
Bot
added
the
no-changelog
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WHY are these changes introduced?
A 31% E2E shard failure rate survived five months because nobody was counting. Job logs expire after 90 days and run history after ~8 months, so by the time anyone looked, most of the evidence was gone. This adds the counting.
WHAT is this pull request doing?
A nightly workflow (
e2e-failure-report.yml) that:tests-pr.ymlrun in the window (every run's job list — the E2E job iscontinue-on-error, so failed shards hide inside green runs and filtering for failed runs misses them).E2E tests (shard …)job and classifies it against the ordered patterns inbin/e2e-failure-modes.json. Job-fatal signatures win over cleanup noise as the primary mode; unmatched failures are recorded asunclassifiedwith their first error line so new modes can be named.e2e-failure-databranch (idempotent, deduped by job ID) — the durable record that outlives log retention.E2E_REPORT_SLACK_WEBHOOKsecret is set.workflow_dispatchtakes adays_backinput for backfills.Validation
Ran a real 8-day backfill locally: 308 runs scanned, 87 failed shards classified, zero unclassified. The seeded dataset is already on the
e2e-failure-databranch. First real finding — this week's modes:(25.8% of runs where E2E ran had a failed shard. Throttling — not cleanup — is currently the top mode.)
How to test your changes?
Measuring impact
🤖 Generated with Claude Code