Skip to content

Automate base replay of random counterexamples (R42) #157

Open

Description

@KyleAMathews

When a random seed or drawn base finds a counterexample on a PR, the rule (v21.2 R42; M "Main's determinism failures" and "Replay by history") is: replay the full history on the base commit. If the base fails too, file it and don't block. Today that step is manual: an agent replays and files, and a rerun turns the job green. CI also cuts the failing history at 2,000 characters (#81), so the history has to be reprinted at the failing commit.

Proposal: when a random or drawn-base unit fails, a CI step writes the full history to an artifact, replays it on the merge base, and marks the job as "main also fails" or "this PR's own". Found in the #90 and #122 reviews.

This issue was written by Isaac.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions