Repository navigation
Clean up unpublished benchmark results after publication failure #3207
Description
Activity
Fixed in
5124ba4cfcand pushed to PR #3200.Cleanup records ownership before the first move, then consults the actual authoritative index. It removes only the new unindexed result, preserves a result already selected by the index even if interrupted immediately after index publication, and preserves output with an explicit failure if the index cannot be read or parsed. INT/TERM preserve their failure status. No rollback copy is created.
The old runner failed the real isolated orchestration regression for an index-move failure. All seven tests pass after the fix, including failure+retry, INT/TERM after result publication, INT/TERM after index publication, pre-existing result refusal, and unreadable/malformed index handling. The tests check historical bytes and temporary-stage removal, isolate inherited Git environment, and are wired into the existing CI benchmark harness job. Shell syntax, actionlint, and diff checks pass.
Review of PR #3200 found that
scripts/run-claude-observation-benchmark.shpublishes its validated result and evidence index with two separate moves. If the second move fails, or an interrupt arrives after the result move, the new result remains unindexed. A subsequent run refuses the existing result path and the evidence directory validator rejects the orphan.Review comment: #3200 (comment)
Fix failure/interruption cleanup for only the newly published result. Preserve the authoritative existing index and historical artifacts, create no rollback copies, and avoid deleting the result if the index was successfully published immediately before interruption. Verify failed index publication and interruption around both moves, plus successful publication/retry behavior.
Filed before the follow-up edit.
Resolution: Merged via #3141. Publication cleanup consults the authoritative index and removes only an unpublished new result. Seven behavioral tests cover failed index moves, retries, both interrupt boundaries, prior results and malformed/unreadable indexes.