Skip to content

Attribute failure journals to test cases - #1356

Merged
taras merged 11 commits into
mainfrom
codex/test-case-journal-attribution-1171
Nov 17, 2026
Merged

taras merged 11 commits into
mainfrom
codex/test-case-journal-attribution-1171

Conversation

@taras

@taras taras commented Nov 16, 2026

Copy link
Copy Markdown
Owner

Why

Failure journals shared a document path across test cases, so a CI download could leave the owning test unclear. Several direct and subprocess harnesses also discarded journals before the assertions that needed them. Advances #1171.

What changes

  • Consolidate failure capture in one test support adapter. Attributed JSONL files have a neighboring .case.json that names the test file, case, execution kind, ordinal, and document source.
  • Apply the adapter to safe live cases in Testing, Web, Core, Git, Agent, ACP, CLI, synthetic REPL, and Workflow tests.
  • Keep an opted-in ORC12 child’s bounded, guarded journal in the parent’s memory through the post-exit assertions. Persist it only if the parent fails.

How to verify it

  • The attributed adapter is covered by Deno, Node 22.23.4, and Bun. Its witnesses exercise repeated names, concurrent roots, detached children, joined cancellation, and no artifacts on success.
  • Each migration commit has focused runtime and typecheck evidence. The ORC12 regression preserves the original failure and saves one attributed child journal after the child exits. The passing control writes no artifact.
  • The compiled CLI diagnostics check passed. The JSONL record format and ordinary CLI output remain compatible.

Scope

This PR delivers the shared adapter and the verified migration batches. Issue #1171 remains open for pending and partially migrated owners in the inventory.

Scope confirmation

  • Changed files support attributed journal capture and migration.
  • Existing replay and guarded fixture ownership is preserved.
  • The description matches the signed candidate and focused evidence.

@taras
taras marked this pull request as ready for review November 16, 2026 19:41
taras added a commit that referenced this pull request Nov 17, 2026
taras added a commit that referenced this pull request Nov 17, 2026
taras added a commit that referenced this pull request Nov 17, 2026
taras added a commit that referenced this pull request Nov 17, 2026
taras added a commit that referenced this pull request Nov 17, 2026
taras added a commit that referenced this pull request Nov 17, 2026
taras added a commit that referenced this pull request Nov 17, 2026
@taras
taras enabled stacked auto-merge (rebase) November 17, 2026 05:38
@taras
taras disabled stacked auto-merge November 17, 2026 11:46
@taras
taras force-pushed the codex/test-case-journal-attribution-1171 branch from b9a7acf to abacea6 Compare November 17, 2026 11:59
@taras
taras added this pull request to stack #1386 November 17, 2026 12:00
@taras
taras enabled stacked auto-merge (rebase) November 17, 2026 12:01
@taras
taras merged commit f518002 into main Nov 17, 2026
64 checks passed
@taras
taras deleted the codex/test-case-journal-attribution-1171 branch November 17, 2026 12:17
taras added a commit that referenced this pull request Nov 17, 2026
taras added a commit that referenced this pull request Nov 17, 2026
taras added a commit that referenced this pull request Nov 17, 2026
taras added a commit that referenced this pull request Nov 17, 2026
taras added a commit that referenced this pull request Nov 17, 2026
* test: attribute failure journals to declaring cases

* test: capture failure journals for Testing document cases

* test: capture failure journals for live WebForm cases

* test: capture failure journals for held Core hosts

* test: retain the ORC12 child journal for parent failures

* test: attribute journals in ordinary Git execution cases

* test: attribute journals in direct Core document cases

* test: attribute failure journals for live synthetic REPL entries

* test: attribute guarded ACP and TestAgent execution journals

* test: attribute safe Agent and nested Testing execution journals

* test: attribute fresh REPL observation entry journals

* test: capture additional direct Core execution journals

* test: attribute fresh workflow observation journals

* test: attribute fresh CLI and Testing execution journals

* test: attribute fresh Git issue scenario journals

* test: retain separate CLI assembly journals for parent failures

* test: attribute guarded CLI host and upgrade journals

* test: attribute guarded Agent provider and usage journals

* test: attribute fresh Answers, Elicit and Fail fixture journals

* test: attribute fresh TestAgent component and launch journals

* test: capture fresh inline REPL selection journals

* test: capture guarded Glob fixture journals

* test: attribute guarded Agent function component journals

* test: retain separate journals for Testing session siblings

* test: capture guarded Core prop evaluation journals

* test: preserve independently bounded subprocess journals

* Adopt shared journals in Core evaluator tests

* Adopt shared journal resource in Each tests

* Use shared failure journal package in evaluator cases

* Migrate Core cases to shared journal resource

* test: adopt shared journal resource in #1356 consumers

* Use package journal entry in Core stream tests

* migrate #1357 ordinary journal cases to shared fixtures

* Format #1356 shared journal tests

* Restore direct it registration in Core held-host tests

* Migrate assert throws journal ownership

* Migrate testing boundary journal owner

* Restore direct test declarations in Agent, REPL and Web

* Migrate testing boundary contract cases

* Migrate test component journal cases

* Migrate use testing journal cases

* Keep ordinary test declarations in two Testing suites

* Migrate workflow generated observation cases

* Remove unused journal case registrar

* Remove unused testing case registrar

* Restore direct it registration in Git tests

* Restore direct Glob and TestAgent test declarations

* Expose the shared test journal resource

* Adopt shared journals in Core evaluator tests

* Use shared failure journal package in evaluator cases

* Integrate #1356 shared journal consumers

* Complete shared journal imports in #1356

* Integrate Luna formatting correction for #1356

* Restore direct test declarations in Agent, REPL and Web

* Keep ordinary test declarations in two Testing suites

* Integrate direct declarations in Testing and Workflow

* Remove unused testing case registrar

* Restore direct test declarations throughout the base journal PR

---------

Co-authored-by: Luna <luna@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant