Skip to content

Add a failures reporter to xmd test - #1364

Merged
taras merged 4 commits into
mainfrom
codex/xmd-test-reporter
Nov 17, 2026
Merged

taras merged 4 commits into
mainfrom
codex/xmd-test-reporter

Conversation

@taras

@taras taras commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Why

Markdown test runs print the full rendered report and progress for each passing test. In a Sidekick Round or CI log, that can bury the failure and its journal path.

What changes

xmd test ./tests --reporter=failures

The failures reporter writes failure and cancellation diagnostics, journal paths, and a final summary with pass and fail counts and elapsed time to stderr. It counts tests from the selected root documents. The default pretty reporter provides the full rendered report and live progress.

The README includes a command for a prepared checkout. The testing spec explains both reporters and shows the summary format.

CI

  • Routine Markdown test invocations use --reporter=failures. The shard announces each Markdown suite as it starts.
  • Native Deno test invocations use --reporter=dot. Deno keeps the failure diagnostics and omits console logs.
  • Test weight measurement and CI shards use the same reporter arguments. The compiled diagnostics smoke test still checks default reporter progress in the child XMD invocation.

How to verify it

  • Deno ran the CLI diagnostics and document target suites. The new cases cover a passing file, a directory with mixed results, journal retention, and rejection of an unsupported reporter.
  • Node passed the focused reporter cases. Deno typecheck and formatting checks passed.
  • The README parses and a source CLI smoke test produced the documented summary.
  • Independent Astra review passed on implementation commit 14225acc3. The next commit adds documentation. Independent review also passed on the CI adoption commit 582122f5a.
  • The runtime launcher, real shard execution, weight measurement, runtime exclusion, and CI workflow suites passed on Deno (134 steps). Focused type and format checks passed.

Scope

  • Adds the reporter choice to xmd test for single files and directory targets.
  • Direct terminal output from commands inside tests keeps its normal routing.

Scope confirmation

  • Every changed file supports the reporter or documents its use.
  • The description matches the diff and local verification results.

@taras
taras enabled auto-merge (squash) November 16, 2026 23:35
@taras
taras merged commit a1f2355 into main Nov 17, 2026
67 checks passed
@taras
taras deleted the codex/xmd-test-reporter branch November 17, 2026 00:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant