diff --git a/.github/workflows/test-smokes-built.yml b/.github/workflows/test-smokes-built.yml index c1b037ed137..dbf32e10668 100644 --- a/.github/workflows/test-smokes-built.yml +++ b/.github/workflows/test-smokes-built.yml @@ -105,14 +105,10 @@ jobs: shell: bash run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" - - name: Compute version marker + - name: Compute build version id: version shell: bash - # BUILD METADATA version marker (+test.): semver-valid and - # range-transparent for every 'quarto-required' gate, while still - # distinguishable from the dev sentinel 99.9.9. NEVER use a '-suffix' - # (prereleases fail plain ranges) or a 4th dot component (not semver). - run: echo "version=$(cat version.txt)+test.$(date +%Y%m%d)" >> "$GITHUB_OUTPUT" + run: echo "version=$(cat version.txt)" >> "$GITHUB_OUTPUT" - name: Build dist tarball uses: ./.github/actions/build-dist-tarball diff --git a/.github/workflows/test-smokes.yml b/.github/workflows/test-smokes.yml index 227a51966af..8dc61d8a1c6 100644 --- a/.github/workflows/test-smokes.yml +++ b/.github/workflows/test-smokes.yml @@ -476,6 +476,9 @@ jobs: env: # Useful as TinyTeX latest release is checked in run-test.sh GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + # This loop owns the per-file ::group:: and ::error:: markers, so the + # test harness must not also emit annotations for this step. + QUARTO_TESTS_GHA_ORCHESTRATED: "1" run: | haserror=0 failed_tests=() @@ -514,6 +517,9 @@ jobs: env: # Useful as TinyTeX latest release is checked in run-test.sh GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + # This loop owns the per-file ::group:: and ::error:: markers, so the + # test harness must not also emit annotations for this step. + QUARTO_TESTS_GHA_ORCHESTRATED: "1" run: | $haserror=$false $failed_tests=@() diff --git a/dev-docs/ci-test-log-grouping-design.md b/dev-docs/ci-test-log-grouping-design.md new file mode 100644 index 00000000000..37637ed2d7c --- /dev/null +++ b/dev-docs/ci-test-log-grouping-design.md @@ -0,0 +1,475 @@ +# Design: GitHub Actions log grouping and failure surfacing for the full (non-bucketed) test run + +Status: **Phases 1–2 implemented** on this branch (Phase 1: annotations + +step summary, `tests/test.ts` + `src/tools/github.ts`; Phase 2: per-file +grouping, `tests/gha-grouping.ts` + the `tests/tools/check-gha-log.ts` +checker; Phase 2.2: navigable, clustered step summaries + trimmed +annotations). Phase 3 remains optional/unscheduled. The fork `workflow_dispatch` +trial matrix (verification item 3) has been run (Linux + Windows, seeded +failures, including the >10-failing-file case) and caught two real bugs, both +fixed on this branch. + +Base: written on top of `test/smoke-tests-built-version` (the built-version +testing branch — see `llm-docs/built-version-testing-architecture.md`). The +non-bucketed full run this design targets is the smoke leg of +`test-smokes-built.yml`, which calls `test-smokes.yml` with `buckets` empty +(binary-mode default `smoke/`), landing in the flat-log path described below. +The playwright and ff-matrix legs pass a non-empty `buckets`, so they take the +per-file bucket loop instead — already grouped by #13787/#13807, orchestrated, +and the harness stays silent there by design. The design is mode-agnostic: in both dev +and binary mode the harness runs in the same `deno test` process, so marker +emission points are identical. + +## Problem + +PR [#13787](https://github.com/quarto-dev/quarto-cli/pull/13787) and +[#13807](https://github.com/quarto-dev/quarto-cli/pull/13807) made bucketed CI +runs navigable by wrapping each test file in a `::group::` at the **workflow +level** (`.github/workflows/test-smokes.yml`, the "Run Smoke Tests as a +bucket on Linux/Windows" per-file loops). That works because the YAML loop +runs `run-tests.sh ` at a time, so group markers stay contiguous. + +The **default path** — `buckets == ''`, the "Run all Smoke Tests +Linux/Windows" steps in `test-smokes.yml` — runs one giant `deno test` over +every discovered test file (in binary mode, `run-tests.sh` defaults this to +`smoke/`). No code on that path emits a single workflow command, so the log is +thousands of flat lines and finding a failure means scrolling or text search. +The smoke leg of `test-smokes-built.yml` reaches this path (empty `buckets`); +the playwright and ff-matrix legs pass non-empty buckets and take the +already-grouped bucket loop instead. + +Additional gaps that apply to *both* paths: + +- No `::error` annotations per failed test → failures are not visible on the + run summary page or PR checks tab (only the playwright integration test + emits one today, in `tests/integration/playwright-tests.test.ts`). +- Nothing writes `GITHUB_STEP_SUMMARY` → no at-a-glance failure table. +- `src/tools/github.ts` already contains a complete, correctly-escaping + workflow-command toolkit (`startGroup`/`endGroup`, `error`/`warning`/`notice` + with `escapeData`/`escapeProperty`, `isGitHubActions()` gating) that is dead + code for tests. + +## Hard constraints (GitHub Actions runner behavior) + +These drive the whole design; sources at the end. + +1. **Groups cannot nest.** A second `::group::` acts as an implicit + `::endgroup::` for the first; content after the inner group ends up + *outside any group*. Still unfixed + ([actions/toolkit#1001](https://github.com/actions/toolkit/issues/1001), + [actions/runner#802](https://github.com/actions/runner/issues/802)). + Consequence: **exactly one owner of grouping per code path**. +2. A stray `::endgroup::` with no open group is harmless — safe to emit + defensively. (Shell caveat: `echo "::endgroup::"` clobbers `$?`; capture + exit codes first. Not an issue in TypeScript.) +3. **Groups cannot be force-expanded** in the UI + ([actions/runner#1036](https://github.com/actions/runner/issues/1036)). + Failure detail must therefore land *outside* groups. +4. **Annotation caps**: 10 `::error` + 10 `::warning` per **step**, 50 total + per **job**; excess silently dropped. Emit at most ~9 per-test error + annotations per step, then one aggregate. +5. **`GITHUB_STEP_SUMMARY`**: 1 MiB per step, max 20 step summaries shown per + job. Content just under the limit can be *silently* dropped + ([actions/runner#4337](https://github.com/actions/runner/issues/4337)) — + self-truncate around ~512 KiB. +6. Annotation messages do **not** render ANSI escapes — strip color codes + before putting captured output into `::error` text or the step summary. + ANSI *inside* group bodies is fine (the log viewer renders it). +7. Parsing is line-based on stdout: commands work from any child process, but + must start at column 0 with a trailing newline. Multi-line messages use + `%0A` escaping (`escapeData` in `src/tools/github.ts` already does this). +8. **Interleaved parallel output corrupts grouping** (process B's `::group::` + implicitly closes process A's). Today this is a non-issue: `run-tests.sh` + invokes `deno test` **without `--parallel`** (the generic-case invocation), + so test files run serially in one process. If `--parallel` is ever added, grouping must move + to buffer-and-flush emission (see Non-goals). + +## Spike results (verified with the pinned Deno v2.7.14) + +A standalone experiment (two test files, markers emitted from inside +`Deno.test` bodies, one seeded failure) confirmed: + +- Marker lines printed from a test body are replayed **verbatim at column 0** + inside Deno's `------- output -------` / `------- post-test output -------` + framing, so the runner parses them. +- Opening a group inside the *first test of a file* yields this net structure + (as the runner parses it): + + ```text + running N tests from ./smoke/foo.test.ts <- visible (outside groups) + [smoke] > first test name ... <- visible + ::group::./smoke/foo.test.ts <- group opens + ...all per-test output and "... ok" lines of the file... + ::endgroup:: <- emitted by THIS file's unload + running M tests from ./smoke/bar.test.ts <- visible + ``` + + i.e. per-file grouping collapses exactly the noise (per-test output blocks + and `ok` lines) while keeping Deno's own file headers visible as structure. + (Phase 2.1 later moved the group open earlier, to registration time, so the + `running N tests from` header and the first-test announcement now land + *inside* the group rather than above it — see "Phase 2.1" below.) +- **Per-file module instances (correction, found via review #1986).** Deno + instantiates each test file's module graph separately: module state resets + between files, and each file gets its own `unload` event, flushed before + the next file's header. The original spike misread the inter-file + `::endgroup::` as the next file's first test closing the previous group + via shared state — it is actually the previous file's own unload. The + observable grouping output is identical either way (which is why the + probes looked right), but it means module-level state is per-FILE: any + genuinely per-step value (the annotation budget) must be coordinated + through the filesystem, and the grouping state machine's cross-file + transition close is in practice a no-op safety net (each instance only + ever sees its own file). +- **On failure**, emitting `::endgroup::` from the failure path *before* + throwing puts the `FAILED` result line **outside** any group (visible in the + collapsed view), with the `::error` annotation adjacent. The next passing + test simply re-opens a group with the same file title (flat sequential + groups — allowed). +- Each file's `unload` `::endgroup::` flushes **before** the next file's + header and before Deno's terminal `ERRORS` / `FAILURES` / summary + sections, so file groups never bleed into the next file and the end-of-run + failure details are never swallowed by the last group. + +Caveat: the exact attribution of output to Deno's `output` vs `post-test +output` blocks is reporter behavior that may shift across Deno upgrades. It +does not affect correctness of the net grouping shown above, but the +verification checker (below) should run again on any Deno bump. + +## Design + +Three phases, each independently shippable and independently revertable. + +### Phase 1 — failure surfacing (no grouping; zero nesting risk) + +All changes in `tests/test.ts`, in the existing failure path that builds the +`━━━ TEST FAILURE:` block (around line 327), reusing `src/tools/github.ts` +helpers (gated on `isGitHubActions()`, so local runs are byte-identical): + +1. **`::error` annotation per failed test**: + `error(message, { file, title })` with + - `file`: repo-relative path of the test file (or the smoke-all `.qmd` + document when the failing test is a smoke-all doc — that is the file a + developer needs to open); + - `title`: the test name (`[smoke] > ...`); + - message: the repro command (`./run-tests.sh `) plus the first ~20 + lines of ANSI-stripped captured output, `%0A`-escaped. + - **Cap and ownership**: GitHub's 10-errors cap is per *workflow step*, + but bucket mode runs many `run-tests.sh` processes inside one step — a + per-process counter cannot implement a per-step budget there, and the + YAML bucket loop already emits its own `::error` per failed file. + Resolution: **the harness emits annotations only when it owns the step** + (`QUARTO_TESTS_GHA_ORCHESTRATED` unset — the default path). One + `deno test` process is the whole step there, BUT a module-level counter + is still NOT a step budget: Deno instantiates each test file's module + graph separately (verified on the pinned 2.7.14 — module state resets + per file and `unload` fires once per file), so the count is coordinated + through a sidecar counter file derived from `GITHUB_STEP_SUMMARY` + (unique per step, runner-writable; no locking needed — files run + sequentially without `--parallel`). Stop at 9; the failure that crosses + the cap emits the single aggregate `::error` inline as the step's 10th + and last annotation (no cross-file end-of-run hook exists to emit it + later). + In bucket mode the YAML loop keeps its per-file `::error` as today. + Known, accepted: if more than 10 bucket files fail, GitHub silently + drops the excess YAML annotations too — the step summary (below) is the + complete record in all modes; annotations are best-effort navigation. +2. **Step summary row per failed test**: append to the file at + `$GITHUB_STEP_SUMMARY` (plain `Deno.writeTextFileSync(..., { append: true })`; + add a small `stepSummary()` helper to `src/tools/github.ts`): + - a table header once per test FILE (module state is per file — see the + spike correction above), then one row per failure: test file / test + name / duration. Per-failure `
output` + blocks (ANSI-stripped excerpt + repro command) are **buffered and + flushed after that file's table at its per-file `unload` event** — GFM + ends a table at the first non-row line, so detail blocks cannot + interleave between rows. Each failing file therefore contributes its + own small table followed by its detail blocks. Accepted consequence: + on a hard process death (panic/OOM) pending unloads never fire and + buffered details are lost — but every row was already appended at + failure time, so the summary tables remain the complete record; + - the size budget must also be per *step*, not per process (bucket mode: + many processes append to the same summary file): before each append, + `stat` the summary file and degrade to name-only rows once it exceeds + ~512 KiB — the file size itself is the cross-process coordination, and + staying at ~half the 1 MiB limit dodges the silent-drop bug. + +Summary rows are emitted in **all** modes (cap-free, cross-process safe); +annotations follow the ownership rule above. Each bucket *job* still gets its +own summary page and annotation set. + +### Phase 2 — per-file grouping for the default path (env-gated harness emission) + +Owner rule: **the harness owns groups and annotations only when nothing else +does.** + +- One env switch, `QUARTO_TESTS_GHA_ORCHESTRATED=1`, set by the two YAML + bucket loops in `test-smokes.yml` alongside their existing + `::group::Testing ${file}` — meaning "an outer orchestrator owns workflow + commands for this step": the harness emits **no groups and no + annotations** (step-summary rows are always emitted; they are cap-free and + size-coordinate via the summary file, see Phase 1). The #13807 bucket log + format does not change at all. Anyone else wrapping `run-tests.sh` in their + own group can use the same switch. +- In `tests/test.ts`, a small module (or an extension of the existing + `src/tools/github.ts` import) tracks the current test-file origin: + - at the start of each test's `execute`, resolve the declaring file (the + harness already knows it — resolved from Deno's per-test `context.origin` + (`testFileFromOrigin`, `tests/test.ts`)); + - on file change: `endGroup()` (defensive, harmless if none open) then + `startGroup(relativePath)`. NOTE (spike correction): since each test + file gets a fresh module instance, an instance in practice only ever + sees its own file — the transition close is a safety net for a future + Deno that shares module state again, not the working mechanism; + - on the failure path (before `fail()` throws): `endGroup()` and clear the + "open" flag, so the `FAILED` line and annotation land outside groups + (verified by the spike). Any failure that escapes the harness's + execute/verify path (init, prereq, setup, teardown — review #1991) must + also close the group before propagating to Deno: an outer catch around + the whole test body closes-and-rethrows, idempotently; + - `globalThis.addEventListener("unload", ...)`: fires once per test FILE + and closes that file's group — this is what actually ends a passing + file's group before the next file's header, and keeps the terminal + `ERRORS`/summary sections ungrouped (verified); + - every emission gated on + `isGitHubActions() && !Deno.env.get("QUARTO_TESTS_GHA_ORCHESTRATED")`. +- **Closure policy: per-file closure at the file's own `unload`** (plus the + failure-path close). A pleasant consequence of per-file module instances: + a direct `Deno.test` file (no harness registration) running between two + harness files executes AFTER the previous file's unload closed its group — + foreign tests' output and `FAILED` lines land ungrouped, visible. The + stale-group swallowing originally accepted under "lazy closure" does not + occur in practice; invariant 3 stays scoped to harness-registered tests + only as a guard against Deno changing its module-instantiation model + (which the checker script would catch). +- Granularity is **per test file**, matching Deno's own "running N tests + from ..." structure. Per-test granularity is the documented fallback if real + logs show files whose single collapsed group is still too coarse (e.g. + `smoke-all.test.ts` runs many documents as separate tests — if needed, treat + each smoke-all *document* as the boundary instead; the harness knows the + document path). + +#### Phase 2.1 — open the group at registration time + +Phase 2 opens a file's group inside the *body of its first test*, which +leaves Deno's `running N tests from ...` header and that first test's +announcement line above the group (the spike snippet above). Phase 2.1 moves +the open earlier — to **registration (module-eval) time**, when the file's +first `test(...)` call runs — so those reporter lines land inside the group. + +- At registration there is no `context.origin` yet, so the declaring file is + recovered from the call stack: `testFileUrlFromStack()` + (`tests/gha-grouping.ts`, kept pure so it is unit-testable with synthetic + stacks) returns the first `.test.ts` frame's `file://` URL, and + `testFileFromStack()` (`tests/test.ts`) turns it into the tests-relative + group title via the existing `testFileFromOrigin`. +- Once per file, via a module-level `registrationGroupAttempted` flag — + per-file module instances (the correction above) make this once-per-file for + free, mirroring `summaryHeaderEmitted` — and gated on `harnessOwnsStep()`, so + local and orchestrated (bucket) runs do no stack resolution and stay + unchanged. +- The body-time `enterTestFileGroup(origin)` from Phase 2 stays as a + self-correcting fallback: a wrong stack guess transitions to the correct + group when the first test runs, and a stack-parse failure is identical to + Phase 2 behavior (the group just opens at body time). Registration-time + opening can therefore only improve placement, never break it. +- This relies on test files registering serially in one process — the same + no-`--parallel` assumption Phase 2 already carries (see the interleaved + parallel-output hard constraint and the `deno test --parallel` non-goal). If + `--parallel` is ever adopted, this moves with the rest of the grouping. + +#### Phase 2.2 — navigable, clustered step summaries + trimmed annotations + +Phases 1–2 emit one step-summary row and one `
` block per failure +plus a per-failure `::error`. On a real 36-failures-per-job run +(cderv/quarto-cli run 29841891595) three problems showed: (a) the table had +no navigation to its detail blocks; (b) 28 of 36 failures shared ONE identical +error yet produced 28 duplicate blocks, wasting the 512 KiB budget and the +reader's time; (c) annotations duplicated the full repro+excerpt already on the +same page. Phase 2.2 addresses all three, in one commit, touching only +`src/tools/github.ts` + `tests/test.ts` (+ unit tests). No workflow or checker +changes; all Phase 1–2 invariants still hold. + +- **Step-wide failure ordinal.** `AnnotationBudget.recordFailure()` already + counts failures once per step via its sidecar file; it now also returns that + count as `ordinal`. In `tests/test.ts` the call moves OUT of the + `harnessOwnsStep()` gate (but stays under `isGitHubActions()`), so every CI + failure — including orchestrated bucket-leg rows — gets an ordinal; the + `emitAnnotation`/`emitAggregate` decisions are still consumed only inside the + gate, and the counter write is a file (not stdout), so orchestrated stdout + stays byte-identical. +- **Failure labels.** Pure `failureLabel(ordinal, runnerOs)` → `L-F7` / `W-F7` + / `M-F7` (Linux/Windows/macOS; unknown → `X`). `RUNNER_OS` is read once in + the gated failure path and passed in; the helper reads no env. The OS prefix + is required because the run summary page concatenates every job's summary — + an unprefixed label would collide across Linux/Windows sections. The label + leads the summary table's new `#` column and precedes each detail block as a + label-only `#### L-F7` heading. +- **No link syntax — plain-text labels (VERIFY-FIRST outcome).** The original + design linked table rows to detail blocks via `[L-F7](#l-f7)` fragment links + against the `#### L-F7` headings. A verification probe settled that this does + NOT work: **fork run 29923715216** (two jobs, prefixed + unprefixed label + probes; authenticated-browser DOM inspection) found step-summary headings + render as bare `

` with **empty `id`**, no `id`s anywhere in the rendered + `markdown-body`, and every fragment link dead — GitHub's step-summary + renderer omits the heading-slug/anchor post-processing that README/issue + rendering applies. So labels are emitted as **plain text** in every row and + heading (no `[…](#…)`); Ctrl+F on a label gives exact two-hit navigation (row + ↔ heading). The OS prefix stays — it keeps that Ctrl+F hit unambiguous across + the concatenated per-job summaries. (This also constrains the future + `ci-run` helper's `verdict`: in-summary deep links are impossible, so + navigation there must be API-based.) +- **Cluster identical errors.** The per-file `pendingSummaryDetails: string[]` + buffer becomes a `Map`, keyed by + `excerptSignature()` = the first THREE non-empty ANSI-stripped excerpt lines + (one line is too generic — many failures share the "Error or Warnings During + Execution" banner — and would over-cluster). Cluster identity is decided at + failure time; the first member's label anchors the cluster and every member's + table row still streams immediately (crash-safety unchanged). At `unload`, + one block per cluster is flushed: `####