Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/wpt-focused-runs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@bomb.sh/web-platform-tests': patch
---

Allow runners to filter configured WPT cases before test registration. Provide a Vitest integration and a Router-based `wpt` command that accepts literal search text, so focused runs omit unrelated cases instead of reporting them as skipped.
1 change: 1 addition & 0 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ jobs:
persist-credentials: false
- uses: jdx/mise-action@c2a87611a18de5b3828c5652fe268e992400cb5c # v4.3.0
- run: pnpm install --frozen-lockfile
- run: pnpm --filter @bomb.sh/web-platform-tests build
- name: Run benchmarks with CodSpeed
uses: CodSpeedHQ/action@78102a468f96d8e0ccb92d0ec49425753fc42062 # v5
with:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ jobs:
persist-credentials: false
- uses: jdx/mise-action@c2a87611a18de5b3828c5652fe268e992400cb5c # v4.3.0
- run: pnpm install --frozen-lockfile
- run: pnpm --filter @bomb.sh/web-platform-tests build
- run: pnpm test

dom-wpt:
Expand Down
10 changes: 7 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,15 +15,18 @@ You need the exact Node and pnpm versions listed under `devEngines` in the root

Any version manager works. If you use [mise](https://mise.jdx.dev), which is what our lint, format, and test checks use, export `MISE_IDIOMATIC_VERSION_FILE_ENABLE_TOOLS=node,pnpm` and run `mise install` so it reads those versions from `package.json`.

Then install the dependencies:
Then install the dependencies and build the packages:

```sh
pnpm install
pnpm build
```

## Commands you'll use

Run these from the repo root. You don't need to build before you run the tests.
Run these from the repo root. Tests and benchmarks reuse the built WPT package.
Rebuild it after changing its sources with `pnpm --filter @bomb.sh/web-platform-tests build`;
you can run tests repeatedly without rebuilding otherwise.

| Command | What it does |
| ------------------------------------------ | ----------------------------------------- |
Expand Down Expand Up @@ -63,12 +66,13 @@ Use `expect.poll` here, because the screen doesn't update the moment you change
Before you push, make sure these pass:

```sh
pnpm build
pnpm lint
pnpm format:check
pnpm test
```

CI runs the same three on every pull request, along with the benchmarks and a workflow that publishes a preview build. In your PR, tell us why the change is needed and link the issue if there is one.
CI runs these checks on every pull request, building the WPT package once before tests or benchmarks. A separate workflow publishes a preview build. In your PR, tell us why the change is needed and link the issue if there is one.

## Be kind

Expand Down
4 changes: 3 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,12 @@
"format": "bsh format",
"format:check": "bsh format --check",
"lint": "bsh lint .",
"test": "pnpm --filter @bomb.sh/web-platform-tests build && bsh test"
"test": "vitest run --config vitest.config.mts --experimental.viteModuleRunner",
"wpt": "wpt --file packages/ui/src/dom.compatibility.test.ts"
},
"devDependencies": {
"@bomb.sh/tools": "^0.8.0",
"@bomb.sh/web-platform-tests": "workspace:*",
"@codspeed/vitest-plugin": "^6.1.0",
"@termless/core": "^0.10.2",
"@termless/test": "^0.10.2",
Expand Down
2 changes: 1 addition & 1 deletion packages/ui/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
"build": "bsh build --dts",
"format": "bsh format",
"lint": "bsh lint",
"test": "pnpm --filter @bomb.sh/web-platform-tests build && bsh test"
"test": "vitest run --config ../../vitest.config.mts --experimental.viteModuleRunner"
},
"dependencies": {
"@bomb.sh/tty": "catalog:"
Expand Down
4 changes: 1 addition & 3 deletions packages/ui/src/dom.compatibility.test.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
import { describe, it, expect } from 'vitest';
import { testDOMCompatibility } from '@bomb.sh/web-platform-tests';
import { testDOMCompatibility } from '@bomb.sh/web-platform-tests/vitest/runner';
import { Document } from './dom.ts';
import config from '../test/wpt.config.ts';

testDOMCompatibility({
runner: { describe, it, expect },
config,
create: () => new Document(() => {}),
});
13 changes: 8 additions & 5 deletions packages/ui/test/README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# UI tests

Tests live alongside the implementation in `src/`; this directory contains their shared fixtures and WPT configuration. Run the UI tests from the workspace root:
Tests live in `src/` and this directory, alongside shared fixtures and WPT configuration. Run the UI tests from the workspace root:

```sh
pnpm --filter @clack/ui test
```

The test script builds the private `@bomb.sh/web-platform-tests` package before running Vitest. Use `pnpm test` from the workspace root to run the workspace tests.
Run `pnpm build` once before testing, and rebuild the private `@bomb.sh/web-platform-tests` package after changing its sources. Test commands reuse that build. Use `pnpm test` from the workspace root to run the workspace tests.

## Terminal fixtures

Expand All @@ -26,13 +26,16 @@ Rendering, scheduling, notifications, and lifecycle behavior belong in the UI te

The configuration explicitly excludes unsupported directories. Add includes or narrow exclusions as cases pass. Selecting a source runs all of our adapted cases from that file; it does not imply complete coverage of the upstream file or interface.

Include the source or case in the configuration while implementing it. Vitest's `-t` can narrow the configured selection by source path and case name:
Include the source or case in the configuration while implementing it. Run matching cases by supplying plain text from a source path or case name:

```sh
pnpm --filter @clack/ui test src/dom.compatibility.test.ts -t 'Document-createElement.html.*createElement\(foo\)'
pnpm wpt createElement
pnpm wpt 'createElement(foo) in HTML document'
```

If the shared package is already built, avoid rebuilding by using `pnpm --filter @clack/ui exec bsh test src/dom.compatibility.test.ts` with the same name filter.
The script invokes the package-owned `wpt` CLI, which uses `@bomb.sh/router` to parse arguments and launches Vitest with the UI test entrypoint. It reuses the existing WPT build. The search is a literal substring, so punctuation needs no regex escaping. Omit the search text (`pnpm wpt`) to run all configured cases. Matching happens before Vitest registers tests, so unrelated cases do not appear as skipped.

The root Vitest configuration imports `wptNameFilter()` from `@bomb.sh/web-platform-tests/vitest`; the UI entrypoint uses the worker adapter from `@bomb.sh/web-platform-tests/vitest/runner`. Direct Vitest runs still support `-t` regular expressions. Watch mode retains the full configured catalogue so changing its interactive name filter can discover other cases.

When implementing behavior, read the source-named file under [`src/wpt`](../../web-platform-tests/src/wpt), implement its operations, and update the configuration once its cases pass. Extend the fixture when cases need document parsing, host input, or cleanup. Implementation-specific rendering, notification, and lifecycle checks remain in the UI tests.

Expand Down
38 changes: 34 additions & 4 deletions packages/web-platform-tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,21 +16,51 @@ The WPT suite must contain only faithful adaptations of existing upstream cases.

The shared suite is the long-term home of upstream WPT coverage for the UI. Implementation-specific rendering, notification, and lifecycle checks belong in the corresponding UI integration tests.

### Vitest integration

Register `wptNameFilter()` in the consumer's Vitest configuration:

```ts
import { defineConfig } from 'vitest/config';
import { wptNameFilter } from '@bomb.sh/web-platform-tests/vitest';

export default defineConfig({ plugins: [wptNameFilter()] });
```

The test entrypoint supplies the DOM implementation and case selection:

```ts
import { testDOMCompatibility } from '@bomb.sh/web-platform-tests';
import { describe, it, expect } from 'vitest';
import { testDOMCompatibility } from '@bomb.sh/web-platform-tests/vitest/runner';
import { createDocument, dispose } from './implementation-fixture.ts';
import config from './wpt.config.ts';

testDOMCompatibility({
runner: { describe, it, expect },
config,
create: createDocument,
dispose,
// input: document => createHostInputSession(document),
});
```

The configuration plugin is safe to import outside a test worker. The separate worker adapter supplies Vitest's assertions and filters cases before registration. Direct Vitest `-t` filters work too; watch mode retains the full configured catalogue so interactive filter changes can discover other cases.

The main `@bomb.sh/web-platform-tests` entrypoint remains independent of Vitest. Other runners can use its `testDOMCompatibility()` with their own `runner: { describe, it, expect, filter? }`.

### CLI

The package provides a `wpt` executable using `@bomb.sh/router`. Install Vitest in the consuming workspace and build the WPT package before running it:

```sh
pnpm --filter @bomb.sh/web-platform-tests build
pnpm exec wpt --file path/to/dom.compatibility.test.ts createElement
pnpm exec wpt --file path/to/dom.compatibility.test.ts 'createElement(foo) in HTML document'
pnpm exec wpt --help
```

`--file` selects the consumer's test entrypoint. `--config` optionally selects its Vitest configuration; otherwise Vitest discovers the configuration from the current working directory. Paths are relative to the caller, so the CLI does not depend on this repository's layout or package manager. The configuration must include `wptNameFilter()` and the entrypoint must use the worker adapter shown above.

Search text is an optional literal substring of a source path or case name. Multiple words are joined with spaces; use `--` before search text beginning with a dash. Omit the search to run all configured cases. The CLI forwards Vitest's exit status and never runs a build. Rebuild after changing WPT package sources, then reuse the output across focused runs.

### Selecting tests

The `config` option is optional; omitting it runs all adapted cases. A `WPTConfig` supplies source-path selectors in `include` and `exclude`; an explicit include always wins. Each value is `true` for the entire source file or an array of exact case names and regular expressions matching case names.
Expand All @@ -50,7 +80,7 @@ export default {

Source paths support `*` within a path segment and `**` across directories; `**/` also matches zero directories. Other characters are literal. Case-name strings match exactly. Empty, invalid, or unmatched selectors fail configuration before any cases register, so stale exclusions remain visible.

Cases have no test-to-test prerequisite gates. A selected case exercises its supporting DOM operations and reports failures normally. Globals and host input are resolved lazily when the case uses them. The runner can narrow selected cases further with its usual name filter.
Cases have no test-to-test prerequisite gates. A selected case exercises its supporting DOM operations and reports failures normally. Globals and host input are resolved lazily when the case uses them. An optional `runner.filter(source, name)` narrows configured cases before registering suites or tests with the runner. Configuration is still validated against the full catalogue. If this filter matches nothing, the harness registers nothing.

Run `pnpm generate` after adding, moving, or removing source files. The build also regenerates the registry from filenames. Only files exporting `register` are test entrypoints; upstream support files are imported by those tests. `node scripts/generate-registry.mjs --check` verifies that the checked-in registry is current.

Expand Down
2 changes: 2 additions & 0 deletions packages/web-platform-tests/bin/wpt.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
#!/usr/bin/env node
await import('../dist/cli.mjs');
5 changes: 5 additions & 0 deletions packages/web-platform-tests/knip.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@
"$schema": "https://unpkg.com/knip@6/schema.json",
"entry": [
"src/index.ts",
"src/cli.ts",
"src/vitest.ts",
"src/vitest/runner.ts",
"bin/*.mjs",
"test/*.test.ts",
"test/browser/compatibility.mjs",
"test/browser/run-compatibility.mjs",
Expand All @@ -10,6 +14,7 @@
],
"project": [
"src/**/*.ts",
"bin/*.mjs",
"test/**/*.{ts,mts,mjs}",
"scripts/*.mjs"
]
Expand Down
30 changes: 27 additions & 3 deletions packages/web-platform-tests/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,19 @@
"types": "./dist/index.d.mts",
"import": "./dist/index.mjs"
},
"./package.json": "./package.json"
"./package.json": "./package.json",
"./vitest": {
"types": "./dist/vitest.d.mts",
"import": "./dist/vitest.mjs"
},
"./vitest/runner": {
"types": "./dist/vitest/runner.d.mts",
"import": "./dist/vitest/runner.mjs"
}
},
"scripts": {
"generate": "node scripts/generate-registry.mjs",
"build": "pnpm generate && bsh build src/index.ts --dts",
"build": "pnpm generate && bsh build src/index.ts src/vitest.ts src/vitest/runner.ts src/cli.ts --dts",
"test": "bsh test",
"typecheck": "tsc -p tsconfig.json && tsc -p test/browser/tsconfig.json",
"test:browser:build": "pnpm generate && vite build --config test/browser/vite.config.mjs",
Expand All @@ -30,6 +38,22 @@
"chai": "^6.2.2",
"playwright": "^1.64.1",
"typescript": "^6.0.2",
"vite": "^8.1.0"
"vite": "^8.1.0",
"vitest": "5.0.5"
},
"bin": {
"wpt": "./bin/wpt.mjs"
},
"dependencies": {
"@bomb.sh/router": "^0.8.0",
"zod": "^4.4.3"
},
"peerDependencies": {
"vitest": "^5.0.5"
},
"peerDependenciesMeta": {
"vitest": {
"optional": true
}
}
}
68 changes: 68 additions & 0 deletions packages/web-platform-tests/src/cli.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
import { spawnSync } from 'node:child_process';
import { fileURLToPath } from 'node:url';
import {
argument,
command,
description,
multiple,
name,
option,
parse,
printErrors,
printHelp,
schema,
} from '@bomb.sh/router';
import * as z from 'zod';

const app = command(
name('wpt'),
description('Run configured Web Platform Tests with literal-text filtering.'),
option(
name('file'),
description('Consumer test entrypoint, relative to the current directory.'),
schema(z.string().min(1)),
),
option(
name('config'),
description('Vitest configuration path; otherwise use normal Vitest discovery.'),
schema(z.string().min(1).optional()),
),
argument(
name('search'),
description('Optional literal search text. Multiple words are joined with spaces.'),
multiple(),
schema(z.array(z.string()).default([])),
),
);

const argv = process.argv.slice(2);
const result = parse(app, { argv: argv.length === 0 ? ['--help'] : argv });
if (!result.ok) {
console.error(printErrors(result));
process.exitCode = 1;
} else if (result.method === 'help') {
process.stdout.write(`${printHelp(result)}\n`);
} else {
const text = [
...result.model.search,
...Array.from(result.literals, (literal) => literal.text),
].join(' ');
const args = ['run', result.model.file];
if (result.model.config) {
args.push('--config', result.model.config);
}
if (text) {
// Keep numeric-looking searches as strings through Vitest's argument parser.
args.push('-t', `(?:${text.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')})`);
}
const vitest = fileURLToPath(new URL('./vitest.mjs', import.meta.resolve('vitest/package.json')));
const child = spawnSync(process.execPath, [vitest, ...args], { stdio: 'inherit' });
if (child.error) {
console.error(child.error.message);
}
if (child.signal) {
process.kill(process.pid, child.signal);
} else {
process.exitCode = child.status ?? 1;
}
}
2 changes: 2 additions & 0 deletions packages/web-platform-tests/src/shared/context.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ export interface CompatibilityExpect {
}

export interface CompatibilityRunner {
/** Narrow configured cases before registering suites or tests with the runner. */
filter?(source: string, name: string): boolean;
describe(name: string, register: () => void): unknown;
it: {
(name: string, run: () => void | Promise<void>): unknown;
Expand Down
12 changes: 7 additions & 5 deletions packages/web-platform-tests/src/shared/test-dom-compatibility.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,16 +30,18 @@ export function testDOMCompatibility<D>(options: DOMCompatibilityOptions<D>): vo
},
});
}
const selected = selectTests(cases, options.config);
if (selected.length === 0) {
options.runner.it.todo('implement DOM features and enable their WPT cases');
return;
}
const groups = new Map<string, RegisteredCase[]>();
for (const test of selectTests(cases, options.config)) {
for (const test of selected) {
if (options.runner.filter && !options.runner.filter(test.source, test.name)) continue;
const group = groups.get(test.source) ?? [];
group.push(test);
groups.set(test.source, group);
}
if (groups.size === 0) {
options.runner.it.todo('implement DOM features and enable their WPT cases');
return;
}
for (const [source, tests] of groups) {
options.runner.describe(source, () => {
for (const test of tests)
Expand Down
29 changes: 29 additions & 0 deletions packages/web-platform-tests/src/vitest.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
/* oxlint-disable bombshell-dev/exported-function-async -- Vitest plugin creation is synchronous. */
import type { Plugin } from 'vitest/config';

/** Serializable name filter passed from Vitest configuration to test workers. */
export interface WPTNamePattern {
source: string;
flags: string;
}

declare module 'vitest' {
interface ProvidedContext {
wptTestNamePattern: WPTNamePattern | null;
}
}

/** Pass Vitest's resolved name filter to WPT before test registration. */
export function wptNameFilter(): Plugin {
return {
name: 'wpt-name-filter',
configureVitest({ vitest, project }) {
// Watch mode needs the full catalogue to discover cases when its filter changes.
const pattern = vitest.config.watch ? undefined : vitest.getGlobalTestNamePattern();
project.provide(
'wptTestNamePattern',
pattern ? { source: pattern.source, flags: pattern.flags } : null,
);
},
};
}
Loading
Loading