test(mosaic): check fake FAPI responses against the OpenAPI spec

.changeset/mosaic-invite-only-role.md

@@ -0,0 +1,2 @@
+---
+---

packages/mosaic/docs/testing.md

@@ -100,7 +100,8 @@ pnpm --filter @clerk/mosaic exec vitest run           # every project
 ```
 
 CI runs feature tests in their own `Mosaic Feature Tests` job, so the repo-wide
-unit test job never needs a browser.
+unit test job never needs a browser. Locally the feature project runs two test
+files at a time, since each one drives a browser.
 
 jsdom and happy-dom have no layout engine, so every rect is 0 and a test that
 asserts on position, size or scrolling always passes there. Name such a test
@@ -188,6 +189,53 @@ handler**, or on a `holdRequests` hold never released or failed. When a
 feature needs a new endpoint, add a handler to `serveFapi` that mirrors what
 FAPI returns.
 
+### Matching the backend
+
+Every FAPI request clerk-js sends, and every mocked response, is checked
+against the FAPI OpenAPI spec, vendored as `fapi-spec.yml`. The check is
+strict: a field the spec doesn't list fails, and so does a missing required
+field or a wrong type. It reads the status and the `_method` override, and
+checks a 4xx or 5xx the spec doesn't document against `ClerkErrors`. A
+violation fails the test from the `afterEach`, with the request and the JSON
+pointer of the bad field.
+
+On the request, the query and the body are checked. Form and multipart values
+are read as the type the spec declares, so `true` passes as a boolean and a
+repeated key as an array. The query params clerk-js adds to every request,
+such as `_clerk_js_version`, pass everywhere. Path parameters and headers are
+not checked.
+
+When a test fails here, fix the builder or the handler so it sends what FAPI
+sends, or the test data so it looks like real data, such as an ID in the
+format a spec `pattern` expects. A builder can make a backend quirk part of its name, for example
+`fapiSamlLinking(verification)` for the bare account that SAML linking
+returns.
+
+When FAPI sends something the spec gets wrong, add an entry to
+`fapi-spec-gaps.ts`:
+
+- `at` and `property` name the spec schema and the field. A request body is
+  the operation's `requestBody` schema. A query is the operation's
+  `x-query-schema`, which the check builds from its parameters.
+- `schema` is what FAPI really sends. `replaces` is the spec's current schema,
+  when the gap widens a field the spec already has.
+- `sentBy` names the `clerk_go` code that sends it, or that reads it for a
+  request field.
+
+The list prunes itself. Once the spec adds the field, or changes the schema
+that `replaces` records, the gap's test fails and the entry should go.
+
+A test that sends a response the spec forbids on purpose, such as a broken
+server response, calls `allowSpecViolation(message)` with the exact violation.
+The test also fails if that violation is never sent.
+
+To pick up backend changes, run the sync, then the feature tests:
+
+```bash
+pnpm --filter @clerk/mosaic sync:fapi-spec   # fetch the spec for SUPPORTED_FAPI_VERSION
+pnpm --filter @clerk/mosaic test:feature
+```
+
 ### Order matters
 
 Call `serveFapi`, then any `holdRequests`, then `renderWithClerk`. Handlers

packages/mosaic/eslint.config.mjs

@@ -5,4 +5,8 @@ import mosaicImportGuards from './eslint/config.mjs';
 
 const root = path.resolve(import.meta.dirname, '../..');
 
-export default [...rootConfig.map(config => ({ ...config, basePath: root })), ...mosaicImportGuards];
+export default [
+  ...rootConfig.map(config => ({ ...config, basePath: root })),
+  ...mosaicImportGuards,
+  { ignores: ['src/__tests__/feature/fapi-spec.yml'] },
+];

packages/mosaic/package.json

@@ -57,6 +57,7 @@
     "lint": "eslint src",
     "lint:attw": "attw --pack . --exclude-entrypoints styles.css --profile esm-only",
     "lint:publint": "publint",
+    "sync:fapi-spec": "node ./scripts/sync-fapi-spec.mjs",
     "test": "vitest run --project primitives --project mosaic",
     "test:browser": "vitest run --project browser",
     "test:ci": "vitest run --project primitives --project mosaic --maxWorkers=70%",
@@ -82,6 +83,7 @@
     "@types/react": "catalog:react",
     "@types/react-dom": "catalog:react",
     "@vitest/eslint-plugin": "1.6.28",
+    "ajv": "^8.20.0",
     "axe-core": "^4.11.3",
     "bundlewatch": "^0.4.2",
     "eslint-plugin-boundaries": "7.3.0",
@@ -93,7 +95,8 @@
     "react": "catalog:react",
     "react-dom": "catalog:react",
     "tsdown": "catalog:repo",
-    "vitest-axe": "^0.1.0"
+    "vitest-axe": "^0.1.0",
+    "yaml": "^2.9.0"
   },
   "peerDependencies": {
     "react": "catalog:peer-react",

packages/mosaic/scripts/sync-fapi-spec.mjs

@@ -0,0 +1,38 @@
+import { readFile, writeFile } from 'node:fs/promises';
+import { resolve } from 'node:path';
+
+const constants = await readFile(
+  resolve(import.meta.dirname, '../../shared/src/internal/clerk-js/constants.ts'),
+  'utf8',
+);
+const version = constants.match(/SUPPORTED_FAPI_VERSION = '([^']+)'/)?.[1];
+if (!version) {
+  throw new Error('Could not read SUPPORTED_FAPI_VERSION from @clerk/shared.');
+}
+
+const specPath = `fapi/${version}.yml`;
+const headers = process.env.GITHUB_TOKEN ? { authorization: `Bearer ${process.env.GITHUB_TOKEN}` } : {};
+const commits = await fetchOk(
+  `https://api.github.com/repos/clerk/openapi-specs/commits?path=${specPath}&per_page=1`,
+  headers,
+).then(response => response.json());
+const sha = commits[0]?.sha;
+if (!sha) {
+  throw new Error(`clerk/openapi-specs has no ${specPath}.`);
+}
+
+const spec = await fetchOk(`https://raw.githubusercontent.com/clerk/openapi-specs/${sha}/${specPath}`).then(response =>
+  response.text(),
+);
+const target = resolve(import.meta.dirname, '../src/__tests__/feature/fapi-spec.yml');
+await writeFile(target, `# clerk/openapi-specs@${sha} ${specPath}\n${spec}`);
+
+console.log(`Synced ${specPath} at ${sha}`);
+
+async function fetchOk(url, headers = {}) {
+  const response = await fetch(url, { headers });
+  if (!response.ok) {
+    throw new Error(`${url} answered ${response.status}`);
+  }
+  return response;
+}