Verify vendored pgstencil provenance and defer its code audit to pgstencil

.github/audit/application-security.md

@@ -9,19 +9,37 @@
 **Output file:** `audit-application.md`
 
 This is a code-and-specs audit of the product's own boundaries — the remote
-control stack, and the local application. You need no GitHub API access and no
-PAT — do not use one.
+control stack, and the local application. You need no PAT — do not use one.
+The two pgstencil provenance checks below do read the GitHub API, but only a
+public repository, which the workflow's default `GITHUB_TOKEN` and the
+operator's own `gh` login both reach; if that API is unreachable, report those
+two checks as `UNVERIFIABLE`.
 
 For Hosted accounts, read `docs/specs/hosted.md`, `hosted/server/`,
-`hosted/src/`, `hosted/scripts/`, `hosted/wrangler.jsonc`, the packed core/auth
-modules in `vendor/`, and `.github/workflows/hosted-preview.yml` and
+`hosted/src/`, `hosted/scripts/`, `hosted/wrangler.jsonc`, and
+`.github/workflows/hosted-preview.yml` and
 `.github/workflows/hosted-production.yml` — `docs/specs/security-hosted.md`'s
 Deployment boundary quantifies over the preview and production paths, which
 live in those scripts and workflows rather than in the Worker. Verify the
-archive hashes against `vendor/build.json`. Distinguish tested code from
-pending production configuration; do not treat local provider simulations as
-live OAuth acceptance, and treat a checked-in placeholder as no evidence about
-an external control.
+vendored packages by their provenance rather than by reading them: hash each
+archive in `vendor/` against `vendor/build.json`; read each archive's own claim
+with `tar -xOf vendor/<archive>.tgz package/dist/provenance.json` and check
+that it names `build.json`'s commit and does not record `dirty`; then check
+that commit against pgstencil `main` and its audit:
+
+```sh
+gh api repos/diffplug/pgstencil/compare/<commit>...main --jq .status
+gh api repos/diffplug/pgstencil/commits/<commit>/check-runs \
+  --jq '.check_runs[] | select(.name=="security-audit") | .conclusion'
+```
+
+The first must be `ahead` or `identical`, the second `success`. The packed code
+itself is audited in `diffplug/pgstencil` by that repository's own
+`security-audit` workflow against its `SECURITY.md`; do not audit the tarballs'
+contents here — audit how `hosted/` configures the adapter. Distinguish tested
+code from pending production configuration; do not treat local provider
+simulations as live OAuth acceptance, and treat a checked-in placeholder as no
+evidence about an external control.
 
 Read, at minimum: `docs/specs/remote-security-model.md` **and its paired
 `docs/specs/remote-security-model.rationale.md`**, `docs/specs/relay.md`,

docs/specs/hosted.md

@@ -9,7 +9,7 @@
 
 **Must run committed Better Auth migrations before deploying code that needs them, never during a Worker request.** Postgres is reached through an uncached Hyperdrive binding. The runtime creates and closes its database pool within each request.
 
-**Must pin locally packed core/auth packages through root pnpm overrides and commit archives, provenance, and lockfile together.** `vendor/build.json` records the source commit, archive hashes, and `dirty` — true for every `--working-tree` build, which production preflight refuses. No runtime import depends on a sibling checkout. The auth migrations remain owned by the package.
+**Must pin locally packed core/auth packages through root pnpm overrides and commit archives, provenance, and lockfile together.** `vendor/build.json` records the source commit, archive hashes, and `dirty` — true for every `--working-tree` build, which production preflight refuses; each archive's `package/dist/provenance.json` names its source commit, and a pack pgstencil marked dirty cannot be vendored. No runtime import depends on a sibling checkout. The auth migrations remain owned by the package.
 
 **Must declare every peer dependency of the pinned archives in `hosted/package.json`**, so they share Hosted's copy and Renovate updates them.
 
@@ -65,7 +65,7 @@ Source of truth: `touchesHosted` in `hosted/scripts/changed.mjs`; `.github/workf
 
 ## Production releases
 
-**Must deploy only manually selected main revisions after Hosted tests/build and accepted clean package provenance.** `verifyPackages` checks the archive hashes; preflight checks uncached Hyperdrive, matching migration/runtime database identity with distinct roles, and required Worker secret names. Back up, encrypt, decrypt, and restore-test before applying migrations; upload only the encrypted archive. Production has no public candidate URL.
+**Must deploy only manually selected main revisions after Hosted tests/build and accepted clean package provenance.** `verifyPackages` checks the archive hashes and each archive's own packed provenance; preflight checks uncached Hyperdrive, matching migration/runtime database identity with distinct roles, and required Worker secret names. Back up, encrypt, decrypt, and restore-test before applying migrations; upload only the encrypted archive. Production has no public candidate URL.
 
 **Must record an immutable annotated hosted/YYYY-MM-DD tag only after live verification.** Tags identify the deployed commit and verification run/attempt; retries are idempotent and redeployments get new tags. Dating and repeat-deployment suffixes: `recordDeployment`. Code rollback never reverses migrations.
 

docs/specs/security-hosted.md

@@ -23,8 +23,11 @@ Pinned by `hosted/server/tests/workers.test.ts` and `hosted/server/tests/policy.
 
 ## Deployment boundary
 
+**Must vendor pgstencil from a commit on its `main`.** A Dormouse branch may vendor a pgstencil branch while a cross-repo change is in flight; `main` must not merge it until pgstencil has.
+
 - **FAIL IF** a production Worker exposes the captured-email inbox or deterministic clock controls, or imports the testing injection module; inspect `hosted/server/worker.ts`, the build configuration, and `hosted/server/tests/worker-entry.ts`.
-- **FAIL IF** an archive's SHA-256 differs from `vendor/build.json`, the core/auth pnpm overrides cease resolving to those archives, or a runtime import depends on a sibling source checkout.
+- **FAIL IF** an archive's SHA-256 differs from `vendor/build.json`, `build.json` records `dirty`, either archive's `package/dist/provenance.json` is missing, records `dirty`, or names a commit other than `build.json`'s, the core/auth pnpm overrides cease resolving to those archives, or a runtime import depends on a sibling source checkout.
+- **FAIL IF** `vendor/build.json`'s commit is not on pgstencil `main` (`gh api repos/diffplug/pgstencil/compare/<commit>...main`, status `ahead` or `identical`), or that commit's `security-audit` check run (`gh api repos/diffplug/pgstencil/commits/<commit>/check-runs`) is missing or not `success`. pgstencil's own audit is the evidence for the packed code; Dormouse audits only how Hosted configures it.
 - **FAIL IF** the local email inbox accepts a foreign Host or Origin or cross-site Fetch Metadata; inspect `allowedDevRequest` in `hosted/server/dev-host-guard.ts`, including the upgrade guard in `hosted/server/dev.ts`.
 
 - **FAIL IF** preview mail or OAuth calls reach external providers, preview configuration copies production routes/bindings, or a preview exposes deterministic time controls; inspect `hosted/server/preview-worker.ts`, `hosted/scripts/preview.mjs`, and `hosted/server/tests/workers.test.ts`.

docs/specs/security.md

@@ -142,13 +142,14 @@ lints can only read.
 `.github/workflows/security-audit.yaml` audits the repository against these
 specs. Three subagents, each owning the specs below, run every `FAIL IF` as a
 mechanical check with evidence, then read their domain adversarially for what
-no check names. A failure, or a run that reaches no verdict, files a public
+no check names. A failure, or a run reaching no verdict, files a public
 issue labeled
 [`security-audit-failure`](https://github.com/diffplug/dormouse/issues?q=is%3Aissue+label%3Asecurity-audit-failure)
 and holds the release; a later pass closes it. Open issues are live; closed
-ones are the record of what tripped and what changed.
+ones record what tripped and changed.
 `scripts/security-audit-local.sh` runs the same prompts locally.
-[security-audit.md](./security-audit.md) is the contract.
+[security-audit.md](./security-audit.md) is the contract. pgstencil audits the
+packages Hosted consumes in its own repository; these checks verify that link.
 
 | Domain | Specs | Covers |
 | --- | --- | --- |

hosted/README.md

@@ -43,13 +43,19 @@ node scripts/sync-pgstencil.mjs /path/to/pgstencil [revision]
 
 This checks out the pgstencil revision (default `HEAD`) in a temporary clean
 worktree, runs `pnpm packages:pack` there, vendors core/auth, records the commit,
-`dirty: false` and SHA-256 hashes in `vendor/build.json`, and installs. To try
-uncommitted pgstencil changes, pass `--working-tree` instead of a revision; it
-packs the checkout as it stands against its local install and always records
-`dirty: true`, which production preflight rejects. The direct Node command also works before the
-archives exist (pnpm may otherwise auto-install first). See
-`docs/specs/hosted.md` -> "Application boundary" for what has to be committed
-together.
+`dirty: false` and SHA-256 hashes in `vendor/build.json`, and installs. Each
+archive carries its own `package/dist/provenance.json`, and the sync refuses one
+that is missing it, that reports a dirty pack, or that names a commit other than
+the one packed here. To try uncommitted pgstencil changes, pass `--working-tree`
+instead of a revision; it packs the checkout as it stands against its local
+install and always records `dirty: true`, which production preflight rejects.
+The sync also warns when the vendored commit is not on pgstencil's
+`origin/main`: a Dormouse branch may vendor a pgstencil branch while a
+cross-repo change is in flight, but Dormouse `main` must vendor a pgstencil
+`main` commit, and the nightly audit fails until it does. The direct Node
+command also works before the archives exist (pnpm may otherwise auto-install
+first). See `docs/specs/hosted.md` -> "Application boundary" for what has to be
+committed together.
 
 Re-run integration tests after every refresh. Vendor an accepted pgstencil
 revision before a production release.

hosted/scripts/production.mjs

@@ -34,27 +34,50 @@ export async function verifyPackages() {
   const manifest = JSON.parse(
     await readFile(new URL("../../vendor/build.json", import.meta.url), "utf8"),
   );
+  assert.match(manifest.commit, /^[a-f0-9]{40}$/);
   assert.equal(
     manifest.dirty,
     false,
-    "Production requires accepted, clean pgstencil provenance; refresh the vendored packages first",
+    "Production requires a sync of a committed pgstencil revision, not --working-tree; refresh the vendored packages first",
   );
-  assert.match(manifest.commit, /^[a-f0-9]{40}$/);
   assert.deepEqual(
     manifest.files.map((entry) => entry.filename).sort(),
     ["pgstencil-0.1.0.tgz", "pgstencil-auth-0.1.0.tgz"],
     "Expected both pinned pgstencil archives",
   );
   for (const entry of manifest.files) {
     assert.match(entry.filename, /^pgstencil(?:-auth)?-[\w.-]+\.tgz$/);
-    const bytes = await readFile(
-      new URL(`../../vendor/${entry.filename}`, import.meta.url),
-    );
+    const archive = new URL(`../../vendor/${entry.filename}`, import.meta.url);
     assert.equal(
-      createHash("sha256").update(bytes).digest("hex"),
+      createHash("sha256")
+        .update(await readFile(archive))
+        .digest("hex"),
       entry.sha256,
       "Vendored archive checksum mismatch",
     );
+    // The archive's own provenance is what pgstencil's audit is keyed to —
+    // docs/specs/security-hosted.md -> "Deployment boundary".
+    const packed = spawnSync(
+      "tar",
+      ["-xOf", fileURLToPath(archive), "package/dist/provenance.json"],
+      { encoding: "utf8" },
+    );
+    assert.equal(
+      packed.status,
+      0,
+      `${entry.filename} carries no package/dist/provenance.json; refresh the vendored packages first`,
+    );
+    const provenance = JSON.parse(packed.stdout);
+    assert.notEqual(
+      provenance.dirty,
+      true,
+      "Production requires accepted, clean pgstencil provenance; refresh the vendored packages first",
+    );
+    assert.equal(
+      provenance.commit,
+      manifest.commit,
+      `${entry.filename} was packed from a commit other than the one vendor/build.json records`,
+    );
   }
 }
 export async function preflight(env, config, api = cloudflare(env)) {

hosted/server/tests/artifacts.test.ts

@@ -5,23 +5,36 @@ import { execFileSync } from "node:child_process";
 import { fileURLToPath } from "node:url";
 
 test("consumed package bytes match the recorded source snapshot", () => {
-  const provenance = JSON.parse(
-    readFileSync("../vendor/build.json", "utf8"),
-  ) as {
+  const build = JSON.parse(readFileSync("../vendor/build.json", "utf8")) as {
     commit: string;
+    dirty: boolean;
     files: { filename: string; sha256: string }[];
   };
-  expect(provenance.commit).toMatch(/^[a-f0-9]{40}$/);
-  expect(provenance.files.map((file) => file.filename).sort()).toEqual([
+  expect(build.commit).toMatch(/^[a-f0-9]{40}$/);
+  // A --working-tree sync records dirty: true; production preflight refuses it.
+  expect(build.dirty).toBe(false);
+  expect(build.files.map((file) => file.filename).sort()).toEqual([
     "pgstencil-0.1.0.tgz",
     "pgstencil-auth-0.1.0.tgz",
   ]);
-  for (const file of provenance.files)
+  for (const file of build.files) {
     expect(
       createHash("sha256")
         .update(readFileSync("../vendor/" + file.filename))
         .digest("hex"),
     ).toBe(file.sha256);
+    // Each archive names the pgstencil commit it was packed from, so the bytes
+    // themselves — not just build.json — say what pgstencil's own audit covers.
+    const provenance = JSON.parse(
+      execFileSync("tar", [
+        "-xOf",
+        "../vendor/" + file.filename,
+        "package/dist/provenance.json",
+      ]).toString(),
+    ) as { commit: string; dirty?: boolean };
+    expect(provenance.commit).toBe(build.commit);
+    expect(provenance.dirty).not.toBe(true);
+  }
   // Same-version tarball refreshes must update installed code as well as metadata.
   for (const [archive, entry] of [
     ["pgstencil-0.1.0.tgz", "pgstencil"],

scripts/spec-word-budgets.json

@@ -20,7 +20,7 @@
   "docs/specs/remote-security-model.md": 4750,
   "docs/specs/security-audit.md": 1950,
   "docs/specs/security-ci.md": 2700,
-  "docs/specs/security-hosted.md": 450,
+  "docs/specs/security-hosted.md": 550,
   "docs/specs/security-local.md": 3150,
   "docs/specs/security-remote.md": 5750,
   "docs/specs/security-supply-chain.md": 1150,

scripts/sync-pgstencil.mjs

@@ -33,6 +33,40 @@ const run = (command, args, cwd, env = process.env) =>
   execFileSync(command, args, { cwd, stdio: "inherit", env });
 const git = (...args) =>
   execFileSync("git", args, { cwd: repository, encoding: "utf8" }).trim();
+const succeeds = (attempt) => {
+  try {
+    attempt();
+    return true;
+  } catch {
+    return false;
+  }
+};
+// Every pgstencil pack writes package/dist/provenance.json: the archive's own
+// claim of the commit it came from, plus `dirty` when it was packed
+// --allow-dirty. Dormouse verifies that claim instead of auditing the packed
+// code, which pgstencil audits in its own repository —
+// docs/specs/security-hosted.md -> "Deployment boundary".
+const provenanceOf = (archive) => {
+  let raw;
+  try {
+    raw = execFileSync(
+      "tar",
+      ["-xOf", archive, "package/dist/provenance.json"],
+      { encoding: "utf8" },
+    );
+  } catch {
+    throw new Error(`${archive} carries no package/dist/provenance.json`);
+  }
+  let claim;
+  try {
+    claim = JSON.parse(raw);
+  } catch {
+    throw new Error(`${archive}: package/dist/provenance.json is not JSON`);
+  }
+  if (typeof claim?.commit !== "string" || !/^[0-9a-f]{40}$/.test(claim.commit))
+    throw new Error(`${archive}: provenance.json names no commit`);
+  return claim;
+};
 const manifest = JSON.parse(
   readFileSync(resolve(root, "hosted/package.json"), "utf8"),
 );
@@ -56,6 +90,9 @@ const overrides = new Map(
 // --working-tree packs the checkout as it stands, for trying unfinished
 // pgstencil changes. That result depends on local state even when git status is
 // clean, so it is always recorded dirty and production preflight refuses it.
+// Either way `commit` is the HEAD of the checkout that was packed, so it is
+// what each archive's own provenance must name; a pack that reports itself
+// dirty (pgstencil's --allow-dirty) is refused below rather than vendored.
 const commit = git(
   "rev-parse",
   "--verify",
@@ -92,11 +129,22 @@ try {
     run("pnpm", ["install", "--frozen-lockfile"], checkout, pgstencil);
   run("pnpm", ["packages:pack"], checkout, pgstencil);
   mkdirSync(resolve(root, "vendor"), { recursive: true });
-  for (const filename of archives)
-    copyFileSync(
-      resolve(checkout, "dist/packages", filename),
-      resolve(root, "vendor", filename),
-    );
+  for (const filename of archives) {
+    const target = resolve(root, "vendor", filename);
+    copyFileSync(resolve(checkout, "dist/packages", filename), target);
+    // Both archives are held to the commit packed here, so a stale pack —
+    // or two archives disagreeing with each other — stops the sync before
+    // build.json can record a commit the bytes do not carry.
+    const claim = provenanceOf(target);
+    if (claim.dirty === true)
+      throw new Error(
+        `${filename} was packed --allow-dirty; a dirty pack cannot be vendored`,
+      );
+    if (claim.commit !== commit)
+      throw new Error(
+        `${filename} was packed from ${claim.commit}, not the ${commit} packed here`,
+      );
+  }
 } finally {
   if (!workingTree) git("worktree", "remove", "--force", checkout);
 }
@@ -117,3 +165,17 @@ run(
   ["--filter", "dormouse-hosted", "update", "pgstencil", "@pgstencil/auth"],
   root,
 );
+// Dormouse `main` must vendor a pgstencil `main` commit, because the nightly
+// audit reads that commit's pgstencil `security-audit` check run — but a
+// Dormouse branch may vendor a pgstencil branch while a cross-repo change is
+// in flight, so this warns and does not fail.
+const fetched = succeeds(() => git("fetch", "origin", "main"));
+const onMain =
+  fetched &&
+  succeeds(() => git("merge-base", "--is-ancestor", commit, "origin/main"));
+if (!onMain)
+  console.warn(
+    fetched
+      ? `warning: pgstencil ${commit} is not on origin/main — Dormouse main must vendor a pgstencil main commit, and the nightly audit fails until it does`
+      : `warning: could not fetch pgstencil origin/main to check ${commit} — Dormouse main must vendor a pgstencil main commit`,
+  );

vendor/build.json

@@ -1,14 +1,14 @@
 {
-  "commit": "297edf6590e61200857b199d69160f0567bbb3c8",
+  "commit": "ddbf96c828c3e6a1129ff7601afeee2e47f80172",
   "dirty": false,
   "files": [
     {
       "filename": "pgstencil-0.1.0.tgz",
-      "sha256": "5ed674d0c62619a2eaa09c40ea447b730670b92ba3e32ae784755b70ed8f2c7e"
+      "sha256": "f9210dfabe12719c0907e797e3bf5e3cbbcd5b9825f5a53343d493bb1aae895f"
     },
     {
       "filename": "pgstencil-auth-0.1.0.tgz",
-      "sha256": "7ada0c3d2031b3d06bec00c623c4f339d4045fd63b1f7a7fe371bd40ca7b7baa"
+      "sha256": "4671664cb76ec8cbfcd9d19e1d40774971aef58a8b05aaa9af48c3e741805b93"
     }
   ]
 }