From b5832c5897ff79f54e4e466506a8626585891672 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Thu, 15 Oct 2026 17:22:24 -0400 Subject: [PATCH 1/4] docs(specs): scheduled purge runs on production only; purge takes private data v0.10.2's sandbox ran the sweep against the published branch it shares with production, and purge never deleted private data. Only the deployment with CFP_SCHEDULED_PURGE=true sweeps; purge deletes the person's private profile, legacy password, reset tokens, claim requests and sessions in the same transaction; an admin endpoint clears private data left by earlier purges, counts first. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01TGqwPjt3v4eMfiHpPmCKnR --- docs/operations/deploy.md | 1 + docs/operations/runbook.md | 27 +++++++++++++ plans/purge-gate-and-private-data.md | 59 ++++++++++++++++++++++++++++ specs/api/moderation.md | 29 ++++++++++++++ specs/behaviors/person-lifecycle.md | 4 +- 5 files changed, 118 insertions(+), 2 deletions(-) create mode 100644 plans/purge-gate-and-private-data.md diff --git a/docs/operations/deploy.md b/docs/operations/deploy.md index 3f78fb7..97d90d3 100644 --- a/docs/operations/deploy.md +++ b/docs/operations/deploy.md @@ -273,6 +273,7 @@ comments. Production pod gets these mounted: | `CFP_DATA_RELOAD_SECRET` | **Secret** (`codeforphilly-secrets`) | Shared bearer token for the hot-reload webhook; when unset the `/api/_internal/reload-data` endpoint returns 503. Must match the `CFP_DATA_RELOAD_SECRET` Actions secret in the data repo. See [runbook.md](runbook.md#hot-reload-webhook). | | `CFP_WEB_DIST_PATH` | ConfigMap | `/app/apps/web/dist` | | `CFP_SITE_HOST` | ConfigMap (patched per cluster) | Public-facing host: `next.codeforphilly.org` in prod until cutover, `codeforphilly.org` after; `next-v2.codeforphilly.org` in sandbox. Drives the markdown renderer's external-link transform, canonical URLs in email, and the SAML IdP metadata's `SingleSignOnService` endpoint URLs. | +| `CFP_SCHEDULED_PURGE` | ConfigMap (patched per cluster) | `"true"` **only in cfp-live-cluster**; unset (`"false"`) everywhere else. Turns on the daily [scheduled spam purge](../../specs/behaviors/person-lifecycle.md#scheduled-spam-purge). Sandbox and production share `published`, so a second deployment with it on would be a second destructive writer on production data. | | `SAML_ENTITY_ID` | ConfigMap | Optional. Stable SAML IdP entity ID / assertion `Issuer` (default `https://codeforphilly.org/api/saml/slack/metadata`). Leave unset everywhere Slack should keep trusting the production IdP identity — it deliberately does **not** follow `CFP_SITE_HOST`, so flipping the host at cutover doesn't require re-registering with Slack. Only set it when standing up a separate IdP registration. See [specs/api/saml.md](../../specs/api/saml.md#idp-identity-and-hosts). | | `POSTMARK_SERVER_TOKEN` | **Secret** (`codeforphilly-secrets`) | Postmark server API token for outbound notifications. When unset, the email notifier falls back to a no-op LoggingNotifier — convenient for dev + tests but means no real emails go out. | | `POSTMARK_MESSAGE_STREAM` | ConfigMap | Postmark message stream for outbound mail (default `outbound`). Must exist on the server the token belongs to. | diff --git a/docs/operations/runbook.md b/docs/operations/runbook.md index c5f8ffb..19db209 100644 --- a/docs/operations/runbook.md +++ b/docs/operations/runbook.md @@ -204,6 +204,33 @@ The helper script resolves the current pod by label selector, so it survives res **Read-only by design** — `git upload-pack` only serves fetch; pushing back to the pod would bypass gitsheets + in-memory state and fight the push daemon. Pull what you need to your local clone, reason about it there, then push to `origin` if appropriate. +## Private data left behind by purges + +Purge deletes a person's private data (profile, legacy password, reset tokens, +claim requests, sessions) in the same transaction since v0.10.3. People purged +or pruned before that still have private data. An administrator clears it from +the running API — never with an out-of-process script, because the API holds +the private store in memory and would write the old files back: + +```js +// In the browser console on the site, signed in as an administrator. +const run = (apply) => fetch('/api/admin/private-data/purge-orphans', { + method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ apply }), +}).then((r) => r.json()).then((j) => console.log(j.data)); +run(false); // counts only — check them +run(true); // deletes; cannot be undone (the private store is not versioned) +``` + +It answers counts only, never emails. Spec: +[api/moderation.md](../../specs/api/moderation.md#post-apiadminprivate-datapurge-orphans). + +## Scheduled spam purge runs on production only + +`CFP_SCHEDULED_PURGE=true` is set in cfp-live-cluster alone +([deploy.md](deploy.md)). Sandbox shares `published`, so a sandbox pod with +the flag on would delete production data. A pod logs +`scheduled spam purge disabled` at boot when the flag is off. + ## Helpful commands ```bash diff --git a/plans/purge-gate-and-private-data.md b/plans/purge-gate-and-private-data.md new file mode 100644 index 0000000..ec44ccd --- /dev/null +++ b/plans/purge-gate-and-private-data.md @@ -0,0 +1,59 @@ +--- +status: in-progress +depends: [scheduled-spam-purge] +specs: + - specs/behaviors/person-lifecycle.md + - specs/api/moderation.md +issues: [] +--- + +# Plan: Gate the scheduled purge to production; purge private data too + +## Scope + +v0.10.2 shipped the scheduled spam purge early (2026-10-15). The sandbox ran +the sweep against `published`, which it shares with production, and purged +27 spam-voted accounts there. The accounts were the right ones, but two +flaws showed: every pod runs the sweep, and purge never touched the private +store, so purged people's private profiles stayed behind. + +**In:** `CFP_SCHEDULED_PURGE` (default off; on only in cfp-live-cluster), and +a pod with it off never sweeps and says so at boot; Purge (button and sweep) +deletes private profile, legacy password, reset tokens, claim requests and +session metadata in the same transaction; `POST +/api/admin/private-data/purge-orphans` (admin, `apply` false/true, counts +only) for private data left by earlier purges and prunes. + +**Out:** an out-of-process script for the cleanup (the API's in-memory +private store would write the old data back). + +## Implements + +- [behaviors/person-lifecycle.md](../specs/behaviors/person-lifecycle.md#purge-cascading-hard-delete-admin-only) +- [api/moderation.md](../specs/api/moderation.md#post-apiadminprivate-datapurge-orphans) +- docs/operations/deploy.md (env table), runbook.md. + +## Approach + +1. `PrivateStoreTx.deletePersonData(personId)`, staged by `store.transact` + like the other private writes; `purgePeople` calls it per person. +2. `SessionMetadataStore.removeAllFor(personId)` after the commit. +3. `scheduledPurgeEnabled(config)` gates the plugin. +4. The orphan cleanup runs under `publicRepo.withLock`, so a sign-up that + has written its private profile but not its public record is never + counted. + +## Validation + +- [ ] `npm run type-check && npm run lint && npm test` clean. +- [ ] Tests: the gate; `deletePersonData` persisted to disk, other people + untouched; Purge deletes the private profile; the cleanup is + admin-only, counts without emails, deletes on `apply`, then counts 0. + +## Risks / unknowns + +- `apply: true` can't be undone; the runbook says to count first. + +## Notes + +## Follow-ups diff --git a/specs/api/moderation.md b/specs/api/moderation.md index ed8e44e..91a4812 100644 --- a/specs/api/moderation.md +++ b/specs/api/moderation.md @@ -156,3 +156,32 @@ as the caller, so the vote carries the voter's pseudonymous git identity. deactivate side effect and its reversal rule. - [api/people.md](people.md) — `deactivate`, `reactivate`, `purge`, and the staff-only fields reused here. + +## POST /api/admin/private-data/purge-orphans + +Administrator only (`404` for anyone else, like the rest of `/api/admin`). +Deletes the private data of every person id the private store knows that has +no `people` record: private profile, legacy password, password-reset tokens, +account-claim requests naming them, and session metadata — what +[Purge](../behaviors/person-lifecycle.md#purge-cascading-hard-delete-admin-only) +now deletes in its own transaction, for everyone purged or pruned before it +did. + +### Request + +```json +{ "apply": false } +``` + +`apply` is required: `false` only counts; `true` deletes and returns the same +counts. Run under the write lock, so a member mid-sign-up (private profile +written, public record not yet) is never counted. + +### Response — 200 + +```json +{ "success": true, "data": { "applied": false, "orphanPeople": 27, "profiles": 27, "legacyPasswords": 0, "claimRequests": 0, "sessions": 0 } } +``` + +Counts only — never emails, names or ids. The private store is not versioned, +so `apply: true` cannot be undone; run `false` first. diff --git a/specs/behaviors/person-lifecycle.md b/specs/behaviors/person-lifecycle.md index 6b67a99..258a749 100644 --- a/specs/behaviors/person-lifecycle.md +++ b/specs/behaviors/person-lifecycle.md @@ -24,7 +24,7 @@ The privacy / self-removal path — members should be able to remove themselves; The garbage-collection path for spam — the runtime sibling of the offline spam-prune (behaviors/spam-exclusion.md). - **Who:** administrators only. -- **Mechanism:** one write-mutex transaction that hard-deletes: the `people` record; their `project-membership`; their `help-wanted-interest`; their person `tag-assignment`; AND their authored `project-update`, `project-buzz`, and `blog-post` records (unlike the prune which nulls authorId — purge DELETES the content, it's garbage). +- **Mechanism:** one write-mutex transaction that hard-deletes: the `people` record; their `project-membership`; their `help-wanted-interest`; their person `tag-assignment`; AND their authored `project-update`, `project-buzz`, and `blog-post` records (unlike the prune which nulls authorId — purge DELETES the content, it's garbage). Their **private** data goes in the same transaction: private profile, legacy password credential, password-reset tokens, account-claim requests naming them, and their session metadata. A purged person leaves nothing behind in either store. - **Projects only they touched go too.** In the same transaction, purge hard-deletes every project the person created that nobody else has touched — spam accounts create spam projects, and a project left behind with no members is a soft-deleted husk an admin can't clear. A project qualifies when both hold: 1. **Created by the person:** `createdById` is the person; for a project with no `createdById` (imported from laddr, or created before the field existed), the person is its `maintainerId`. 2. **No one else touched it:** every membership is the person's, every update was authored by them, every buzz posted by them, every help-wanted role posted by them with no interest from anyone else, and no RSVP from anyone else names the project. @@ -38,7 +38,7 @@ The garbage-collection path for spam — the runtime sibling of the offline spam A `spam` vote on the admin members screen deactivates the person at once; three days later, if nobody has disagreed, the site purges them on its own. Staff shouldn't have to come back and press a second button for every spam account, and three days is long enough for another staff member to look and object. -- **When:** an in-process job runs once a day (and once shortly after boot). +- **When:** an in-process job runs once a day (and once shortly after boot) — **only on the deployment where `CFP_SCHEDULED_PURGE=true`, which is production alone.** Sandbox and production serve the same `published` branch, so any pod that runs the sweep is deleting production data; exactly one may. With the flag off the pod never sweeps and logs that at boot. - **Who it takes:** every person who 1. has at least one human `spam` vote, and every human vote on them is `spam` — a `legit` or `uncertain` vote from anyone holds them back; 2. got their newest human vote at least **3 days** ago; From 4ef9f67ecce3ef6d26a43d29dd618146a794673b Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Thu, 15 Oct 2026 17:22:24 -0400 Subject: [PATCH 2/4] fix(api): gate the scheduled purge; purge deletes private data - CFP_SCHEDULED_PURGE (default "false") must be "true" for a pod to sweep; otherwise it logs that the sweep is disabled. - PrivateStoreTx.deletePersonData removes profile, legacy password, reset tokens and claim requests; purge stages it with the public delete and clears session metadata after the commit. - POST /api/admin/private-data/purge-orphans (admin, apply: false|true) clears private data of people with no public record, under the write lock, answering counts only. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01TGqwPjt3v4eMfiHpPmCKnR --- apps/api/src/app.ts | 3 +- apps/api/src/auth/session-metadata.ts | 13 ++ apps/api/src/env.ts | 8 ++ apps/api/src/plugins/spam-purge.ts | 17 ++- apps/api/src/routes/moderation.ts | 36 ++++++ apps/api/src/routes/people.ts | 2 + apps/api/src/services/person-purge.ts | 3 + apps/api/src/services/private-orphans.ts | 78 ++++++++++++ apps/api/src/services/spam-purge.ts | 5 + apps/api/src/store/private/base.ts | 29 ++++- apps/api/src/store/private/interface.ts | 6 + apps/api/src/store/store.ts | 6 +- apps/api/tests/purge-private.test.ts | 152 +++++++++++++++++++++++ 13 files changed, 350 insertions(+), 8 deletions(-) create mode 100644 apps/api/src/services/private-orphans.ts create mode 100644 apps/api/tests/purge-private.test.ts diff --git a/apps/api/src/app.ts b/apps/api/src/app.ts index d3e17d0..00404d8 100644 --- a/apps/api/src/app.ts +++ b/apps/api/src/app.ts @@ -158,12 +158,13 @@ export async function buildApp(opts: BuildAppOptions = {}): Promise { + let removed = 0; + for (const [jti, meta] of this.#map) { + if (meta.personId === personId) { + this.#map.delete(jti); + removed++; + } + } + if (removed > 0) await this.flush(privateStore); + return removed; + } + getAll(personId: string): SessionMeta[] { const result: SessionMeta[] = []; for (const meta of this.#map.values()) { diff --git a/apps/api/src/env.ts b/apps/api/src/env.ts index 3986408..c9f1a2e 100644 --- a/apps/api/src/env.ts +++ b/apps/api/src/env.ts @@ -74,6 +74,13 @@ export const EnvSchema = z.object({ * image; unset in dev (Vite owns 5173). */ CFP_WEB_DIST_PATH: z.string().optional(), + /** + * Whether this pod runs the daily scheduled spam purge + * (specs/behaviors/person-lifecycle.md#scheduled-spam-purge). Off by default: + * sandbox and production share `published`, so exactly one deployment — + * production — may run a destructive sweep against it. + */ + CFP_SCHEDULED_PURGE: z.enum(['true', 'false']).default('false'), /** * Host of the public-facing site (e.g. `codeforphilly.org` in prod, * `next-v2.codeforphilly.org` in sandbox). Used by the server-side @@ -147,6 +154,7 @@ export const envJsonSchema = { SAML_ENTITY_ID: { type: 'string', default: SAML_ENTITY_ID_DEFAULT }, SLACK_TEAM_HOST: { type: 'string', default: 'codeforphilly.slack.com' }, CFP_WEB_DIST_PATH: { type: 'string' }, + CFP_SCHEDULED_PURGE: { type: 'string', enum: ['true', 'false'], default: 'false' }, CFP_SITE_HOST: { type: 'string', default: 'codeforphilly.org' }, POSTMARK_SERVER_TOKEN: { type: 'string' }, POSTMARK_MESSAGE_STREAM: { type: 'string', default: 'outbound' }, diff --git a/apps/api/src/plugins/spam-purge.ts b/apps/api/src/plugins/spam-purge.ts index 3980189..11beb8e 100644 --- a/apps/api/src/plugins/spam-purge.ts +++ b/apps/api/src/plugins/spam-purge.ts @@ -8,6 +8,14 @@ import fp from 'fastify-plugin'; import { SPAM_PURGE_INTERVAL_MS, SpamPurgeService } from '../services/spam-purge.js'; +/** + * Whether this pod runs the sweep: only when CFP_SCHEDULED_PURGE is "true", + * and never under tests (they call the sweep directly). + */ +export function scheduledPurgeEnabled(config: { NODE_ENV: string; CFP_SCHEDULED_PURGE: string }): boolean { + return config.NODE_ENV !== 'test' && config.CFP_SCHEDULED_PURGE === 'true'; +} + declare module 'fastify' { interface FastifyInstance { spamPurge: SpamPurgeService; @@ -20,6 +28,7 @@ async function spamPurgePlugin(fastify: FastifyInstance): Promise { fts: fastify.fts, store: fastify.store, log: fastify.log, + sessionMetadata: fastify.sessionMetadata, }); fastify.decorate('spamPurge', service); @@ -31,8 +40,10 @@ async function spamPurgePlugin(fastify: FastifyInstance): Promise { let interval: ReturnType | undefined; fastify.addHook('onReady', async () => { - // Tests boot many apps; the sweep is exercised directly there. - if (fastify.config.NODE_ENV === 'test') return; + if (!scheduledPurgeEnabled(fastify.config)) { + fastify.log.info('scheduled spam purge disabled (CFP_SCHEDULED_PURGE is not "true")'); + return; + } run(); interval = setInterval(run, SPAM_PURGE_INTERVAL_MS); }); @@ -44,5 +55,5 @@ async function spamPurgePlugin(fastify: FastifyInstance): Promise { export default fp(spamPurgePlugin, { name: 'spam-purge', fastify: '5.x', - dependencies: ['services'], + dependencies: ['services', 'session-middleware'], }); diff --git a/apps/api/src/routes/moderation.ts b/apps/api/src/routes/moderation.ts index 87335e6..35212fc 100644 --- a/apps/api/src/routes/moderation.ts +++ b/apps/api/src/routes/moderation.ts @@ -14,6 +14,7 @@ import { ApiNotFoundError, ApiValidationError } from '../lib/errors.js'; import { getCallerSession } from '../services/permissions.js'; import { buildTransactionOptions } from '../store/commit-meta.js'; import type { MemberOrigin, VoteFilter } from '../services/moderation.js'; +import { findOrphanPrivateData, purgeOrphanPrivateData } from '../services/private-orphans.js'; function requireStaffOr404(request: FastifyRequest): void { const level = request.session.accountLevel; @@ -139,4 +140,39 @@ export async function moderationRoutes(fastify: FastifyInstance): Promise }); }, ); + + // POST /api/admin/private-data/purge-orphans (administrator only) + // Spec: specs/api/moderation.md → POST /api/admin/private-data/purge-orphans + fastify.post( + '/api/admin/private-data/purge-orphans', + { + schema: { + tags: ['admin'], + summary: 'Delete private data of people with no public record (admin only)', + body: { + type: 'object', + properties: { apply: { type: 'boolean' } }, + required: ['apply'], + additionalProperties: false, + }, + }, + }, + async (request) => { + if (request.session.accountLevel !== 'administrator') throw new ApiNotFoundError('Not found'); + const { apply } = request.body as { apply: boolean }; + // Under the write lock: account creation writes the private profile + // before the public record, and must never look orphaned mid-write. + const found = await fastify.publicRepo.withLock(async () => { + const publicIds = new Set(fastify.inMemoryState.people.keys()); + const result = await findOrphanPrivateData(publicIds, fastify.store.private, fastify.sessionMetadata); + if (apply && result.personIds.length > 0) { + await purgeOrphanPrivateData(result, fastify.store.private, fastify.sessionMetadata); + } + return result; + }); + const { personIds, ...counts } = found; + request.log.info({ apply, orphanPeople: personIds.length, ...counts }, 'private orphan purge'); + return ok({ applied: apply, orphanPeople: personIds.length, ...counts }); + }, + ); } diff --git a/apps/api/src/routes/people.ts b/apps/api/src/routes/people.ts index 1d4dde0..ba3de3b 100644 --- a/apps/api/src/routes/people.ts +++ b/apps/api/src/routes/people.ts @@ -239,6 +239,7 @@ export async function peopleRoutes(fastify: FastifyInstance): Promise { }, }, async (request, reply) => { const { slug } = request.params as { slug: string }; + const personId = fastify.inMemoryState.personIdBySlug.get(slug); const result = await fastify.store.transact( buildTransactionOptions({ request, @@ -250,6 +251,7 @@ export async function peopleRoutes(fastify: FastifyInstance): Promise { async (tx) => fastify.services.peopleWrite.purge(tx, slug, request.session), ); result.value.stateApply.apply(fastify.inMemoryState, fastify.fts); + if (personId) await fastify.sessionMetadata.removeAllFor(personId, fastify.store.private); return reply.code(204).send(); }); diff --git a/apps/api/src/services/person-purge.ts b/apps/api/src/services/person-purge.ts index b398cda..cef740d 100644 --- a/apps/api/src/services/person-purge.ts +++ b/apps/api/src/services/person-purge.ts @@ -43,6 +43,9 @@ async function purgePersonRecords( // 1. Delete the person record. await tx.public.people.delete(existing); stateApply.removePerson(personId, existing.slug); + // Their private data goes in the same transaction (profile, legacy password, + // reset tokens, claim requests). + tx.private.deletePersonData(personId); // 2. Cascade-delete project-memberships. const membershipIds = state.membershipsByPerson.get(existing.slug) ?? new Set(); diff --git a/apps/api/src/services/private-orphans.ts b/apps/api/src/services/private-orphans.ts new file mode 100644 index 0000000..3bd6c33 --- /dev/null +++ b/apps/api/src/services/private-orphans.ts @@ -0,0 +1,78 @@ +/** + * Private data left by people who no longer exist publicly — the cleanup + * behind `POST /api/admin/private-data/purge-orphans` + * (specs/api/moderation.md). Purge deletes private data in the same + * transaction now; everyone purged or pruned before that left a private + * profile, legacy password, claim requests and session metadata behind. + * + * Runs inside the API on purpose: the API holds the private store in memory + * and rewrites whole files on each private write, so an out-of-process + * cleanup would be undone by the next write. + */ +import type { SessionMetadataStore } from '../auth/session-metadata.js'; +import type { PrivateStore } from '../store/private/index.js'; + +export interface OrphanPrivateData { + /** Person ids with private data and no public record. */ + readonly personIds: string[]; + readonly profiles: number; + readonly legacyPasswords: number; + readonly claimRequests: number; + readonly sessions: number; +} + +/** Find private data whose person is not in `publicIds`. */ +export async function findOrphanPrivateData( + publicIds: ReadonlySet, + privateStore: PrivateStore, + sessions: SessionMetadataStore, +): Promise { + const orphans = new Set(); + let profiles = 0; + let legacyPasswords = 0; + let claimRequests = 0; + let sessionCount = 0; + + for await (const profile of privateStore.listAllProfiles()) { + if (!publicIds.has(profile.personId)) { + orphans.add(profile.personId); + profiles++; + } + } + for (const personId of privateStore.indices.legacyPasswordByPersonId.keys()) { + if (!publicIds.has(personId)) { + orphans.add(personId); + legacyPasswords++; + } + } + for (const req of await privateStore.listAllClaimRequests()) { + const ids = [req.claimedPersonId, req.requesterPersonId].filter((id): id is string => !!id); + const gone = ids.filter((id) => !publicIds.has(id)); + if (gone.length > 0) { + gone.forEach((id) => orphans.add(id)); + claimRequests++; + } + } + for (const personId of orphans) sessionCount += sessions.getAll(personId).length; + + return { + personIds: [...orphans].sort(), + profiles, + legacyPasswords, + claimRequests, + sessions: sessionCount, + }; +} + +/** Delete the private data of `found.personIds`. */ +export async function purgeOrphanPrivateData( + found: OrphanPrivateData, + privateStore: PrivateStore, + sessions: SessionMetadataStore, +): Promise { + await privateStore.transact(async (tx) => { + for (const personId of found.personIds) tx.deletePersonData(personId); + }); + for (const personId of found.personIds) await sessions.removeAllFor(personId, privateStore); +} + diff --git a/apps/api/src/services/spam-purge.ts b/apps/api/src/services/spam-purge.ts index ed1dc75..94e3ac5 100644 --- a/apps/api/src/services/spam-purge.ts +++ b/apps/api/src/services/spam-purge.ts @@ -11,6 +11,7 @@ import type { FtsEngine } from '../store/fts.js'; import type { InMemoryState } from '../store/memory/state.js'; import { StateApply } from '../store/state-apply.js'; import type { Store } from '../store/store.js'; +import type { SessionMetadataStore } from '../auth/session-metadata.js'; import { purgePeople } from './person-purge.js'; /** How long a spam vote stands before the sweep acts on it. */ @@ -48,6 +49,7 @@ export interface SpamPurgeDeps { readonly fts: FtsEngine; readonly store: Store; readonly log: FastifyBaseLogger; + readonly sessionMetadata: SessionMetadataStore; } export class SpamPurgeService { @@ -86,6 +88,9 @@ export class SpamPurgeService { throw err; } stateApply.apply(this.#d.state, this.#d.fts); + for (const person of queued) { + await this.#d.sessionMetadata.removeAllFor(person.id, this.#d.store.private); + } this.#d.log.info({ purged: slugs }, 'scheduled spam purge'); return slugs; } diff --git a/apps/api/src/store/private/base.ts b/apps/api/src/store/private/base.ts index 591f0bd..a210d73 100644 --- a/apps/api/src/store/private/base.ts +++ b/apps/api/src/store/private/base.ts @@ -174,6 +174,7 @@ export abstract class BasePrivateStore implements PrivateStore { const profilesSnapshot = new Map(this.profiles); const legacySnapshot = new Map(this.legacyPasswords); const claimRequestsSnapshot = new Map(this.claimRequests); + const passwordTokensSnapshot = new Map(this.passwordTokens); // Staged mutations applied only in-memory during the handler const stagedProfilePuts: Map = new Map(); @@ -181,6 +182,7 @@ export abstract class BasePrivateStore implements PrivateStore { const stagedLegacyPuts: Map = new Map(); const stagedLegacyDeletes: Set = new Set(); const stagedClaimRequestPuts: Map = new Map(); + const stagedPersonPurges: Set = new Set(); const tx: PrivateStoreTx = { putProfile: (profile) => { @@ -205,6 +207,11 @@ export abstract class BasePrivateStore implements PrivateStore { const parsed = AccountClaimRequestSchema.parse(req); stagedClaimRequestPuts.set(parsed.id, parsed); }, + deletePersonData: (personId) => { + stagedPersonPurges.add(personId); + stagedProfilePuts.delete(personId); + stagedLegacyPuts.delete(personId); + }, }; let result: T; @@ -215,6 +222,7 @@ export abstract class BasePrivateStore implements PrivateStore { this.profiles = profilesSnapshot; this.legacyPasswords = legacySnapshot; this.claimRequests = claimRequestsSnapshot; + this.passwordTokens = passwordTokensSnapshot; this.rebuildIndices(); throw err; } @@ -235,19 +243,33 @@ export abstract class BasePrivateStore implements PrivateStore { for (const [id, req] of stagedClaimRequestPuts) { this.claimRequests.set(id, req); } + for (const personId of stagedPersonPurges) { + this.profiles.delete(personId); + this.legacyPasswords.delete(personId); + for (const [id, req] of this.claimRequests) { + if (req.claimedPersonId === personId || req.requesterPersonId === personId) this.claimRequests.delete(id); + } + for (const [hash, token] of this.passwordTokens) { + if (token.personId === personId) this.passwordTokens.delete(hash); + } + } this.rebuildIndices(); // Flush to backend const flushOps: Promise[] = []; - if (stagedProfilePuts.size > 0 || stagedProfileDeletes.size > 0) { + const purging = stagedPersonPurges.size > 0; + if (purging || stagedProfilePuts.size > 0 || stagedProfileDeletes.size > 0) { flushOps.push(this.flushProfiles()); } - if (stagedLegacyPuts.size > 0 || stagedLegacyDeletes.size > 0) { + if (purging || stagedLegacyPuts.size > 0 || stagedLegacyDeletes.size > 0) { flushOps.push(this.flushLegacyPasswords()); } - if (stagedClaimRequestPuts.size > 0) { + if (purging || stagedClaimRequestPuts.size > 0) { flushOps.push(this.flushClaimRequests()); } + if (purging) { + flushOps.push(this.flushPasswordTokens()); + } try { await Promise.all(flushOps); @@ -257,6 +279,7 @@ export abstract class BasePrivateStore implements PrivateStore { this.profiles = profilesSnapshot; this.legacyPasswords = legacySnapshot; this.claimRequests = claimRequestsSnapshot; + this.passwordTokens = passwordTokensSnapshot; this.rebuildIndices(); throw new PrivateStoreError( 'private_store_unavailable', diff --git a/apps/api/src/store/private/interface.ts b/apps/api/src/store/private/interface.ts index d0cfd39..c3ca282 100644 --- a/apps/api/src/store/private/interface.ts +++ b/apps/api/src/store/private/interface.ts @@ -26,6 +26,12 @@ export interface PrivateStoreTx { putLegacyPassword(cred: LegacyPasswordCredential): void; deleteLegacyPassword(personId: string): void; putClaimRequest(req: AccountClaimRequest): void; + /** + * Delete everything the private store holds for a person: profile, legacy + * password, password-reset tokens, and account-claim requests naming them + * (specs/behaviors/person-lifecycle.md → Purge). + */ + deletePersonData(personId: string): void; } /** diff --git a/apps/api/src/store/store.ts b/apps/api/src/store/store.ts index cb3ce8b..4f72596 100644 --- a/apps/api/src/store/store.ts +++ b/apps/api/src/store/store.ts @@ -79,6 +79,7 @@ export class Store { const stagedLegacyPasswordPuts: LegacyPasswordCredential[] = []; const stagedLegacyPasswordDeletes: string[] = []; const stagedClaimRequestPuts: AccountClaimRequest[] = []; + const stagedPersonPurges: string[] = []; const privateTx: PrivateStoreTx = { putProfile: (profile) => { stagedPrivatePuts.push(profile); }, @@ -86,6 +87,7 @@ export class Store { putLegacyPassword: (cred) => { stagedLegacyPasswordPuts.push(cred); }, deleteLegacyPassword: (personId) => { stagedLegacyPasswordDeletes.push(personId); }, putClaimRequest: (req) => { stagedClaimRequestPuts.push(req); }, + deletePersonData: (personId) => { stagedPersonPurges.push(personId); }, }; const hasPrivateMutations = () => @@ -93,7 +95,8 @@ export class Store { stagedPrivateProfileDeletes.length > 0 || stagedLegacyPasswordPuts.length > 0 || stagedLegacyPasswordDeletes.length > 0 || - stagedClaimRequestPuts.length > 0; + stagedClaimRequestPuts.length > 0 || + stagedPersonPurges.length > 0; const flushPrivate = async (): Promise => { if (!hasPrivateMutations()) return; @@ -103,6 +106,7 @@ export class Store { for (const cred of stagedLegacyPasswordPuts) tx.putLegacyPassword(cred); for (const id of stagedLegacyPasswordDeletes) tx.deleteLegacyPassword(id); for (const req of stagedClaimRequestPuts) tx.putClaimRequest(req); + for (const id of stagedPersonPurges) tx.deletePersonData(id); }); }; diff --git a/apps/api/tests/purge-private.test.ts b/apps/api/tests/purge-private.test.ts new file mode 100644 index 0000000..853eb7f --- /dev/null +++ b/apps/api/tests/purge-private.test.ts @@ -0,0 +1,152 @@ +/** + * Purge removes private data too, the scheduled sweep is opt-in per pod, and + * the admin orphan cleanup clears private data left by earlier purges. + * Specs: behaviors/person-lifecycle.md → Purge, Scheduled spam purge; + * api/moderation.md → POST /api/admin/private-data/purge-orphans. + */ +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import type { FastifyInstance } from 'fastify'; + +import { buildApp } from '../src/app.js'; +import { mintSessionFor } from '../src/auth/issue.js'; +import { scheduledPurgeEnabled } from '../src/plugins/spam-purge.js'; +import { FilesystemPrivateStore } from '../src/store/private/index.js'; +import { createFullDataRepo, createPrivateStorageDir } from './helpers/test-full-repo.js'; +import { seedRawToml } from './helpers/seed-fixtures.js'; + +const JWT_KEY = 'test-jwt-signing-key-at-least-32-chars!!'; +const T = '2026-05-01T00:00:00Z'; +const ID = { + alice: '01951a3c-0000-7000-8000-a9000000cafe', + bob: '01951a3c-0000-7000-8000-b9000000cafe', + admin: '01951a3c-0000-7000-8000-d9000000cafe', + staff: '01951a3c-0000-7000-8000-c9000000cafe', + ghost: '01951a3c-0000-7000-8000-e9000000beef', +}; + +const profile = (personId: string, email: string) => ({ personId, email, emailRefreshedAt: T, updatedAt: T }); +const claim = (id: string, claimedPersonId: string) => ({ + id, + type: 'pre-onboarding' as const, + claimedPersonId, + claimedSlug: 'someone', + requesterGithubLogin: 'someone', + requesterGithubId: 1, + requesterPersonId: null, + evidence: '', + status: 'open' as const, + submittedAt: T, + reviewedAt: null, + reviewedBy: null, + reviewedReason: null, +}); + +describe('scheduledPurgeEnabled', () => { + it('runs only with CFP_SCHEDULED_PURGE=true outside tests', () => { + expect(scheduledPurgeEnabled({ NODE_ENV: 'production', CFP_SCHEDULED_PURGE: 'true' })).toBe(true); + expect(scheduledPurgeEnabled({ NODE_ENV: 'production', CFP_SCHEDULED_PURGE: 'false' })).toBe(false); + expect(scheduledPurgeEnabled({ NODE_ENV: 'test', CFP_SCHEDULED_PURGE: 'true' })).toBe(false); + }); +}); + +describe('PrivateStoreTx.deletePersonData', () => { + it('removes profile, legacy password, reset tokens and claim requests, and only that person\'s', async () => { + const dir = await createPrivateStorageDir(); + try { + const store = new FilesystemPrivateStore({ CFP_PRIVATE_STORAGE_PATH: dir.path }); + await store.load(); + await store.putProfile(profile(ID.alice, 'alice@example.org')); + await store.putProfile(profile(ID.bob, 'bob@example.org')); + await store.putLegacyPassword({ personId: ID.alice, passwordHash: 'x', importedAt: T }); + await store.putPasswordToken({ tokenHash: 'a'.repeat(64), personId: ID.alice, issuedAt: T, expiresAt: T }); + await store.putClaimRequest(claim('01951a3c-0000-7000-8000-0000000000c1', ID.alice)); + await store.putClaimRequest(claim('01951a3c-0000-7000-8000-0000000000c2', ID.bob)); + + await store.transact(async (tx) => tx.deletePersonData(ID.alice)); + + // Read back from disk, not just memory. + const reread = new FilesystemPrivateStore({ CFP_PRIVATE_STORAGE_PATH: dir.path }); + await reread.load(); + expect(await reread.getProfile(ID.alice)).toBeNull(); + expect(await reread.findPersonIdByEmail('alice@example.org')).toBeNull(); + expect(await reread.getLegacyPassword(ID.alice)).toBeNull(); + expect(await reread.getPasswordToken('a'.repeat(64))).toBeNull(); + expect((await reread.listAllClaimRequests()).map((r) => r.claimedPersonId)).toEqual([ID.bob]); + expect(await reread.getProfile(ID.bob)).not.toBeNull(); + } finally { + await dir.cleanup(); + } + }); +}); + +describe('purge and the orphan cleanup', () => { + let dataRepo: { path: string; cleanup: () => Promise }; + let privateStore: { path: string; cleanup: () => Promise }; + let app: FastifyInstance; + const cookie = async (id: string, level: 'user' | 'staff' | 'administrator') => + `cfp_session=${(await mintSessionFor(id, level, JWT_KEY)).accessToken}`; + + beforeAll(async () => { + dataRepo = await createFullDataRepo(); + privateStore = await createPrivateStorageDir(); + for (const [slug, id, level] of [ + ['alice', ID.alice, 'user'], + ['bob', ID.bob, 'user'], + ['admin-user', ID.admin, 'administrator'], + ['staff-user', ID.staff, 'staff'], + ] as const) { + const toml = [`id = "${id}"`, `slug = "${slug}"`, `fullName = "Test ${slug}"`, `accountLevel = "${level}"`, `createdAt = "${T}"`, `updatedAt = "${T}"`].join('\n'); + await seedRawToml(dataRepo.path, `people/${slug}.toml`, toml, `seed ${slug}`); + } + app = await buildApp({ + serverOptions: { logger: false }, + overrideEnv: { + CFP_DATA_REPO_PATH: dataRepo.path, + STORAGE_BACKEND: 'filesystem', + CFP_PRIVATE_STORAGE_PATH: privateStore.path, + CFP_JWT_SIGNING_KEY: JWT_KEY, + NODE_ENV: 'test', + }, + }); + await app.store.private.putProfile(profile(ID.alice, 'alice@example.org')); + await app.store.private.putProfile(profile(ID.bob, 'bob@example.org')); + // Left behind by a purge from before purge cleaned up private data. + await app.store.private.putProfile(profile(ID.ghost, 'ghost@example.org')); + await app.store.private.putLegacyPassword({ personId: ID.ghost, passwordHash: 'x', importedAt: T }); + }, 60_000); + + afterAll(async () => { + await app.close(); + await dataRepo.cleanup(); + await privateStore.cleanup(); + }); + + it('Purge deletes the person\'s private profile in the same transaction', async () => { + const res = await app.inject({ method: 'POST', url: '/api/people/alice/purge', headers: { cookie: await cookie(ID.admin, 'administrator') } }); + expect(res.statusCode).toBe(204); + expect(await app.store.private.getProfile(ID.alice)).toBeNull(); + expect(await app.store.private.getProfile(ID.bob)).not.toBeNull(); + }); + + it('the orphan cleanup is admin-only, counts first, then deletes', async () => { + const url = '/api/admin/private-data/purge-orphans'; + const staff = await app.inject({ method: 'POST', url, payload: { apply: false }, headers: { cookie: await cookie(ID.staff, 'staff') } }); + expect(staff.statusCode).toBe(404); + + const admin = await cookie(ID.admin, 'administrator'); + const dry = await app.inject({ method: 'POST', url, payload: { apply: false }, headers: { cookie: admin } }); + expect(dry.statusCode).toBe(200); + expect(dry.json().data).toEqual({ applied: false, orphanPeople: 1, profiles: 1, legacyPasswords: 1, claimRequests: 0, sessions: 0 }); + expect(dry.body).not.toContain('ghost@example.org'); + expect(await app.store.private.getProfile(ID.ghost)).not.toBeNull(); + + const applied = await app.inject({ method: 'POST', url, payload: { apply: true }, headers: { cookie: admin } }); + expect(applied.json().data.applied).toBe(true); + expect(await app.store.private.getProfile(ID.ghost)).toBeNull(); + expect(await app.store.private.getLegacyPassword(ID.ghost)).toBeNull(); + expect(await app.store.private.getProfile(ID.bob)).not.toBeNull(); + + const again = await app.inject({ method: 'POST', url, payload: { apply: false }, headers: { cookie: admin } }); + expect(again.json().data.orphanPeople).toBe(0); + }); +}); From 86c7d03aa0e6805b6b700d863afdc2fb9522352f Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Thu, 15 Oct 2026 17:22:43 -0400 Subject: [PATCH 3/4] docs(plans): close out purge-gate-and-private-data Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01TGqwPjt3v4eMfiHpPmCKnR --- plans/purge-gate-and-private-data.md | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/plans/purge-gate-and-private-data.md b/plans/purge-gate-and-private-data.md index ec44ccd..832d0b1 100644 --- a/plans/purge-gate-and-private-data.md +++ b/plans/purge-gate-and-private-data.md @@ -1,5 +1,5 @@ --- -status: in-progress +status: done depends: [scheduled-spam-purge] specs: - specs/behaviors/person-lifecycle.md @@ -45,8 +45,8 @@ private store would write the old data back). ## Validation -- [ ] `npm run type-check && npm run lint && npm test` clean. -- [ ] Tests: the gate; `deletePersonData` persisted to disk, other people +- [x] `npm run type-check && npm run lint && npm test` clean. +- [x] Tests: the gate; `deletePersonData` persisted to disk, other people untouched; Purge deletes the private profile; the cleanup is admin-only, counts without emails, deletes on `apply`, then counts 0. @@ -56,4 +56,17 @@ private store would write the old data back). ## Notes +- Incident: #253 merged 35 s before Release #254 was cut, so v0.10.2 + carried it; the sandbox's boot sweep wrote 5e11d592e ("people: + scheduled spam purge (27)") to the shared `published`. Production was + held on v0.10.1 and goes straight to v0.10.3. +- Devbox (flag unset): the pod logged "scheduled spam purge disabled" and + wrote no data commit. +- Chose an in-API admin endpoint over the operator script first proposed: + the API rewrites whole private-store files from memory on its next + private write, so a script beside a live pod would be undone. + ## Follow-ups + +- **None.** Operator steps (set the flag in cfp-live-cluster, run the + cleanup with Chris's OK) are in the runbook and with the release. From f641aa01dbb26a2fc3b2e0de2fe7dbaf3b0ac2b5 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Thu, 15 Oct 2026 17:22:48 -0400 Subject: [PATCH 4/4] docs(plans): record purge-gate-and-private-data PR Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01TGqwPjt3v4eMfiHpPmCKnR --- plans/purge-gate-and-private-data.md | 1 + 1 file changed, 1 insertion(+) diff --git a/plans/purge-gate-and-private-data.md b/plans/purge-gate-and-private-data.md index 832d0b1..d7edf51 100644 --- a/plans/purge-gate-and-private-data.md +++ b/plans/purge-gate-and-private-data.md @@ -5,6 +5,7 @@ specs: - specs/behaviors/person-lifecycle.md - specs/api/moderation.md issues: [] +pr: 255 --- # Plan: Gate the scheduled purge to production; purge private data too