[disnet]

Requested by Tim · project thread

Why

An audit of the account's CPU-ms (October: ~40.6M against the 30M included) found about 27.5M came from a Pages project outside this repo (disnetdev.com, handled separately). Skyreader's own Workers came to about 12M. This PR goes after the biggest avoidable Skyreader costs.

What changes

1. The Semble/Margin membership poll is incremental, and this also fixes a sync bug. pollBackedMembership used to re-resolve every card in the collection on every poll, one getRecord each. Two prod collections had grown past the snapshot's 900-subrequest budget. In the last week, 170 polls ran out part way, so the poll never completed and the collection never synced again. Each of those Saved-list opens still spent ~400ms of CPU (p99 ~1.1s) and ~900 PDS fetches.

Now stored link URIs go to the snapshot as skipLinks, following the pattern room-sync.ts already uses:

• Only new links are resolved, plus a random slice of 25 stored links per poll, so edited or deleted cards are still noticed.
• A stored row leaves when a whole listing no longer names its link, when its URL is tombstoned, or when a re-check finds its card gone.
• A newly resolved member is kept even from an incomplete snapshot, so a big collection fills in across polls.
• An incomplete snapshot still never empties the table and never stamps last_successful_backing_poll. full: true semantics are unchanged.
• New table backed_skipped_links (migration 0118) remembers links that hold no article (note cards, cards with no URL, deleted items), so they are skipped too. It is wiped wherever membership is wiped.
• Enable's eager body fill now runs whenever members landed, not only on a complete poll.

2. Staging cron every 5 minutes (*/5 * * * *). Staging's per-minute cron was ~20% of Skyreader's Workers CPU on an environment with almost no traffic. The cron handler treats */5 as the main cron; all the minute-gated phases (every 5th minute, every 15th, the hour) still land. The document reconcile re-lists 5 authors per run, which keeps the per-minute rate. The admin scales its cron and poller-row staleness thresholds by the cadence it reads from the cron string. RUNBOOK and plan doc updated.

3. Notification bell polls every 3 minutes instead of every minute. Each poll sends 2 requests from every open tab.

4. "No body" answers are remembered across page loads for 24h. Previously, every boot asked again, in batches of 200, about every save imported without its text (~4.6k /api/saved/bodies calls a day, and rising). A body that lands later bumps updatedAt, which still forces a refetch. Opening a save still asks on its own.

Action needed after merge

• Staging heartbeat monitor: set backend-cron (staging) to a 5-minute period with ~15 minutes' grace, or it will page.
• Uptime checks (monitor config, not in this PR): consider running the deep health check every 10 minutes, and staging checks less often. Together they're ~35s CPU a day.

Review

Ran the code-review skill at high effort plus an independent adversarial reviewer.

Fixed:

• Non-article links were re-fetched on every poll, which could exhaust the budget again → backed_skipped_links.
• A known-empty restore race: a second hydrateBodies, or a persist write, could run before the restore and overwrite the saved entries → one shared restore promise, awaited before persisting and before list-surface lookups.
• An early return dropped a re-check's non-transient skips.
• Enable's eager fill was gated on complete.
• Staging's reconcile was running 5× slower.
• Future-dated persisted entries outlived the TTL.
• Stale comments describing the wholesale replace.

Left alone, deliberately:

• Duplicate links (two cards for the same URL) still re-resolve the second link on each poll. This is rare and costs one fetch each.
• Stub enrichment rows for kept members are no longer re-seeded every poll. That only matters if backedUnsave throws mid-delete; the refresh slice re-seeds them eventually, and re-seeding every row every poll is the cost this PR removes.
• Random refresh instead of a rotating oldest-first slice. Rotation needs another column; random coverage is fine for display metadata.
• Percolator nudge and rate-limit cleanup run every 5 minutes in staging. That's acceptable for staging.

Testing

• backend: full suite, 1,877 tests passing. New tests cover skip-and-refresh, removal on a whole listing, keeping partial progress, drop vs. keep on a re-check (unresolvable vs. transient), a card whose URL changed, and non-article links.
• frontend: utils + stores, 1,207 passing; npm run check clean. New batcher tests cover the TTL and persist/restore.
• admin: ops tests, 24 passing, including the cadence scaling.

---

Generated by Claude Code