fix(lobbying): scraper cursor, stats, and backfill reliability at current data scale

[nesanders]

Summary

Fixes three related Firestore document/field size-limit bugs in the lobbying scraper, all of which only surfaced when the weekly incremental scraper (--mode weekly) was run for the first time against the full production-scale dataset (300K+ filings, 25K+ registrants). Prior to this, only --mode backfill had ever actually run.

Context on how these were found: running the weekly scraper against dev surfaced two crashes back to back; fixing the second one surfaced a third, closely related issue. All three are variants of the same root cause — a single Firestore document/field has a hard size ceiling, and several places in this pipeline stored unboundedly-growing state in one doc.

1. Weekly cursor doc exceeded Firestore's 1MB document limit

scrapers/lobbying stored the entire processed-disclosure-URL history and per-registrant summary cache as two fields (processedDiscUrls, summaryDiscCache) on a single document. Since this was the scraper's first-ever weekly run, it had to rebuild that cache from scratch — the document grew past 1MB partway through, and every write after that point failed silently, meaning every registrant processed afterward was skipped for the rest of the run.

Fix: moved to subcollections — scrapers/lobbying/processedUrls/{hash} and scrapers/lobbying/summaryCache/{hash}, one small doc per URL — mirroring the pattern the backfill cursor (scrapers/lobbyingBackfill/processedUrls) already used. Point lookups replace the in-memory set/dict that used to get loaded and saved as one blob each time.

2. compute_stats() crashed on the full-collection stream

compute_stats() read lobbyingFilings/lobbyingRegistrants via one unbounded .stream() call each. At current scale (373K+ filings) the query timed out server-side, and the installed google-cloud-firestore client's automatic stream-retry hit an internal bug (AttributeError: '_UnaryStreamMultiCallable' object has no attribute '_retry') instead of recovering.

Fix: cursor-paginated reads in 50K-doc batches (_iter_collection in writer.py), with retry=None to disable the buggy built-in retry and a manual retry loop that re-issues a fresh, small query on failure instead of trying to resume a broken stream. Verified this reads the full collection correctly (exact matching counts) in ~3.5 minutes total, no crash.

3. billSummaries_{court} JSON blob exceeded the 1MB field-size limit

Once (2) was fixed, compute_stats() reached a third, related limit: the per-court billSummaries_{court} document stores a data field containing JSON.stringify()'d per-bill counts. For the current session (court 194, ~5,600 bills), that string is now 1,057KB — just over Firestore's 1,048,487-byte field limit. Courts 192/193 are close behind (1,015KB / 973KB) and will cross it within the next couple of sessions as filings accumulate.

Fix: same pattern as (1) — one small doc per bill in a bills subcollection (lobbyingMeta/billSummaries_{court}/bills/{billId}) instead of one JSON blob per court. Applied consistently across writer.py (Python scraper), seedLobbyingStats.ts (TS admin script), and the frontend fetcher (components/db/lobbying.ts) so all three writers/readers agree on the new shape.

4. completedYears permanently skipped a year based on a partial scan

Flagged as a separate follow-up in the original PR description, now fixed here too (previously listed under "Not in scope").

run_backfill() marked a year "complete" after one pass and skipped it forever on every future run. That's wrong for the current (still-accruing) year: a run partway through the year would mark it complete based on whatever existed at that moment, silently missing every disclosure filed afterward. This is exactly what happened to 2026 in production — marked complete in July with 0 disclosures captured, meaning no future backfill run would ever see it again.

run_backfill already has a fully correct, granular completeness check — _is_backfill_processed, a per-URL subcollection lookup (same pattern as fix #1). The year-level flag only ever bought a coarse fast-path (skip re-listing a year's registrants entirely), and it's what caused the bug. Removed it: every run now always re-lists every requested year (one cheap HTTP request per year) and relies solely on the per-URL cursor for correctness.

Tests: added tests/test_scrape.py with a small in-memory Firestore fake (real enough to simulate write-then-read-back across calls) covering both cursor systems, including a regression test that reproduces the exact bug scenario. Verified the regression test actually catches the bug by checking out the pre-fix scrape.py and confirming 4/4 backfill tests fail against it, then pass again with the fix restored. Also validated live against dev (--mode backfill --year 2005 --limit 2, run twice) — the year is re-listed both times, already-processed disclosures are correctly not reprocessed.

Validation

All fixes were validated live against the dev Firestore project (not just unit-level):

• Ran the weekly scraper (--limit 3) against dev after the cursor fix — confirmed it correctly skipped already-processed disclosures and correctly wrote new ones, verified the new subcollections were populated as expected.
• Ran _iter_collection standalone against the full lobbyingFilings (373,004 docs) and lobbyingRegistrants (25,633 docs) collections — exact matching counts, no crash, ~3.5 min total.
• Ran compute_stats() end-to-end against dev — completed in 273s, wrote all 11 courts (184–194) including the previously-failing 194th, with correct per-bill data verified directly in Firestore.
• Full project typecheck (npx tsc --noEmit) is clean except for two pre-existing, unrelated errors in functions/lib.

Not in scope

• Deploying the fixed scraper to Cloud Run / wiring up a Cloud Scheduler trigger for --mode weekly is a separate follow-up once this merges.

Steps to test/reproduce

1. cd lobbying-scraper && python3 -m pytest tests/ — 53 tests, all green (45 pre-existing + 8 new in test_scrape.py).
2. cd lobbying-scraper && python3 scrape.py --mode weekly --limit 3 against a project with existing data — should complete without error and skip previously-processed disclosures.
3. yarn firebase-admin run-script seedLobbyingStats --env dev — should complete without a "too many index entries" or field-size error, and write per-bill docs under lobbyingMeta/billSummaries_{court}/bills/.
4. yarn firebase-admin run-script checkLobbyingFreshness --env dev — new diagnostic script; shows recent fetch timestamps, cursor subcollection sizes, and current-year totals.
5. Load /lobbying/bills in the frontend — bill summaries should still render correctly now that they're read from the bills subcollection instead of a JSON blob.

[nesanders]

Migrating to codeforboston#2221