Skip to content

Finance backfill - #2315

Open
J-C-L wants to merge 15 commits into
codeforboston:mainfrom
J-C-L:finance-backfill
Open

Finance backfill#2315
J-C-L wants to merge 15 commits into
codeforboston:mainfrom
J-C-L:finance-backfill

Conversation

@J-C-L

@J-C-L J-C-L commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

This is the bulk of the work for backfilling campaign finance data for legislators. (Issue: #2118)
Major changes:

  • Member matching and finance scraping are now scoped per court.
  • There are new backfill scripts for member matching (OCPF IDs with member codes) and finance scraping
  • There is a new script to script set a member mappings by hand, for cases not automatically matched, and the manual matches always win.
  • Script input is validated with zod and gives clear errors.

Other changes:

  • Renamed matchOcpfMembersV2 to matchOcpfMembers.
  • The finance scrape now warns on OCPF record types it doesn't classify.
  • Entries added to generalCourts (in constants.ts) for 189 (2015-16), 190 (2017-18), 191 (2019-20).

Checklist

  • [N/A] On the frontend, I've made my strings translate-able.
  • [N/A] If I've added shared components, I've added a storybook story.
  • [N/A] I've made pages responsive and look good on mobile.
  • [N/A] If I've added new Firestore queries, I've added any new required indexes to firestore.indexes.json (Please do not only create indexes through the Firebase Web UI, even though the error messages may reccommend it - indexes created this way may be obliterated by subsequent deploys)

Steps to test/reproduce

Locally (on emulator):

  1. Confirm the current-court path still works unchanged for member matching and finance scraping:
  1. Test backfillOcpfMemberMapping against a historical court:
  • Run: yarn firebase-admin run-script backfillOcpfMemberMapping --env local --courts=192
  • Confirm console output prints matched/unmatched/ambiguous with names.
  • Confirm in Firestore:
    /generalCourts/192/ocpfMemberMappingConfig/ exists with ocpfMemberMapping and ocpfMemberMappingFlags
  • Can run with verbose and see additional terminal logs output:
    yarn firebase-admin run-script backfillOcpfMemberMapping --env local --courts 192 --verbose
  • Test multi-court + bad input together. Confirm it fails with explanation of bad input
    Ex: yarn firebase-admin run-script backfillOcpfMemberMapping --env local --courts 193,192xxx
  • Test an obviously bogus court: Confirm output shows “Court 999 failed: Error: No members found for court 999. Nothing written."
    Ex: yarn firebase-admin run-script backfillOcpfMemberMapping --env local --courts 999
  1. Test new script for manual fixing.
  • First confirm that member AMS2 is listed in /generalCourts/192/ocpfMemberMappingConfig/ocpfMemberMappingFlags
  • Run yarn firebase-admin run-script setOcpfMemberMapping --env local --court 192 --memberCode AMS2 --cpfId 16980
  • Confirm member AMS2 is now in /generalCourts/192/ocpfMemberMappingConfig/ocpfMemberMapping with “manual: true”, and removed from/generalCourts/192/ocpfMemberMappingConfig/ocpfMemberMappingFlags
  • Re-run yarn firebase-admin run-script backfillOcpfMemberMapping --env local --courts=192
  • Confirm that member AMS2 is still listed in ocpfMemberMapping and is not in ocpfMemberMappingFlags
  1. Test backfillOcpfFinance against a historical court
  • Run yarn firebase-admin run-script backfillOcpfFinance --env local --courts=192
  • Confirm in Firestore /generalCourts/192/membersFinance docs exist
  • Confirm finance tab populates on website:
    http://localhost:3000/legislators/192/AMS2#finance
  • Test the fail-fast guard:
    Run yarn firebase-admin run-script backfillOcpfFinance --env local --courts=999
    Confirm output shows: Court 999 failed: Error: No general court entry for court 999. Nothing written.
    Confirm court 999 is not written into Firestore
  • Test multi-court:
    Run yarn firebase-admin run-script backfillOcpfFinance --env local --courts=192,193
    Confirm both courts run and the succeeded/failed summary lists both.
  • Test bad input:
    Run yarn firebase-admin run-script backfillOcpfFinance --env local --courts=192,193,abc
    Confirm that for non numeric, whole set fails with: --courts (item 3): Expected number, received nan
    Run yarn firebase-admin run-script backfillOcpfFinance --env local --courts=192,193,999
    Confirm that for one bad court, rest will run, output includes a summary of which succeeded and which failed
  1. Test that we see a warning for an unknown OCPF record type
    *Run yarn firebase-admin run-script backfillOcpfFinance --env local --courts=192
  • Confirm output includes: Warning logged: Unclassified OCPF record type — no amount counted; classify it in accumulateItem
    Note: This proves the defensive logging is working.
    Later, when we want to perfect the data, we need to look into these types and classify them in REPORT_TYPE_HANDLING/KNOWN_IGNORED_RECORD_TYPE_IDS.

J-C-L and others added 15 commits October 27, 2026 14:50
@J-C-L
moving ocpf member matching to be scoped per court
d97b68e
@J-C-L
renamed matchOcpfMembersV2 to matchOcpfMembers
bd78456
@J-C-L @claude
warn on unclassified OCPF record types in finance scrape
8d7e8e4 
Report types already logged a warning when unrecognized, defaulting to
a safe itemsOnly treatment. Record types fell through to a silent
default with no distinction between deliberately-ignored codes (205,
220) and a genuinely new one — this makes that failure mode loud too.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@J-C-L @claude
extract matchMembersForCourt for per-court OCPF matching
648362e 
Pulls the matching logic out of the matchOcpfMembers HTTP handler into
a reusable, court-parameterized function so the upcoming backfill
script can match historical courts too. Active-filers-only filtering
now applies only to the current court; historical courts drop it
since a legislator who served years ago has likely closed their OCPF
committee by now.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@J-C-L @claude
add backfillOcpfMemberMapping script for historical OCPF member matching
c7d08da 
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@J-C-L @claude
add backfillOcpfFinance script and court-parameterize finance scraping
69a42fb 
Extracts runOcpfFinanceScrape into scrapeFinanceForCourt(court, options),
deriving YEARS per-call from generalCourts instead of a module-level
constant tied to the current court. Adds generalCourts entries for
189-191 so historical courts can be targeted. The two existing
triggers are unaffected — both still just call
scrapeFinanceForCourt(currentGeneralCourt, ...).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@J-C-L
enable scripts to take a single court in addition to a list of courts
7042c3b
@J-C-L
using zod to parse input
6ce4260
@J-C-L
giving user nicer error message for bad script input
748a667
@J-C-L
further improvements to error message for bad script input
23dbecf
@J-C-L @claude
quiet member-mapping backfill logs by default, add --verbose, print J…
ae28ed9 
…SON summary

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@J-C-L
added comment for an edge case about name matching
6a010a7
@J-C-L @claude
add setOcpfMemberMapping script and manual flag so manual matches alw…
4d68540 
…ays win

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@J-C-L @claude
fail member matching for a court with no members instead of writing e…
a18aede 
…mpty mapping docs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@J-C-L @claude
print a one-line error instead of a stack trace for an unknown court …
6777f1b 
…in the finance backfill

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
maple-dev Ready Ready Preview Oct 27, 2026 9:40pm UTC

Request Review

This branch was successfully deployed

1 active deployment
Preview – maple-dev — 6777f1bc Deployed Oct 27, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sashamaryl sashamaryl Awaiting requested review from sashamaryl sashamaryl is a code owner
@mertbagt mertbagt Awaiting requested review from mertbagt mertbagt is a code owner
@nesanders nesanders Awaiting requested review from nesanders nesanders is a code owner
@mvictor55 mvictor55 Awaiting requested review from mvictor55 mvictor55 is a code owner
@timblais timblais Awaiting requested review from timblais timblais is a code owner
@alexjball alexjball Awaiting requested review from alexjball alexjball is a code owner
@Mephistic Mephistic Awaiting requested review from Mephistic Mephistic is a code owner
@kiminkim724 kiminkim724 Awaiting requested review from kiminkim724 kiminkim724 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@J-C-L