improving lobbying data pipeline reliability - #2
Closed
nesanders wants to merge 132 commits into
Closed
Conversation
When a logged-out user clicks the Follow button on the Testimony Detail page, the action silently fails because uid is undefined and Firestore rejects the operation with "Missing or insufficient permissions." This adds an auth guard that redirects to /login?redirect=currentPath, matching the existing auth redirect pattern used elsewhere in the app. Closes codeforboston#2059 Co-authored-with: Claude Code
Scrapes the MA Secretary of State lobbying portal (sec.state.ma.us/LobbyistPublicSearch)
and writes structured data to Firestore for joining with MAPLE bill data.
New collections:
- /lobbyingRegistrants — one doc per (registrant, year), regType Lobbyist|Employer
- /lobbyingFilings — one doc per (registrant, client, bill, court), with billId
null for Executive/Other chambers so the join guard is type-level
Key design points:
- billId is constructed as {chamberPrefix}{integer} (e.g. H1234, SD56) to match
Bill.id in the existing bills collection; raw integer + chamber stored separately
- Entity name normalization pipeline ported from reference implementation (10 steps:
d/b/a stripping, legal entity words, punctuation, THE, ampersand, typo fix, etc.)
- Both raw and *Norm name fields stored for provenance and grouping
- Live Cloud Function scrapes current+prior year on a 24h schedule with a
summaryDiscCache to avoid re-fetching summary pages in steady state
- Backfill admin script handles full 2005-present history with a Firestore
subcollection cursor (/scrapers/lobbyingBackfill/processedUrls) that scales
to ~50k URLs and is safely resumable
Files:
- functions/src/lobbying/{types,normalize,portal,scrapeLobbying,index}.ts
- scripts/firebase-admin/backfillLobbying.ts
- firestore.rules + firestore.indexes.json updated
- docs/lobbying-disclosure-ingestion.md: full plan, test plan, future work
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The MA SoS portal is protected by Imperva WAF, which uses TLS fingerprinting to classify HTTP clients before examining headers. Python's requests library produces a fingerprint that Imperva allows through; Node.js does not. A standalone Cloud Run container (Python 3.12) is therefore used for the scheduled ingestion instead of a Cloud Function. lobbying-scraper/ — Cloud Run container (3 pip deps: requests, beautifulsoup4, google-cloud-firestore): - scrape.py: entry point with --mode weekly (incremental, fast exit if nothing new) and --mode backfill (full 2005-present history, resumable subcollection cursor). Weekly mode caches summary URL→disc URL mappings so prior-year registrants with no new filings require zero additional HTTP requests. - portal.py: HTTP session management + HTML parsing for all three portal page levels (search POST, summary GET, disclosure GET). Handles both modern (>=2013) and legacy (<2013) disclosure formats. - normalize.py: port of functions/src/lobbying/normalize.ts — 10-step entity name normalization pipeline, must match the TypeScript version exactly. - writer.py: Firestore document construction and batch writes. Schema matches types.ts (lobbyingRegistrants, lobbyingFilings collections). scripts/firebase-admin/backfillLobbying.ts — simplified to spawn scrape.py as a subprocess; all HTTP and Firestore logic moved to Python. functions/src/lobbying/http/ — thin Python HTTP helper kept for reference; not used in the current architecture. Note: server-side IP reputation behavior with Imperva untested. Build and run the container on Cloud Run with --dry-run to validate before full deploy. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Per code review feedback: the TypeScript Firebase Function and backfill script added no value — the portal's TLS fingerprinting requirements mean Node.js cannot reach it, so the TS HTTP layer was non-functional and the backfill script was just a thin subprocess wrapper with no benefit over calling scrape.py directly. Removed: - functions/src/lobbying/scrapeLobbying.ts (broken Cloud Function) - functions/src/lobbying/portal.ts (non-functional TS HTTP layer) - functions/src/lobbying/http/ (unused Python fetch helper) - scripts/firebase-admin/backfillLobbying.ts (shell wrapper, no value) - scrapeLobbying export from functions/src/index.ts Kept: - functions/src/lobbying/types.ts — Firestore schema; imported by frontend - functions/src/lobbying/normalize.ts — normalization pipeline - lobbying-scraper/ — the working Cloud Run container (unchanged) The historical backfill is now run directly: python3 lobbying-scraper/scrape.py --mode backfill Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Added functions to read district snapshots from files and strip ignored HTML tags. - Updated `parseSecDistricts` to utilize the new function for cleaning HTML input. - Introduced tests for parsing SEC snapshots in `parseSecDistricts.test.ts` to ensure correct district counts for Senate and House. - Modified the scraping script to read HTML from local files if not fetching from the web.
- Implemented a feature flag to conditionally render the legislator profile page. - Added a redirect to a 404 page if the feature flag for legislators is disabled.
- Removed unused imports and refactored test functions in `parseSecDistricts.test.ts` to build HTML snapshots for Senate and House districts. - Updated the project configuration in `project.yml` to correct the formatting of the languages list.
Delete unused, vestigial asdf config file
…ucture
Portal parser (portal.py):
- Hybrid era (2014-2018): per-client compensation from Panel1 divs — was
silently $0 due to missing code path (e.g. Murphy Donoghue 2016: $990k)
- Legacy era (2009-2013): per-client totals from 'Compensation received'
column, with dedup of (client, amount) pairs before summing — was silently
$0 per client (e.g. ML Strategies 2011: $641k across 23 clients)
- Legacy bill semicolon separator: 'H73; Title' now parsed correctly
- 'Total amount' summary row excluded from compensation pairs
- HTTP retry on 429/500/502/503/504 (was aborting on first transient error)
- parse_summary() and parse_disclosure_detail() split out as pure functions
(no I/O) so the offline reparse driver can call them without a session
GCS archiving (archive.py):
- Write-only cold storage: every fetched Summary/CompleteDisclosure page
saved as gs://{project}-lobbying-archive/raw_html/{sha1(url)}.html
- Enabled by ARCHIVE_RAW=1 env var; no-op otherwise
- Failures are logged but never interrupt the live scrape path
Offline reparse driver (reparse_archive.py):
- Lists CompleteDisclosure blobs from GCS, resolves registrant meta from
Firestore via disclosureUrls array_contains, re-runs pure parsers,
writes back via writer.py; resumable via /scrapers/lobbyingReparse cursor
Pytest suite (tests/test_portal_parser.py, 26 tests):
- All 4 eras verified against committed gzipped fixture pages
- Compensation totals, client counts, bill counts, era detection asserted
- Specific bug-fix regressions: Total-amount artifact, H73 semicolon,
hybrid Panel1 compensation, 2007 _total_salary_ fallback
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- writer.py: move firestore import out of TYPE_CHECKING block so
firestore.ArrayUnion() is available at runtime (was NameError)
- writer.py/scrape.py/reparse_archive.py: strip leading slash from
Firestore path constants (SCRAPER_DOC, BACKFILL_DOC, REPARSE_DOC) —
db.document('/scrapers/x') raises ValueError: odd path element count
- scrape.py: add os import; pass GOOGLE_CLOUD_PROJECT to firestore.Client()
so local runs target the correct project rather than the ADC default
Verified: 3 registrants / 6 disclosures written to digital-testimony-dev;
re-run writes 0 (cursor working).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… 4 HTML eras - Replace TypeScript/ts-node test plan steps with Python equivalents - Document all 4 CompleteDisclosure HTML format eras (2005-2008, 2009-2013, 2014-2018, 2019+) including parser behavior and known quirks - Update Historical Backfill section with GOOGLE_CLOUD_PROJECT and ARCHIVE_RAW=1; remove local environment references - Add partial backfill pattern (--limit 50 across all years) as large-scale dev validation step - Remove stale Function Export section referencing deleted scrapeLobbying function - Update Step 8 (deploy) from Firebase Function commands to Cloud Run job commands - Fix cursor path prose to remove leading slashes; fix jsdom → beautifulsoup4 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Embed actual registrant and filing documents from dev Firestore to illustrate the data model concretely for reviewers - Correct registrantId description from "slugified" to SHA-256 hash (matches the actual implementation in writer.py) - filingId description likewise updated to SHA-256 hash Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* remove squish effect * Browse All Testimony button responsiveness tweaked * revert surface-gradiant * tweaks 3 * removed width: 100vw from peopleSaying class in css * revert old herosection text * cleanup
…rict-pages Smoss/scrape ma district pages
…oston#2176) pages/dev/token.tsx was missing getStaticProps, so next-i18next never loaded the common/auth/footer namespaces — Navbar and Footer rendered raw translation keys (e.g. navigation.bills, logInSignUp). Also adds MCP_SERVER_URL to the prod functions env so the mcpProxy Firebase Function can reach the Cloud Run MCP server. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…ader Legislator Page Header refactor
…n field Pre-2009 filings report compensation as a single entity total with no per-client breakdown. Previously this was stored as a fake client entry with clientName="_total_salary_", which leaked into client browse lists and required filtering at every consumer. Store it instead as legacyTotalCompensation on the registrant document, leaving the clients[] array empty for those filings. Update the TypeScript type and all tests accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds mock-based tests for write_registrant and write_filings covering: - legacyTotalCompensation is None for modern filings - legacyTotalCompensation is set correctly for pre-2009 filings - clients list structure and normalization - no-op guards (empty entity_name, None year) - write_filings bill count and empty-list short-circuit Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tors/add-k-g-k add k-g-k as a contributor for code, and design
…ston#2206) Closes codeforboston#2205 Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* remove shoopty whoop * minty fresh * cleanup
* maple profile * added profile social links * reinstate biography * reinstate TestimonyTab * corrected stroke-width to strokeWidth * cleanup * clarity * margin tweaks * cleanup * cleanup testimonyTab * more cleanup * tweaked disclaimer border * added public and limit to query
* Election scraping * Further election scraping * More robust elections * Prettier * Various election bugfixes * Tests and election id handling * Prettier
- Major copy edits to How Testimony Works, Legislative Process, Writing Effective Testimony, Why Use MAPLE page copy - Fix assorted typos including in the How MAPLE Uses AI and Mission & Goals copy
* added committee lifecycle reports to the exclusion list to prevent double-counting with bank reports * langauage clarification for handling un-itemized totals * Added election cycle start balance to firestore * removed non-contribution receipts from total received on finance tab
Election bugfix
…r-tab-deep-links Link legislator profile tabs to URL hashes
…tors/add-ericpastorm add ericpastorm as a contributor for code
* Combine votes modal * Create vote dropdown
…auth-redirect Fix Follow button silent failure for logged-out users
…tors/add-slarson add slarson as a contributor for code
… limit
The live weekly cursor (scrapers/lobbying) stored the entire processed-URL
history and per-registrant summary cache as two fields on a single Firestore
document. Firestore documents have a hard 1MB size limit; once run against
the full corpus, this document grew past it partway through, and every write
after that point failed silently — meaning every registrant processed for
the rest of that run (and any future run building on that cursor) was
silently skipped rather than captured.
Moved to subcollections — scrapers/lobbying/processedUrls/{hash} and
scrapers/lobbying/summaryCache/{hash}, one small doc per URL — mirroring the
pattern the backfill cursor (scrapers/lobbyingBackfill/processedUrls)
already used elsewhere in this same file. Point lookups replace the
in-memory set/dict that used to get loaded and saved as one blob on every
write.
Tested with a small in-memory Firestore fake (tests/test_scrape.py) real
enough to simulate write-then-read-back across calls, including a
regression test asserting the parent cursor doc never grows beyond a
handful of small fields. Confirmed this test fails against the pre-fix code
by checking it out and rerunning the suite, then passes again with the fix
restored. Also validated live against a project with existing data.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Owner
Author
|
Opening against codeforboston/maple directly instead. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The weekly lobbying scraper's incremental cursor (
scrapers/lobbying) stores which disclosure URLs it's already processed as two fields — a growing URL list and a per-registrant cache — on a single Firestore document. Firestore caps a document at 1MB. Run against the full corpus, that document grows past the limit partway through, and every write after that point fails silently: the scraper thinks it succeeded, but every registrant processed for the rest of the run (and beyond, since the cursor itself stops updating) never actually gets recorded as done. In practice this meant the weekly scraper silently stopped making progress once enough disclosures accumulated.Fix
Moved the cursor to subcollections — one small document per URL — mirroring the pattern the backfill cursor (
scrapers/lobbyingBackfill/processedUrls) already uses elsewhere in this same file. No document ever grows large enough to hit the limit, regardless of how much history accumulates.Testing
tests/test_scrape.py(new): unit tests against a small in-memory Firestore fake, including a regression test asserting the parent cursor document stays small. Confirmed this actually catches the bug by checking out the pre-fix code and rerunning the suite against it (fails), then restoring the fix (passes).pytest tests/) still green — 49 tests total.--mode weekly --dry-run).