Add Cloud Armor rate limiting for api.policyengine.org - #3756
Merged
Conversation
Two targeted per-IP throttle rules (metadata 30/60s, calculate 75/60s), preview-first rollout with empirical enforcement gates. No blanket or ban rules: the frontends poll /economy/ and /report/ at ~1 req/s, so only never-polled path families are rate-limited. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3756 +/- ##
==========================================
+ Coverage 79.92% 80.03% +0.11%
==========================================
Files 70 70
Lines 4338 4342 +4
Branches 808 809 +1
==========================================
+ Hits 3467 3475 +8
+ Misses 652 648 -4
Partials 219 219 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Policy live on both backend services in preview. Runbook expression corrected: Cloud Armor regex rejects capture groups, (?:-full)? used. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Parameterized commands ($PROJECT/$POLICY/$BACKENDS placeholders); live rule values live in the exported snapshots under docs/migration/armor/, with only a compact current-deployment summary in the runbook. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Malformed household payloads (e.g. employment_income sent as a dict) are client errors — mostly facebookexternalhit bot traffic, ~300-700/day — but the catch-all in get_calculate turned them into HTTP 500s, polluting error-rate monitoring. Catch SituationParsingError explicitly and return 400 with the standard error shape; genuine internal failures still 500. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3759
Fixes #3760
Summary
docs/migration/lb-cloud-armor-runbook.md: Cloud Armor Standard policypol-api-lbattached to both LB backend services (bs-app-engine,bs-cloud-run)/{cc}/metadataat 30/60s,/{cc}/calculate(-full)at 75/60s — created in preview mode; enforcement gated on ≥24h preview observation + 7-day empirical per-IP rate analysis/economy/and/report/at ~1 req/s per client, and NAT'd offices aggregate users onto single IPsdocs/migration/armor/as they change (mirrors theurlmap/convention)Also included: calculate malformed-payload 400 fix (#3760)
get_calculate(serving/calculateand/calculate-full) now catchesSituationParsingErrorbefore the generic catch-all and returns 400 with the standard error shape; unexpected exceptions still 500. These payloads are overwhelmingly bot traffic (facebookexternalhit, 300–700/day) and were the dominant noise source in 5xx monitoring. Tests: 400 on both routes + control that real errors still 500. Folded in from #3757 — this PR's merge carries it through the deploy chain.Context
Overnight 2026-07-21 bot waves (facebookexternalhit re-crawl + three scraper networks) churned Cloud Run scale-outs all night; each ~3-min import boot queued requests into monitor timeouts and 504s. This rate-limits the abuse surface at the edge for both backends. Cost ≈ $7.30/mo.
🤖 Generated with Claude Code