From af31361e5478f63ab0228c30fa51b618b407da10 Mon Sep 17 00:00:00 2001 From: Anthony Volk Date: Tue, 21 Jul 2026 17:16:01 +0300 Subject: [PATCH 1/6] Add Cloud Armor rate-limiting runbook for lb-api Two targeted per-IP throttle rules (metadata 30/60s, calculate 75/60s), preview-first rollout with empirical enforcement gates. No blanket or ban rules: the frontends poll /economy/ and /report/ at ~1 req/s, so only never-polled path families are rate-limited. Co-Authored-By: Claude Fable 5 --- changelog.d/lb-cloud-armor-runbook.added.md | 1 + docs/migration/lb-cloud-armor-runbook.md | 124 ++++++++++++++++++++ 2 files changed, 125 insertions(+) create mode 100644 changelog.d/lb-cloud-armor-runbook.added.md create mode 100644 docs/migration/lb-cloud-armor-runbook.md diff --git a/changelog.d/lb-cloud-armor-runbook.added.md b/changelog.d/lb-cloud-armor-runbook.added.md new file mode 100644 index 000000000..b2472ebec --- /dev/null +++ b/changelog.d/lb-cloud-armor-runbook.added.md @@ -0,0 +1 @@ +Add Cloud Armor rate-limiting runbook and policy snapshots for the public API load balancer. diff --git a/docs/migration/lb-cloud-armor-runbook.md b/docs/migration/lb-cloud-armor-runbook.md new file mode 100644 index 000000000..8d4ff9bc0 --- /dev/null +++ b/docs/migration/lb-cloud-armor-runbook.md @@ -0,0 +1,124 @@ +# Cloud Armor rate-limiting runbook (lb-api) + +Repeated-use runbook for the Cloud Armor security policy on the public API load +balancer. Created 2026-07-21 in response to overnight bot waves (2026-07-21 +00:00–06:00Z: facebookexternalhit re-crawl wave + three scraper networks, bursts +to 893 req/30min in the traffic trough) that churned Cloud Run scale-outs +(1→4→1→4, 18 instance boots) — each ~3-min import boot queues early-bind-routed +requests, producing monitor timeouts and user-facing 504s, and (the prior week) +CPU-saturating the single App Engine instance. + +## Shape + +- **Policy:** `pol-api-lb` (Cloud Armor Standard, pay-as-you-go), project + `policyengine-api`. +- **Attached to BOTH backend services** of URL map `lb-api`: `bs-app-engine` and + `bs-cloud-run` — one policy protects both platforms at the edge. +- **Rules: targeted throttles only. No blanket rules, no bans.** The frontends + poll `/economy/...` (society-wide calcs, 1 req/s per client while pending — + `SocietyWideCalcStrategy.getRefetchConfig`) and `/report/{id}` (~1 req/s + observed); NAT'd offices sum several users onto one client IP. Any rule whose + match includes a polled path family will 429 legitimate sessions — treat that + as a design error, not a tuning problem. + +| Priority | Action | Match (CEL) | Per-IP limit | Rationale | +|---|---|---|---|---| +| 1000 | throttle → 429 | `request.path.matches('^/[a-z]{2}/metadata$')` | 30 / 60s | 10–11 MB CPU+transfer-heavy scrape target; fetched ~once per app session, never polled | +| 1100 | throttle → 429 | `request.path.matches('^/[a-z]{2}/calculate(-full)?$')` | 75 / 60s | Single-shot in app-v2 (no polling); generous headroom for NAT + retry storms (~24/min worst observed legit); catches single-IP hammering | +| 2147483647 | allow (default) | `*` | — | | + +Deliberately NOT rate-limited: `/economy/`, `/report/`, `/simulation/`, +`/household/...` (polled or retry-prone app surfaces — a 429 there breaks a live +session) and scanner 404 paths (~150 ms each; no capacity impact). Per-IP +throttles barely touch facebookexternalhit (Meta spreads across many IPs in +2a03:2880::/29); the app-side malformed-payload 400 fix is the facebook-bot +mitigation. + +## Known caveat + +Cloud Armor only sees traffic that crosses the LB. The default +`*.run.app` / `*.appspot.com` URLs bypass it. Ingress deliberately stays open +through PR 4 (CI smoke tests need `*.run.app`); bots observed so far target +`api.policyengine.org`, so coverage is acceptable. Revisit at PR 17 (ingress +lockdown). + +## Create (executed 2026-07-21; preview mode) + +```sh +gcloud compute security-policies create pol-api-lb \ + --project=policyengine-api \ + --description="Rate limiting for api.policyengine.org (lb-api). See docs/migration/lb-cloud-armor-runbook.md" + +gcloud compute security-policies rules create 1000 \ + --project=policyengine-api --security-policy=pol-api-lb \ + --expression="request.path.matches('^/[a-z]{2}/metadata$')" \ + --action=throttle \ + --rate-limit-threshold-count=30 --rate-limit-threshold-interval-sec=60 \ + --conform-action=allow --exceed-action=deny-429 --enforce-on-key=IP \ + --preview + +gcloud compute security-policies rules create 1100 \ + --project=policyengine-api --security-policy=pol-api-lb \ + --expression="request.path.matches('^/[a-z]{2}/calculate(-full)?$')" \ + --action=throttle \ + --rate-limit-threshold-count=75 --rate-limit-threshold-interval-sec=60 \ + --conform-action=allow --exceed-action=deny-429 --enforce-on-key=IP \ + --preview + +gcloud compute backend-services update bs-cloud-run --security-policy=pol-api-lb --global --project=policyengine-api +gcloud compute backend-services update bs-app-engine --security-policy=pol-api-lb --global --project=policyengine-api +``` + +Post-attach verification: header-sampled curls (split unchanged, all 200), +BetterStack green, one live-suite run. Export a snapshot after every policy +change: + +```sh +gcloud compute security-policies export pol-api-lb --project=policyengine-api \ + --file-name=docs/migration/armor/$(date -u +%Y%m%dT%H%M%SZ)-pol-api-lb.yaml +``` + +## Preview → enforce gates + +Rules start in `--preview` (matched + logged, never enforced). Enforce a rule +only when BOTH hold: + +1. **Preview observation (≥24h, must span an overnight bot window):** LB log + entries carry `jsonPayload.previewSecurityPolicy` (`name`, `outcome`). + Group would-be THROTTLE/DENY hits by client IP network, user agent, and + path. Gate: **zero hits from monitor / app-referer / polling traffic; hits + present on known scraper networks.** +2. **Empirical threshold check:** from ≥7 days of LB logs, compute the per-IP + peak 60s request rate for each matched path family, split app-referer vs + other. Gate: **worst legitimate rate ≤ half the rule's threshold.** + +Enforce per rule with: + +```sh +gcloud compute security-policies rules update 1000 \ + --project=policyengine-api --security-policy=pol-api-lb --no-preview +``` + +Then export + commit a new snapshot, and verify on the next bot wave: 429s in +LB logs (`jsonPayload.enforcedSecurityPolicy.outcome="DENY"`), Cloud Run +`instance_count` stays low overnight, no BetterStack incidents. + +If a legitimate source appears in preview hits: raise that rule's threshold +(`rules update --rate-limit-threshold-count=`), keep preview, restart +the observation window. + +## Tuning / rollback + +- Loosen/tighten a rule: `rules update --rate-limit-threshold-count=` + (snapshot after). +- Disable enforcement fast: `rules update --preview` (back to + log-only; propagates in minutes). +- Full detach (the rollback): `gcloud compute backend-services update + bs-cloud-run --security-policy="" --global --project=policyengine-api` (and + same for `bs-app-engine`). The policy remains, unattached. + +## Cost + +Standard tier: $5/policy/mo + $1/rule/mo + $0.75/M requests (global policies). +At ~0.3–0.45M req/mo ≈ **$7.30/mo** for 1 policy + 2 rules. Verified against the +Billing Catalog API 2026-07-21. From 5e660f18dac69948dfc6ccf405da800e17fd0aac Mon Sep 17 00:00:00 2001 From: Anthony Volk Date: Tue, 21 Jul 2026 17:23:45 +0300 Subject: [PATCH 2/6] Add pol-api-lb policy snapshot (preview mode, attached 2026-07-21) Policy live on both backend services in preview. Runbook expression corrected: Cloud Armor regex rejects capture groups, (?:-full)? used. Co-Authored-By: Claude Fable 5 --- .../armor/20260721T142259Z-pol-api-lb.yaml | 50 +++++++++++++++++++ docs/migration/lb-cloud-armor-runbook.md | 2 +- 2 files changed, 51 insertions(+), 1 deletion(-) create mode 100644 docs/migration/armor/20260721T142259Z-pol-api-lb.yaml diff --git a/docs/migration/armor/20260721T142259Z-pol-api-lb.yaml b/docs/migration/armor/20260721T142259Z-pol-api-lb.yaml new file mode 100644 index 000000000..7e1194801 --- /dev/null +++ b/docs/migration/armor/20260721T142259Z-pol-api-lb.yaml @@ -0,0 +1,50 @@ +creationTimestamp: '2026-07-21T07:16:39.177-07:00' +description: Rate limiting for api.policyengine.org (lb-api). See docs/migration/lb-cloud-armor-runbook.md +fingerprint: b-PKxv9L5OA= +id: '200433714284642728' +kind: compute#securityPolicy +labelFingerprint: 42WmSpB8rSM= +name: pol-api-lb +rules: +- action: throttle + description: '' + kind: compute#securityPolicyRule + match: + expr: + expression: request.path.matches('^/[a-z]{2}/metadata$') + preview: true + priority: 1000 + rateLimitOptions: + conformAction: allow + enforceOnKey: IP + exceedAction: deny(429) + rateLimitThreshold: + count: 30 + intervalSec: 60 +- action: throttle + description: '' + kind: compute#securityPolicyRule + match: + expr: + expression: request.path.matches('^/[a-z]{2}/calculate(?:-full)?$') + preview: true + priority: 1100 + rateLimitOptions: + conformAction: allow + enforceOnKey: IP + exceedAction: deny(429) + rateLimitThreshold: + count: 75 + intervalSec: 60 +- action: allow + description: default rule + kind: compute#securityPolicyRule + match: + config: + srcIpRanges: + - '*' + versionedExpr: SRC_IPS_V1 + preview: false + priority: 2147483647 +selfLink: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/securityPolicies/pol-api-lb +type: CLOUD_ARMOR diff --git a/docs/migration/lb-cloud-armor-runbook.md b/docs/migration/lb-cloud-armor-runbook.md index 8d4ff9bc0..4792d0abd 100644 --- a/docs/migration/lb-cloud-armor-runbook.md +++ b/docs/migration/lb-cloud-armor-runbook.md @@ -24,7 +24,7 @@ CPU-saturating the single App Engine instance. | Priority | Action | Match (CEL) | Per-IP limit | Rationale | |---|---|---|---|---| | 1000 | throttle → 429 | `request.path.matches('^/[a-z]{2}/metadata$')` | 30 / 60s | 10–11 MB CPU+transfer-heavy scrape target; fetched ~once per app session, never polled | -| 1100 | throttle → 429 | `request.path.matches('^/[a-z]{2}/calculate(-full)?$')` | 75 / 60s | Single-shot in app-v2 (no polling); generous headroom for NAT + retry storms (~24/min worst observed legit); catches single-IP hammering | +| 1100 | throttle → 429 | `request.path.matches('^/[a-z]{2}/calculate(?:-full)?$')` | 75 / 60s | Single-shot in app-v2 (no polling); generous headroom for NAT + retry storms (~24/min worst observed legit); catches single-IP hammering. NOTE: Cloud Armor regex rejects capture groups — use non-capturing `(?:...)` | | 2147483647 | allow (default) | `*` | — | | Deliberately NOT rate-limited: `/economy/`, `/report/`, `/simulation/`, From 4ccb4fbdd193f7ce375fbb0f1dc409fd86ae2858 Mon Sep 17 00:00:00 2001 From: Anthony Volk Date: Tue, 21 Jul 2026 17:24:37 +0300 Subject: [PATCH 3/6] Fix remaining capture-group expression in runbook command block Co-Authored-By: Claude Fable 5 --- docs/migration/lb-cloud-armor-runbook.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/migration/lb-cloud-armor-runbook.md b/docs/migration/lb-cloud-armor-runbook.md index 4792d0abd..1c89e607f 100644 --- a/docs/migration/lb-cloud-armor-runbook.md +++ b/docs/migration/lb-cloud-armor-runbook.md @@ -59,7 +59,7 @@ gcloud compute security-policies rules create 1000 \ gcloud compute security-policies rules create 1100 \ --project=policyengine-api --security-policy=pol-api-lb \ - --expression="request.path.matches('^/[a-z]{2}/calculate(-full)?$')" \ + --expression="request.path.matches('^/[a-z]{2}/calculate(?:-full)?$')" \ --action=throttle \ --rate-limit-threshold-count=75 --rate-limit-threshold-interval-sec=60 \ --conform-action=allow --exceed-action=deny-429 --enforce-on-key=IP \ From ada68298b5f2833d42d13df7028229e204bddeb9 Mon Sep 17 00:00:00 2001 From: Anthony Volk Date: Tue, 21 Jul 2026 22:15:11 +0300 Subject: [PATCH 4/6] Generalize the Cloud Armor runbook to procedure over state Parameterized commands ($PROJECT/$POLICY/$BACKENDS placeholders); live rule values live in the exported snapshots under docs/migration/armor/, with only a compact current-deployment summary in the runbook. Co-Authored-By: Claude Fable 5 --- docs/migration/lb-cloud-armor-runbook.md | 195 ++++++++++++----------- 1 file changed, 104 insertions(+), 91 deletions(-) diff --git a/docs/migration/lb-cloud-armor-runbook.md b/docs/migration/lb-cloud-armor-runbook.md index 1c89e607f..35c412bea 100644 --- a/docs/migration/lb-cloud-armor-runbook.md +++ b/docs/migration/lb-cloud-armor-runbook.md @@ -1,124 +1,137 @@ -# Cloud Armor rate-limiting runbook (lb-api) - -Repeated-use runbook for the Cloud Armor security policy on the public API load -balancer. Created 2026-07-21 in response to overnight bot waves (2026-07-21 -00:00–06:00Z: facebookexternalhit re-crawl wave + three scraper networks, bursts -to 893 req/30min in the traffic trough) that churned Cloud Run scale-outs -(1→4→1→4, 18 instance boots) — each ~3-min import boot queues early-bind-routed -requests, producing monitor timeouts and user-facing 504s, and (the prior week) -CPU-saturating the single App Engine instance. - -## Shape - -- **Policy:** `pol-api-lb` (Cloud Armor Standard, pay-as-you-go), project - `policyengine-api`. -- **Attached to BOTH backend services** of URL map `lb-api`: `bs-app-engine` and - `bs-cloud-run` — one policy protects both platforms at the edge. -- **Rules: targeted throttles only. No blanket rules, no bans.** The frontends - poll `/economy/...` (society-wide calcs, 1 req/s per client while pending — - `SocietyWideCalcStrategy.getRefetchConfig`) and `/report/{id}` (~1 req/s - observed); NAT'd offices sum several users onto one client IP. Any rule whose - match includes a polled path family will 429 legitimate sessions — treat that - as a design error, not a tuning problem. - -| Priority | Action | Match (CEL) | Per-IP limit | Rationale | -|---|---|---|---|---| -| 1000 | throttle → 429 | `request.path.matches('^/[a-z]{2}/metadata$')` | 30 / 60s | 10–11 MB CPU+transfer-heavy scrape target; fetched ~once per app session, never polled | -| 1100 | throttle → 429 | `request.path.matches('^/[a-z]{2}/calculate(?:-full)?$')` | 75 / 60s | Single-shot in app-v2 (no polling); generous headroom for NAT + retry storms (~24/min worst observed legit); catches single-IP hammering. NOTE: Cloud Armor regex rejects capture groups — use non-capturing `(?:...)` | -| 2147483647 | allow (default) | `*` | — | | - -Deliberately NOT rate-limited: `/economy/`, `/report/`, `/simulation/`, -`/household/...` (polled or retry-prone app surfaces — a 429 there breaks a live -session) and scanner 404 paths (~150 ms each; no capacity impact). Per-IP -throttles barely touch facebookexternalhit (Meta spreads across many IPs in -2a03:2880::/29); the app-side malformed-payload 400 fix is the facebook-bot -mitigation. +# Cloud Armor rate-limiting runbook + +Repeated-use runbook for managing Cloud Armor security policies on a load +balancer's backend services: creating throttle rules safely (preview first), +deciding when to enforce them, tuning, and rolling back. The authoritative +record of what is currently deployed is the newest exported snapshot in +`docs/migration/armor/` — this document describes procedure, not state. + +## Current deployment (summary — see newest snapshot for truth) + +| | | +|---|---| +| Policy | `pol-api-lb`, attached to both backend services of the public API LB | +| Rules | Per-IP throttles on the metadata and calculate path families; thresholds in the snapshot | +| Mode | Created in preview 2026-07-21; enforcement pending the gates below | + +Origin: overnight bot waves saturated backends / churned autoscaling +(2026-07-13 → 2026-07-21 incidents; details in the cutover execution plan). + +## Design principles + +1. **Only throttle path families the frontends never poll.** Client apps poll + some endpoints at ~1 req/s per session (society-wide calculation status, + report status), and NAT'd offices aggregate many users onto one client IP. + A per-IP rule that matches a polled path will 429 legitimate sessions — + that is a design error, not a threshold-tuning problem. Before adding any + rule, check the frontend code and LB logs for polling behavior on the + matched paths. +2. **No blanket rules, no ban actions.** Throttling (429 per excess request) + self-heals the moment a client slows down; bans amplify any false positive. +3. **Preview first, always.** Every new or changed rule starts in preview + (matched and logged, never enforced) and graduates only through the gates + below. +4. **Rate limits address volume abuse, not expensive-request abuse.** A + handful of heavy requests that saturate capacity sits below any threshold + that spares real users — that is a capacity/scaling problem, not a Cloud + Armor problem. +5. Cloud Armor's CEL `.matches()` regex rejects capture groups — use + non-capturing `(?:...)`. ## Known caveat -Cloud Armor only sees traffic that crosses the LB. The default -`*.run.app` / `*.appspot.com` URLs bypass it. Ingress deliberately stays open -through PR 4 (CI smoke tests need `*.run.app`); bots observed so far target -`api.policyengine.org`, so coverage is acceptable. Revisit at PR 17 (ingress -lockdown). +Cloud Armor only sees traffic that crosses the load balancer. Serverless +default URLs (`*.run.app`, `*.appspot.com`) bypass it entirely. This is +acceptable while ingress must stay open for CI smoke tests; revisit when +ingress is locked down to the LB. -## Create (executed 2026-07-21; preview mode) +## Procedures + +Set once per shell: ```sh -gcloud compute security-policies create pol-api-lb \ - --project=policyengine-api \ - --description="Rate limiting for api.policyengine.org (lb-api). See docs/migration/lb-cloud-armor-runbook.md" +PROJECT= +POLICY= +BACKENDS=" [ ...]" +``` -gcloud compute security-policies rules create 1000 \ - --project=policyengine-api --security-policy=pol-api-lb \ - --expression="request.path.matches('^/[a-z]{2}/metadata$')" \ - --action=throttle \ - --rate-limit-threshold-count=30 --rate-limit-threshold-interval-sec=60 \ - --conform-action=allow --exceed-action=deny-429 --enforce-on-key=IP \ - --preview +### Create a policy and attach it + +```sh +gcloud compute security-policies create $POLICY --project=$PROJECT \ + --description=". See docs/migration/lb-cloud-armor-runbook.md" -gcloud compute security-policies rules create 1100 \ - --project=policyengine-api --security-policy=pol-api-lb \ - --expression="request.path.matches('^/[a-z]{2}/calculate(?:-full)?$')" \ +for BS in $BACKENDS; do + gcloud compute backend-services update $BS \ + --security-policy=$POLICY --global --project=$PROJECT +done +``` + +Immediately after attaching: verify serving is unaffected (sampled curls +against the public hostname, uptime monitor green, one live-suite run). + +### Add a per-IP throttle rule (in preview) + +```sh +gcloud compute security-policies rules create \ + --project=$PROJECT --security-policy=$POLICY \ + --expression="request.path.matches('')" \ --action=throttle \ - --rate-limit-threshold-count=75 --rate-limit-threshold-interval-sec=60 \ + --rate-limit-threshold-count= --rate-limit-threshold-interval-sec= \ --conform-action=allow --exceed-action=deny-429 --enforce-on-key=IP \ --preview - -gcloud compute backend-services update bs-cloud-run --security-policy=pol-api-lb --global --project=policyengine-api -gcloud compute backend-services update bs-app-engine --security-policy=pol-api-lb --global --project=policyengine-api ``` -Post-attach verification: header-sampled curls (split unchanged, all 200), -BetterStack green, one live-suite run. Export a snapshot after every policy -change: +### Snapshot after every policy change ```sh -gcloud compute security-policies export pol-api-lb --project=policyengine-api \ - --file-name=docs/migration/armor/$(date -u +%Y%m%dT%H%M%SZ)-pol-api-lb.yaml +gcloud compute security-policies export $POLICY --project=$PROJECT \ + --file-name=docs/migration/armor/$(date -u +%Y%m%dT%H%M%SZ)-$POLICY.yaml ``` -## Preview → enforce gates +Commit the snapshot (mirrors the `urlmap/` convention). -Rules start in `--preview` (matched + logged, never enforced). Enforce a rule -only when BOTH hold: +### Preview → enforce gates -1. **Preview observation (≥24h, must span an overnight bot window):** LB log - entries carry `jsonPayload.previewSecurityPolicy` (`name`, `outcome`). - Group would-be THROTTLE/DENY hits by client IP network, user agent, and - path. Gate: **zero hits from monitor / app-referer / polling traffic; hits - present on known scraper networks.** -2. **Empirical threshold check:** from ≥7 days of LB logs, compute the per-IP - peak 60s request rate for each matched path family, split app-referer vs - other. Gate: **worst legitimate rate ≤ half the rule's threshold.** +Enforce a rule only when BOTH hold: -Enforce per rule with: +1. **Preview observation (≥24h, spanning the traffic pattern the rule + targets — e.g. an overnight bot window):** LB log entries carry + `jsonPayload.previewSecurityPolicy` (`name`, `outcome`). Group would-be + throttle hits by client IP network, user agent, and path. Gate: **zero + hits from monitors, app-referred traffic, or polling clients; hits present + on the abusive sources the rule targets.** +2. **Empirical threshold check:** from ≥7 days of LB logs, compute the per-IP + peak request rate over the rule's interval for the matched path family, + split legitimate vs other. Gate: **worst legitimate rate ≤ half the + threshold.** ```sh -gcloud compute security-policies rules update 1000 \ - --project=policyengine-api --security-policy=pol-api-lb --no-preview +gcloud compute security-policies rules update \ + --project=$PROJECT --security-policy=$POLICY --no-preview ``` -Then export + commit a new snapshot, and verify on the next bot wave: 429s in -LB logs (`jsonPayload.enforcedSecurityPolicy.outcome="DENY"`), Cloud Run -`instance_count` stays low overnight, no BetterStack incidents. +Snapshot and commit; verify on the next abuse wave: 429s appear in LB logs +(`jsonPayload.enforcedSecurityPolicy.outcome="DENY"`), autoscaler churn and +monitor incidents stop. -If a legitimate source appears in preview hits: raise that rule's threshold -(`rules update --rate-limit-threshold-count=`), keep preview, restart -the observation window. +If a legitimate source shows up in preview hits: raise the threshold +(`rules update --rate-limit-threshold-count=`), stay in +preview, restart the observation window. -## Tuning / rollback +### Tuning / rollback -- Loosen/tighten a rule: `rules update --rate-limit-threshold-count=` +- Change a threshold: `rules update --rate-limit-threshold-count=` (snapshot after). - Disable enforcement fast: `rules update --preview` (back to log-only; propagates in minutes). -- Full detach (the rollback): `gcloud compute backend-services update - bs-cloud-run --security-policy="" --global --project=policyengine-api` (and - same for `bs-app-engine`). The policy remains, unattached. +- Full detach: `gcloud compute backend-services update + --security-policy="" --global --project=$PROJECT` per backend service. The + policy remains, unattached. ## Cost -Standard tier: $5/policy/mo + $1/rule/mo + $0.75/M requests (global policies). -At ~0.3–0.45M req/mo ≈ **$7.30/mo** for 1 policy + 2 rules. Verified against the -Billing Catalog API 2026-07-21. +Standard tier, pay-as-you-go: $5/policy/mo + $1/rule/mo + $0.75/M requests +evaluated (global policies; verified against the Billing Catalog API +2026-07-21). Order-of-magnitude ~$10/mo for one policy with a few rules at +sub-million monthly request volume. From 3f1159584b787a1479e95ccf790023db39fcd476 Mon Sep 17 00:00:00 2001 From: Anthony Volk Date: Tue, 21 Jul 2026 17:30:32 +0300 Subject: [PATCH 5/6] Return 400 for calculate payloads that fail situation parsing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Malformed household payloads (e.g. employment_income sent as a dict) are client errors — mostly facebookexternalhit bot traffic, ~300-700/day — but the catch-all in get_calculate turned them into HTTP 500s, polluting error-rate monitoring. Catch SituationParsingError explicitly and return 400 with the standard error shape; genuine internal failures still 500. Co-Authored-By: Claude Fable 5 --- .../calculate-malformed-payload-400.fixed.md | 1 + policyengine_api/endpoints/household.py | 24 ++++- .../test_calculate_error_statuses.py | 99 +++++++++++++++++++ 3 files changed, 122 insertions(+), 2 deletions(-) create mode 100644 changelog.d/calculate-malformed-payload-400.fixed.md create mode 100644 tests/unit/endpoints/test_calculate_error_statuses.py diff --git a/changelog.d/calculate-malformed-payload-400.fixed.md b/changelog.d/calculate-malformed-payload-400.fixed.md new file mode 100644 index 000000000..1a63c6479 --- /dev/null +++ b/changelog.d/calculate-malformed-payload-400.fixed.md @@ -0,0 +1 @@ +Return 400 instead of 500 when a calculate payload fails situation parsing. diff --git a/policyengine_api/endpoints/household.py b/policyengine_api/endpoints/household.py index 24e14661d..314871a00 100644 --- a/policyengine_api/endpoints/household.py +++ b/policyengine_api/endpoints/household.py @@ -10,6 +10,7 @@ format_unrecognized_inputs_message, ) from policyengine_api.utils.payload_validators import validate_country +from policyengine_core.errors import SituationParsingError def get_countries(): @@ -45,7 +46,11 @@ def add_yearly_variables(household, country_id, countries=None): if variables[variable]["isInputVariable"]: household[entity_plural][entity][ variables[variable]["name"] - ] = {household_year: variables[variable]["defaultValue"]} + ] = { + household_year: variables[variable][ + "defaultValue" + ] + } else: household[entity_plural][entity][ variables[variable]["name"] @@ -97,7 +102,9 @@ def get_household_year(household): @validate_country -def get_household_under_policy(country_id: str, household_id: str, policy_id: str): +def get_household_under_policy( + country_id: str, household_id: str, policy_id: str +): """Get a household's output data under a given policy. Args: @@ -274,6 +281,19 @@ def get_calculate(country_id: str, add_missing: bool = False) -> dict: try: result = country.calculate(household_json, policy_json) + except SituationParsingError as e: + # Malformed household payloads (e.g. a dict where a number belongs) + # are client errors, not server errors — mostly bot traffic. + response_body = dict( + status="error", + message=f"Invalid household payload: {e}", + result=None, + ) + return Response( + json.dumps(response_body), + status=400, + mimetype="application/json", + ) except Exception as e: logging.exception(e) response_body = dict( diff --git a/tests/unit/endpoints/test_calculate_error_statuses.py b/tests/unit/endpoints/test_calculate_error_statuses.py new file mode 100644 index 000000000..d37d72868 --- /dev/null +++ b/tests/unit/endpoints/test_calculate_error_statuses.py @@ -0,0 +1,99 @@ +from flask import Flask +import pytest +from policyengine_core.errors import SituationParsingError + +from policyengine_api.endpoints import household as household_endpoint + +from .test_calculate_deprecated_inputs import DummyCountry + +HOUSEHOLD = { + "people": { + "you": { + "age": {"2026": 40}, + "employment_income": {"2026": 30_000}, + } + } +} + + +class ParsingErrorCountry(DummyCountry): + def calculate(self, household, policy): + raise SituationParsingError( + ["people", "you", "employment_income", "2026"], + "Can't deal with value: expected type number, received '{}'.", + ) + + +class CrashingCountry(DummyCountry): + def calculate(self, household, policy): + raise RuntimeError("engine exploded") + + +def make_client(monkeypatch, country, add_missing=False): + monkeypatch.setattr( + household_endpoint, + "get_countries", + lambda: {"us": country}, + ) + app = Flask(__name__) + if add_missing: + monkeypatch.setattr( + household_endpoint, + "add_yearly_variables", + lambda household, country_id: household, + ) + + def handler(country_id): + return household_endpoint.get_calculate( + country_id, add_missing=True + ) + + app.add_url_rule( + "//calculate-full", + "calculate_full", + handler, + methods=["POST"], + ) + else: + app.add_url_rule( + "//calculate", + "calculate", + household_endpoint.get_calculate, + methods=["POST"], + ) + return app.test_client() + + +def test__calculate__returns_400_on_situation_parsing_error(monkeypatch): + client = make_client(monkeypatch, ParsingErrorCountry()) + + response = client.post("/us/calculate", json={"household": HOUSEHOLD}) + + assert response.status_code == 400 + payload = response.get_json() + assert payload["status"] == "error" + assert payload["result"] is None + assert payload["message"].startswith("Invalid household payload") + + +def test__calculate_full__returns_400_on_situation_parsing_error(monkeypatch): + client = make_client(monkeypatch, ParsingErrorCountry(), add_missing=True) + + response = client.post("/us/calculate-full", json={"household": HOUSEHOLD}) + + assert response.status_code == 400 + payload = response.get_json() + assert payload["status"] == "error" + assert payload["result"] is None + assert payload["message"].startswith("Invalid household payload") + + +def test__calculate__returns_500_on_unexpected_error(monkeypatch): + client = make_client(monkeypatch, CrashingCountry()) + + response = client.post("/us/calculate", json={"household": HOUSEHOLD}) + + assert response.status_code == 500 + payload = response.get_json() + assert payload["status"] == "error" + assert "engine exploded" in payload["message"] From 8c08b5f89398f6196dd4e1b90860b9b5316c4013 Mon Sep 17 00:00:00 2001 From: Anthony Volk Date: Tue, 21 Jul 2026 22:22:38 +0300 Subject: [PATCH 6/6] Apply ruff format Co-Authored-By: Claude Fable 5 --- policyengine_api/endpoints/household.py | 10 ++-------- tests/unit/endpoints/test_calculate_error_statuses.py | 4 +--- 2 files changed, 3 insertions(+), 11 deletions(-) diff --git a/policyengine_api/endpoints/household.py b/policyengine_api/endpoints/household.py index 314871a00..616da54e7 100644 --- a/policyengine_api/endpoints/household.py +++ b/policyengine_api/endpoints/household.py @@ -46,11 +46,7 @@ def add_yearly_variables(household, country_id, countries=None): if variables[variable]["isInputVariable"]: household[entity_plural][entity][ variables[variable]["name"] - ] = { - household_year: variables[variable][ - "defaultValue" - ] - } + ] = {household_year: variables[variable]["defaultValue"]} else: household[entity_plural][entity][ variables[variable]["name"] @@ -102,9 +98,7 @@ def get_household_year(household): @validate_country -def get_household_under_policy( - country_id: str, household_id: str, policy_id: str -): +def get_household_under_policy(country_id: str, household_id: str, policy_id: str): """Get a household's output data under a given policy. Args: diff --git a/tests/unit/endpoints/test_calculate_error_statuses.py b/tests/unit/endpoints/test_calculate_error_statuses.py index d37d72868..286bab6a3 100644 --- a/tests/unit/endpoints/test_calculate_error_statuses.py +++ b/tests/unit/endpoints/test_calculate_error_statuses.py @@ -44,9 +44,7 @@ def make_client(monkeypatch, country, add_missing=False): ) def handler(country_id): - return household_endpoint.get_calculate( - country_id, add_missing=True - ) + return household_endpoint.get_calculate(country_id, add_missing=True) app.add_url_rule( "//calculate-full",