Skip to content

Enforce Cloud Armor metadata rate-limit rule #3762

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
anth-volk wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Enforce Cloud Armor metadata rate-limit rule #3762

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/armor-enforce-metadata.changed.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Enforce the Cloud Armor metadata rate-limit rule; keep the calculate rule in preview.
50 changes: 50 additions & 0 deletions docs/migration/armor/20260721T214911Z-pol-api-lb-metadata-enforced.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
creationTimestamp: '2026-07-21T07:16:39.177-07:00'
description: Rate limiting for api.policyengine.org (lb-api). See docs/migration/lb-cloud-armor-runbook.md
fingerprint: dT_5QmeI6h8=
id: '200433714284642728'
kind: compute#securityPolicy
labelFingerprint: 42WmSpB8rSM=
name: pol-api-lb
rules:
- action: throttle
description: ''
kind: compute#securityPolicyRule
match:
expr:
expression: request.path.matches('^/[a-z]{2}/metadata$')
preview: false
priority: 1000
rateLimitOptions:
conformAction: allow
enforceOnKey: IP
exceedAction: deny(429)
rateLimitThreshold:
count: 30
intervalSec: 60
- action: throttle
description: ''
kind: compute#securityPolicyRule
match:
expr:
expression: request.path.matches('^/[a-z]{2}/calculate(?:-full)?$')
preview: true
priority: 1100
rateLimitOptions:
conformAction: allow
enforceOnKey: IP
exceedAction: deny(429)
rateLimitThreshold:
count: 75
intervalSec: 60
- action: allow
description: default rule
kind: compute#securityPolicyRule
match:
config:
srcIpRanges:
- '*'
versionedExpr: SRC_IPS_V1
preview: false
priority: 2147483647
selfLink: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/securityPolicies/pol-api-lb
type: CLOUD_ARMOR
8 changes: 7 additions & 1 deletion docs/migration/lb-cloud-armor-runbook.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,13 @@ record of what is currently deployed is the newest exported snapshot in
|---|---|
| Policy | `pol-api-lb`, attached to both backend services of the public API LB |
| Rules | Per-IP throttles on the metadata and calculate path families; thresholds in the snapshot |
| Mode | Created in preview 2026-07-21; enforcement pending the gates below |
| Mode | metadata rule **ENFORCED** 2026-07-21 (worst legit observed 5/min vs 30 threshold — 6× headroom); calculate rule **stays in preview at 75/60s** — observed legit/partner clients run 39–88/min there, so enforcing would 429 real use (incl. the partner API fallback) |

Note on the preview gate: Cloud Armor **throttle** rules in preview only ever
log `CONFORM` (never `EXCEEDED`), so the `previewSecurityPolicy` outcome field
cannot be used to count would-be-throttled requests. Judge a throttle rule's
enforcement readiness from **raw per-IP request-rate analysis of LB logs**, not
from preview outcome counts.

Origin: overnight bot waves saturated backends / churned autoscaling
(2026-07-13 → 2026-07-21 incidents; details in the cutover execution plan).
Expand Down
Loading