P3 - container-runtime/copilot-runtime-bazel-cache-container-crashloopbackoff on catalog_service:copilot-runtime-bazel-cache,kube_cluster:dotcom-2-gc-cus-02,kube_deployment:copilot-runtime-bazel-cache,kube_namespace:copilot-runtime-b... - 2026-10-18T11:21Z #5184

Alert Details

Source: [P3] [Triggered] container-runtime/copilot-runtime-bazel-cache-container-crashloopbackoff on catalog_service:copilot-runtime-bazel-cache,kube_cluster:dotcom-2-gc-cus-02,kube_deployment:copilot-runtime-bazel-cache,kube_namespace:copilot-runtime-bazel-cache-production
Status: CRITICAL
Triggered at: 2026-10-18T11:21Z

This issue was created because the "create_github_issue_on_alert:true" tag was applied to the monitor. This issue was created by ISP.

%%%
At least one container in deployment copilot-runtime-bazel-cache (service copilot-runtime-bazel-cache, namespace copilot-runtime-bazel-cache-production) in cluster dotcom-2-gc-cus-02 has continuously reported CrashLoopBackOff for 10 minutes. The container keeps crashing and being restarted.

Check the container logs and recent deploys for a crashing process, a failing health check, or bad config.

If the container was recently OOMKilled, treat the OOMKilled finding as the root cause and investigate memory pressure first rather than opening a second remediation track for the resulting crash loop.

Dashboard: Kubernetes Workload Observability

Playbook: https://github.com/github/ops/blob/master/docs/playbooks/kubernetes/workload-alerts/crashloopbackoff.md

NOTE: Fleet-wide platform-default check (priority 2). When this fires, a GitHub issue is opened against your service's Service Catalog owning team.

HOW TO OPT OUT

1. Pick ONE of the following exclusions:

   • !catalog_service:<your-service-slug> — excludes all your workloads
   • !kube_deployment:<your-deployment> — excludes one deployment
   • !kube_namespace:<your-namespace> — excludes an entire namespace

2. Open a PR in github/datadog-monitoring adding your chosen exclusion inside every {...} filter block in this monitor's query. The CPU monitors (usage-vs-limits and throttling) have FOUR filter blocks — the exclusion must appear in every one of them or the filter is inconsistent. Memory and error monitors have one block. Repeat on all six monitor families for a full opt-out.

Cluster-level opt-outs are not accepted — this is a per-service safety net.

Questions? #compute-platform-support

@webhook-isp-alert

min(last_10m):default_zero(sum:kubernetes_state.container.status_report.count.waiting{!kube_namespace:enterprise-web-staging, !catalog_service:github,!catalog_service:token-scanning-service,!catalog_service:pyroscope,!catalog_service:N/A,!kube_deployment:N/A,kube_deployment:*,catalog_service:*,!kube_cluster_profile:test,!kube_cluster_profile:akstest,!kube_cluster_profile:actionspoc,!kube_cluster_profile:review-lab,!kube_cluster_profile:proxima-review-lab,!kube_cluster_stamp:test*,reason:crashloopbackoff} by {kube_namespace,kube_deployment,catalog_service,kube_cluster}) >= 1

The monitor was last triggered at Sun Oct 18 2026 11:21:16 UTC.

[Monitor Status] · [Edit Monitor] · [Related Logs]
%%%

Tags

alerting_stage:notifying,catalog_service:copilot-runtime-bazel-cache,create_github_issue_on_alert:true,fleet-wide,kube_cluster:dotcom-2-gc-cus-02,kube_deployment:copilot-runtime-bazel-cache,kube_namespace:copilot-runtime-bazel-cache-production,managed:true,moda_health_check:kubernetes.container_crash_loop_backoff,monitor,priority:3,priority:p3,reason:crashloopbackoff,routing_mode:fleetwide,team:compute-platform

On-Call

(none)