[CAP-3944] Add container health to Kubernetes Pods dashboard

[AntoineEsteve]

What does this PR do?

Improves container-level health visibility in the Kubernetes Pods Overview dashboard:

• adds a Container Health group for waiting reasons, last termination reasons, accumulated restarts, and containers that are running but not ready
• replaces memory usage with memory working set and adds the CPU throttled-period ratio
• excludes succeeded pods from the Pods requiring attention list

Validated with ddev validate dashboards kubernetes and against live metric data, including large query timeframes.

Before	After
(none)

Motivation

Restore the container troubleshooting workflow requested in CAP-3944, following the investigation in CAP-3888, without reintroducing queries that fail to load in large organizations.

QA

You can try this dashboard here, and compare it with the previous version here.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[cit-pr-commenter-54b7da]

evalya-impact-summary

evalya impact analysis

Impact analysis: 0 selected, 0 skipped (of 0 test tasks)
Publish tasks:   2 (always emitted)
Diff (1 file):
  kubernetes/assets/dashboards/kubernetes_pods.json

Debug a specific task: evalya plan impact --path <path> --task <task>

Learn more about CI impact filtering

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-04T13:17:37.294779Z	e297f69	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[dd-octo-sts]

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
agent-reqs	Verify check versions match the Agent requirements file	✅
ci	Validate CI configuration and code coverage settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
config	Validate default configuration files against spec.yaml	✅
dep	Verify dependency pins are consistent and Agent-compatible	✅
http	Validate integrations use the HTTP wrapper correctly	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
labeler	Validate PR labeler config matches integration directories	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
license-headers	Validate Python files have proper license headers	✅
licenses	Validate third-party license attribution list	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
openmetrics	Validate OpenMetrics integrations disable the metric limit	✅
package	Validate Python package metadata and naming	✅
qa-label	Validate the pull request declares whether it needs QA for the next Agent release	✅
readmes	Validate README files have required sections	✅
saved-views	Validate saved view JSON file structure and fields	✅
version	Validate version consistency between package and changelog	✅

View full run

[L3n41c]

Code review

Found 4 issues:

1. Containers by Last Termination Reason sums a sticky current-state gauge with no .rollup() control (bug due to kubelet/datadog_checks/kubelet/kubelet.py _submit_container_state_metric, which does self.gauge(gauge_name, 1, tags + reason_tags) on every check run for as long as the container's lastState reason persists)

This is a population count, not an event count: a container that was OOMKilled once and has been healthy for days keeps contributing 1 until its pod is replaced. #23266 deliberately moved away from naive-summing this metric, replacing it with clamp_min(diff(kubernetes.containers.restarts), 0) * kubernetes.containers.last_state.terminated{reason:oomkilled}.rollup(max), and the sibling asset still carries .rollup(max) in kubernetes_clusters.json#L4539. Without explicit rollup control the bar heights vary with the dashboard time window, which is exactly the "large query timeframes" case described in the PR body.

integrations-core/kubernetes/assets/dashboards/kubernetes_pods.json

Lines 503 to 505 in e297f69

	"name": "query1",
	"query": "sum:kubernetes.containers.last_state.terminated{$scope,$cluster,$namespace,$deployment,$statefulset,$daemonset,$job,$cronjob} by {reason}",
	"semantic_mode": "combined"

2. Restarts Across Active Containers plots a cumulative counter raw, with no diff()/change() (bug due to kubelet/datadog_checks/kubelet/kubelet.py:622, self.gauge(self.NAMESPACE + '.containers.restarts', restart_count, tags))

Summed and plotted as-is, the line does not show restart activity, and it drops when a crash-looping pod is deleted and its replacement starts back at 0 — reading as "restarts subsided" when the opposite is true. This repository already wraps the same metric for that reason in monitor_pods_restarting.json#L28: change(max(last_5m),last_5m):sum:kubernetes.containers.restarts{*} by {kube_cluster_name,pod_name} > 5.

integrations-core/kubernetes/assets/dashboards/kubernetes_pods.json

Lines 535 to 537 in e297f69

	"name": "query1",
	"query": "sum:kubernetes.containers.restarts{$scope,$cluster,$namespace,$deployment,$statefulset,$daemonset,$job,$cronjob}",
	"semantic_mode": "combined"

3. CPU Throttled Periods title and alias name a count, but the widget plots a percentage (docs/developer/guidelines/dashboards.md says "Do not indicate the unit in the graph title because unit types are displayed automatically from metadata. An exception to this is if the calculation of the query represents a different type of unit.")

The formula is 100 * throttled / periods and the declared unit is percent, so this is that exception: the derived unit differs from the underlying kubernetes.cpu.cfs.* metrics. Both the title and the series alias currently suggest a raw period count, which misrepresents the y-axis magnitude. Something like "CPU throttling %" would match.

integrations-core/kubernetes/assets/dashboards/kubernetes_pods.json

Lines 949 to 956 in e297f69

	{
	"formula": "100 * throttled / periods",
	"alias": "Throttled Periods",
	"number_format": {
	"unit": {
	"type": "canonical_unit",
	"unit_name": "percent"
	}

4. Two of the four changes requested by CAP-3944 are not implemented, and the predicate moved the opposite way from what the ticket asks

The ticket's item 3 asks to "Add a pod infrastructure list ordered by restarts desc, with the dashboard template filters but without the current readiness/failure predicate that hides recovered pods", and item 4 asks to "Make the intended drill-down explicit". No infrastructure list was added and no note widget documents the drill-down; instead the existing predicate is tightened with AND NOT pod_phase:succeeded. That change is sound on its own merits — it matches the !pod_phase:succeeded convention established in #12440 and used by the sibling widgets — but it leaves the acceptance criterion "A paginated infrastructure list identifies pods with the highest restart counts and supports drill-down to the responsible container(s)" unmet, which is the gap CAP-3888 recorded as the original customer need. Worth confirming whether this is intentionally deferred to a follow-up.

integrations-core/kubernetes/assets/dashboards/kubernetes_pods.json

Lines 344 to 346 in e297f69

	"resource_type": "pod",
	"query_string": "$scope $cluster $namespace $deployment $statefulset $daemonset $job $cronjob ((field#metadata.creationTimestamp:>=1hr AND NOT kube_condition_ready:true AND NOT pod_phase:succeeded) OR pod_phase:failed)"
	},

Minor, non-blocking: the four new timeseries omit show_legend/legend_layout/legend_columns, while every other timeseries in this file sets them, so the backend default applies and adjacent x-axes may not align (dashboards.md). Layout, widget IDs, metric existence, template-variable coverage (all 8, including $cronjob) and the boolean logic of the infra-list predicate all check out.

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[sblumenthal]

A few comments, I am not sure that all of them require change, vs just a bit more context

[sblumenthal]

I am not sure I fully understand the intended use case of this graph... why would the total number of currently running pods that have terminated previously be relevent?

[sblumenthal]

What is the reason for this change?

[sblumenthal]

This should wait until this PR (and this one) are merged, as right now these metrics aren't supported in OTEL

Same deal with the working_set metric above

[sblumenthal]

I am still a bit confused on the overall choice of graphs and the intent behind this change

We can see an older version of the dashboard, before the big refactor, here

There was also this slightly more recent change which is not captured in that link, was removed with the refactor, and does not seem to be present in the current iteration presented here

It does not appear that the old queries were incorporated / influenced the graphs in this PR, so I am curious what the driving force is here