[yuechao-qin]

What

PENDING covers two unrelated failures with one number: a task waiting for capacity
that does not exist, and a task that is simply misconfigured. A four-hour wait looks
identical either way. This labels both instruments with why, so an alert can tell them
apart.

• no capacity, versus a broken image reference
• a cluster-wide shortage, versus one misconfigured task
• not ours to fix, versus ours

Why

A latency alert that cannot see the reason routes both cases the same way. The capacity
rule assumes a slow multi-GPU task means scarcity, so a task stuck on
CreateContainerConfigError pages the wrong people — and a genuine shortage lands in a
channel nobody is on call for.

	Today	With the reason
4h, no GPU in the cluster	slow transition	waiting_for_node
4h, image does not exist	slow transition	image_unavailable
4h, bad secret reference	slow transition	bad_config
Who is paged	the same rule for all three	only the ones that are ours

Changes

execution.pending_reason, on the transition histogram and on the slow counter.

A closed vocabulary, not the Kubernetes string

The raw reason is an open set a Kubernetes release can grow, so it is mapped rather than
labelled:

Kubernetes says	We store
no containerStatuses yet	waiting_for_node
ContainerCreating, PodInitializing	starting_container
ImagePullBackOff, ErrImagePull, ImageInspectError, RegistryUnavailable	image_unavailable
InvalidImageName, CreateContainerConfigError, CreateContainerError	bad_config
anything else	other
the task is not waiting	none

Six values, fixed. A Kubernetes release adding a reason widens other; it cannot add a
series.

The reason has to be captured before it is needed

By the time a task stops waiting, its pod is running and waiting.reason is gone. So it
is stamped on the way through, during the wait:

  t=0      PENDING, no containers yet          stamp "waiting_for_node"
  t=2m     poll, pod says ImagePullBackOff     stamp "image_unavailable"
  t=4h     RUNNING                             read the stamp, label the transition

• the stamp is written by a listener on ContainerExecution.launcher_data, which the
  orchestrator assigns on every sweep that sees a change — the only thing that updates
  while the status does not
• the serialised pod arrives as the event's own argument, so no Kubernetes call and no
  database read happen on the sweep path
• at the transition the stamp is read through a relationship already in the session; if
  it would have to be fetched, the point is labelled none rather than putting a
  SELECT inside the flush in progress

Cardinality

The histogram is the expensive half: it is a native histogram, so each new label value
multiplies buckets. The bound is ×6, and only transitions leaving PENDING can carry
a value other than none — the realised multiplier is lower, but it has not been
measured against production yet.

Tests

• each Kubernetes reason maps to its own bucket, parametrised over all six
• an unknown Kubernetes reason becomes other rather than its own label value
• a pod with no containers yet is waiting_for_node
• a running pod yields no reason at all
• a non-Kubernetes launcher yields no reason
• a poll stamps the reason onto the container execution
• the reason survives the pod starting to run — the case the change exists for
• a reason behind a lazy load is skipped rather than fetched
• what lands in the JSON column is a plain string, not the enum member
• the histogram and the counter each carry the reason on a real wait-then-run