Skip to content

feat(observability): label the platform, accelerator, cluster and namespace a task ran on - #409

Draft
yuechao-qin wants to merge 1 commit into
ycq/observability-02-annotation-readers-refactorfrom
ycq/observability-03-new-tier0-labels
Draft

yuechao-qin wants to merge 1 commit into
ycq/observability-02-annotation-readers-refactorfrom
ycq/observability-03-new-tier0-labels

Conversation

@yuechao-qin

@yuechao-qin yuechao-qin commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

What

The status-transition histogram carries only the two statuses, so every wait looks the
same. These five labels separate the ones that are ours from the ones that are not:

  • a GPU family is scarce right now, versus the orchestrator being slow for everyone
  • one tenant's cluster is not giving machines back, versus our queue being deep
  • one namespace is queueing, versus all of them

Why

An alert that cannot tell those apart either pages for someone else's capacity problem
or stays silent through ours.

Label on the histogram Values
today execution.status.from the status the execution left
today execution.status.to the status it reached
today execution.run_attempt first, repeat, none
new execution.cloud_provider google, nebius, davies, none
new execution.cluster us-ce1-lt3, b300-cluster, us-ce1-eo9, none
new execution.accelerator_type NVIDIA-B300-SXM6-PC, B300-SXM6-PC, none, unparseable
new execution.accelerator_count 0, 1, 3, 4, 8
new k8s.namespace.name about 42 distinct in a day, and the set grows

Changes

Every label is resolved, not copied, and nothing is ever omitted.

Label Where the value comes from
execution.cloud_provider the registered target map, so an alias is not a second platform
execution.cluster the same map; the annotation names a target, not a cluster
execution.accelerator_type the parsed annotation — 8 raw strings in production are 2 models
execution.accelerator_count the same parse; a malformed annotation degrades the labels, not the measurement
k8s.namespace.name the task's own service account, else the cluster's namespace

An absent or unroutable value is labelled "none" rather than left off. An attribute
that disappears forms a second series that no filter on the first one covers, which is
the failure these labels exist to prevent.

Target gains a namespace, so a task without its own service account still reports
the namespace it actually ran in — about half of all traffic.

Nothing queries. Every value is read off task_spec, a loaded column, because this runs
inside before_commit where a lazy load would put a SELECT in the commit path.

Tests

  • each label resolved from a registered map, and from an alias, to the same series
  • the accelerator annotation parsed from all 7 shapes production writes, including the
    two that differ by one space and the two that cannot be parsed at all
  • a namespace from a service account, from a cluster, and from neither
  • the exhaustive attribute assertion lists all eight, so a new label cannot be added
    without a test naming it

@yuechao-qin
yuechao-qin added this pull request to stack #408 October 26, 2026 21:18
@yuechao-qin yuechao-qin changed the title feat(observability): label the platform, accelerator and cluster a task ran on feat(observability): label the platform, accelerator, cluster and namespace a task ran on Oct 26, 2026
…sk ran on

The status-transition histogram carries only the two statuses, so a queue that is slow
because one GPU family is scarce is indistinguishable from the orchestrator being slow
for everyone. Adds the labels that separate them.

Each label is resolved rather than copied: the provider annotation holds whatever the
client wrote, so two spellings of one cluster are collapsed through the registered
target map instead of forming two series. Nothing is ever omitted -- an absent or
unroutable value is labelled "none", because an attribute that disappears forms a
second series that no filter on the first one covers.
@yuechao-qin
yuechao-qin force-pushed the ycq/observability-03-new-tier0-labels branch from 5910b82 to 7291f07 Compare October 27, 2026 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant