Repository navigation
feat(observability): label the platform, accelerator, cluster and namespace a task ran on - #409
Draft
yuechao-qin wants to merge 1 commit into
Conversation
…sk ran on The status-transition histogram carries only the two statuses, so a queue that is slow because one GPU family is scarce is indistinguishable from the orchestrator being slow for everyone. Adds the labels that separate them. Each label is resolved rather than copied: the provider annotation holds whatever the client wrote, so two spellings of one cluster are collapsed through the registered target map instead of forming two series. Nothing is ever omitted -- an absent or unroutable value is labelled "none", because an attribute that disappears forms a second series that no filter on the first one covers.
yuechao-qin
force-pushed
the
ycq/observability-03-new-tier0-labels
branch
from
October 27, 2026 16:33
5910b82 to
7291f07
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The status-transition histogram carries only the two statuses, so every wait looks the
same. These five labels separate the ones that are ours from the ones that are not:
Why
An alert that cannot tell those apart either pages for someone else's capacity problem
or stays silent through ours.
execution.status.fromexecution.status.toexecution.run_attemptexecution.cloud_providerexecution.clusterexecution.accelerator_typeexecution.accelerator_countk8s.namespace.nameChanges
Every label is resolved, not copied, and nothing is ever omitted.
execution.cloud_providerexecution.clusterexecution.accelerator_typeexecution.accelerator_countk8s.namespace.nameAn absent or unroutable value is labelled
"none"rather than left off. An attributethat disappears forms a second series that no filter on the first one covers, which is
the failure these labels exist to prevent.
Targetgains anamespace, so a task without its own service account still reportsthe namespace it actually ran in — about half of all traffic.
Nothing queries. Every value is read off
task_spec, a loaded column, because this runsinside
before_commitwhere a lazy load would put a SELECT in the commit path.Tests
two that differ by one space and the two that cannot be parsed at all
without a test naming it