Skip to content

refactor(observability): one home for the task annotations telemetry reads - #407

Draft
yuechao-qin wants to merge 1 commit into
ycq/observability-01-run-attemptfrom
ycq/observability-02-annotation-readers-refactor
Draft

yuechao-qin wants to merge 1 commit into
ycq/observability-01-run-attemptfrom
ycq/observability-02-annotation-readers-refactor

Conversation

@yuechao-qin

@yuechao-qin yuechao-qin commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

What

Four annotation keys were read from three different modules, one of which imported a
Kubernetes launcher purely to borrow its constants. This puts every read in one place
and adds the lookup that turns whatever a task's provider annotation says into a
provider and a cluster.

No behaviour changes. Every value emitted today is still emitted, identically.

Why

The metric emitter is about to become a fourth reader, and it runs inside
before_commit:

  TODAY                                      WHAT A FOURTH READER WOULD ADD
  ─────────────────────────────────────      ──────────────────────────────────
  execution_tracing   declares cloud_provider    metrics.py declares it a 3rd time
  contextual_logging  declares it again          + re-parses the accelerators string
  execution_tracing   imports kubernetes_        + needs cluster, which only the
                      launchers for 3 strings      launcher config knows

That last line is the problem. The cluster a task ran on is not in its annotations —
the annotation holds a key or an alias, and only the deployment's launcher config says
which cluster each resolves to. Reading that config per status change would put a file
read inside a database commit.

So the mapping is registered once at startup and looked up in memory.

Changes

  instrumentation/execution_annotations.py   new, the only module that knows a key
  instrumentation/execution_tracing.py       -1 declaration, -1 launcher import
  instrumentation/contextual_logging.py      -1 declaration
CLOUD_PROVIDER_KEY declared here — no launcher owns it, routing sets it
the three resource keys aliased from kubernetes_launchers, not respelled, so a rename there cannot silently empty these readers
cloud_provider() the annotation as written
target() resolved to Target(provider, cluster) through the registered map
accelerators() {"X":"1"} and {"X": "1"} parse to one request
register_targets() called once at startup by the deployment

Target.cluster_from_annotation covers a provider that declares no clusters of its own
and lets each task name one. The deployment registers that annotation key, so no
deployment-specific key appears in this repository.

Every reader is pure — task_spec is a loaded column, never a relationship — so all of
them are safe inside before_commit, where a lazy load would put a SELECT in the
commit path.

The caller arrives separately: targeting/routing.py in Shopify's oasis-backend builds
the map from its launcher config at create_launcher(). Until something calls
register_targets, the map is empty and target() returns None, which is the same
answer it gives for an unrecognised value.

Tests

  • 16 cases over the three readers: absent, empty, malformed, two models, whitespace
  • alias and canonical key resolve to the same target
  • a provider naming its own cluster, and one that names none
  • the existing tracing and logging tests are unchanged and still pass

@yuechao-qin
yuechao-qin added this pull request to stack #408 October 26, 2026 18:51
@yuechao-qin
yuechao-qin force-pushed the ycq/observability-02-annotation-readers-refactor branch 2 times, most recently from f1622ba to 94bbbe8 Compare October 26, 2026 20:48
…reads

The cloud_provider key was declared twice in instrumentation/, and the three
resource keys were reached by importing a launcher purely to borrow its
constants. Collects every read into instrumentation/execution_annotations.py:
the resource keys are aliased from kubernetes_launchers rather than respelled,
and cloud_provider -- which no launcher owns, since routing sets it -- is
declared once.

Also adds Target and register_targets(), the lookup that turns whatever a
task's provider annotation says into a provider and a cluster. A provider that
declares no clusters of its own registers the annotation key its tasks use, so
no deployment-specific key appears in this repository.

Pure readers: task_spec is a loaded column, never a relationship, so these are
safe inside before_commit where a lazy load would put a SELECT in the commit
path. No behaviour change.
@yuechao-qin
yuechao-qin force-pushed the ycq/observability-02-annotation-readers-refactor branch from 94bbbe8 to 70e6786 Compare October 27, 2026 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant