Skip to content

fix(dynamo): emit each generation metric as a single series - #3988

Open
yupengtang wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
yupengtang:fix/dynamo-metrics-alias-dedup
Open

fix(dynamo): emit each generation metric as a single series#3988
yupengtang wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
yupengtang:fix/dynamo-metrics-alias-dedup

Conversation

@yupengtang

@yupengtang yupengtang commented Sep 4, 2026

Copy link
Copy Markdown

What does this PR do ?

Makes the managed Dynamo backend log each generation metric as a single series instead of plotting the alias next to the raw source it was built from.

DynamoMetricsSampler.snapshot resolves each entry of CANONICAL_LOGGER_ALIASES to the first source name present in the sample dict, then deletes only that one:

source = next((name for name in sources if name in metrics), None)
metrics[alias] = dict(metrics[source]) if source is not None else {}
if source is not None:
    del metrics[source]

A managed Dynamo worker exports the Dynamo gauge and the vLLM twin underneath it, so an alias usually has more than one source present. Only the winner was removed, so each losing twin stayed in the dict and got logged beside the alias it had just fed: inflight_batch_sizes next to vllm_num_requests_running, num_pending_samples next to vllm_num_requests_waiting, kv_cache_usage_perc next to vllm_kv_cache_usage_perc. logger.py then plots every surviving key twice, once as per_worker_ and once as average_.

The fix is to consume every source that is present and keep the first one as the value, so precedence is unchanged.

Two strays called out in LE-21 fall out of the same mechanism rather than needing new machinery:

  • vllm:num_requests_waiting is a curated prefix, so it also matches vllm:num_requests_waiting_by_reason. That metric exists in vLLM 0.23.0 (v1/metrics/loggers.py), which is what the Dynamo venv gets from ai-dynamo[vllm]==1.3.0.post1. Summed over its reason labels it is the same waiting count, so I listed it as a source of num_pending_samples and let it be consumed instead of surfacing on its own.
  • vllm_gpu_cache_usage_perc was unreachable as an alias source. No curated include prefix collects vllm:gpu_cache_usage_perc, and the existing test_metrics_parser_and_sampler_aliases already asserts "vllm_gpu_cache_usage_perc" not in parsed. It is the older vLLM spelling of what vllm_kv_cache_usage_perc already covers, so I dropped it instead of adding the prefix back. Happy to go the other way if you would rather keep a fallback for older vLLM reachable.

One thing I deliberately left alone: LE-21 also notes that kv_cache_usage_perc resolves to dynamo_component_gpu_cache_usage_percent here but to vllm:kv_cache_usage_perc on the vLLM backend, and that Dynamo's own docs disagree on the scale (prometheus_names.rs says 0.0-1.0, DASHBOARD_METRICS.md says 0-100). Normalizing that changes the meaning of a logged value across backends, so it seemed like your call rather than mine. Can follow up once you decide which scale is canonical.

Issues

Addresses LE-21 of #3459. Leaving that issue open since the other items are untouched.

Usage

No config or API change. Existing runs with policy.generation.backend: dynamo just stop emitting the duplicate series. For one scrape carrying both sources of all four aliases:

series emitted
before generation_tokens, inflight_batch_sizes, kv_cache_usage_perc, num_pending_samples, vllm_kv_cache_usage_perc, vllm_num_requests_running, vllm_num_requests_waiting, vllm_num_requests_waiting_by_reason
after generation_tokens, inflight_batch_sizes, kv_cache_usage_perc, num_pending_samples

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

Three tests added next to the existing one in tests/unit/models/generation/test_dynamo_generation.py:

  • test_snapshot_consumes_every_source_of_an_alias - Dynamo gauge and vLLM twin both scraped; the alias keeps the Dynamo value and neither source survives.
  • test_snapshot_folds_vllm_waiting_by_reason_into_the_pending_alias - the by_reason variant is absorbed instead of plotted.
  • test_curated_snapshot_leaves_no_alias_source_beside_its_alias - the general invariant, that no alias source name is left in a snapshot.

All three fail on main and pass with the change. Reverting just metrics.py to its pre-fix version:

PASSED test_metrics_parser_and_sampler_aliases
FAILED test_snapshot_consumes_every_source_of_an_alias
FAILED test_snapshot_folds_vllm_waiting_by_reason_into_the_pending_alias
FAILED test_curated_snapshot_leaves_no_alias_source_beside_its_alias
3 failed, 1 passed

The existing test_metrics_parser_and_sampler_aliases passes either way. It only ever has one source present per alias, which is why this went unnoticed.

With the change, the five Dynamo unit files pass together:

pytest tests/unit/models/generation/test_dynamo_{generation,arguments,http_client,managed_runtime,token_wrapper}.py
104 passed

That is CPU only. I don't have a GPU box, so the vLLM-backed modules in tests/unit/models/generation/ don't import for me and I could not run the functional suite; this change doesn't touch either path, but a CI run would be good to confirm.

ruff 0.9.9 check, import sort and format are clean on both files.

No docs change: docs/guides/dynamo-generation.md does not enumerate the metric series, so there is nothing there that goes stale. Say the word if you want the canonical four listed in the guide and I will add them here.

`DynamoMetricsSampler.snapshot` resolved each canonical alias to the first
source present and deleted only that one. A managed worker exports the
Dynamo gauge *and* the vLLM twin underneath it, so every losing twin stayed
in the dict and was plotted beside the alias it had just fed. A scrape
carrying both sources for all four aliases produced eight series where the
vLLM backend ships four. Consume every source that is present, keeping the
first as the value.

Two strays resolved with it:

- `vllm:num_requests_waiting` is a curated *prefix*, so it also matches
  vLLM 0.23.0's `vllm:num_requests_waiting_by_reason`. Summed over its
  labels that is the same waiting count, so it is now a source of
  `num_pending_samples` and gets consumed instead of surfacing on its own.
- `vllm_gpu_cache_usage_perc` was unreachable as an alias source: no
  curated include prefix collects `vllm:gpu_cache_usage_perc`, which the
  existing `test_metrics_parser_and_sampler_aliases` already asserts. It is
  the older vLLM spelling of the metric `vllm_kv_cache_usage_perc` already
  covers, so it is dropped rather than given its prefix back.

The existing alias test only ever had one source present per alias, which
is why the duplication went unnoticed. The new tests scrape both.

Signed-off-by: Yupeng Tang <85978465+yupengtang@users.noreply.github.com>
@yupengtang
yupengtang requested review from a team as code owners September 4, 2026 05:15
@copy-pr-bot

copy-pr-bot Bot commented Sep 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant