fix(dynamo): emit each generation metric as a single series - #3988
Open
yupengtang wants to merge 1 commit into
Open
fix(dynamo): emit each generation metric as a single series#3988yupengtang wants to merge 1 commit into
yupengtang wants to merge 1 commit into
Conversation
`DynamoMetricsSampler.snapshot` resolved each canonical alias to the first source present and deleted only that one. A managed worker exports the Dynamo gauge *and* the vLLM twin underneath it, so every losing twin stayed in the dict and was plotted beside the alias it had just fed. A scrape carrying both sources for all four aliases produced eight series where the vLLM backend ships four. Consume every source that is present, keeping the first as the value. Two strays resolved with it: - `vllm:num_requests_waiting` is a curated *prefix*, so it also matches vLLM 0.23.0's `vllm:num_requests_waiting_by_reason`. Summed over its labels that is the same waiting count, so it is now a source of `num_pending_samples` and gets consumed instead of surfacing on its own. - `vllm_gpu_cache_usage_perc` was unreachable as an alias source: no curated include prefix collects `vllm:gpu_cache_usage_perc`, which the existing `test_metrics_parser_and_sampler_aliases` already asserts. It is the older vLLM spelling of the metric `vllm_kv_cache_usage_perc` already covers, so it is dropped rather than given its prefix back. The existing alias test only ever had one source present per alias, which is why the duplication went unnoticed. The new tests scrape both. Signed-off-by: Yupeng Tang <85978465+yupengtang@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Makes the managed Dynamo backend log each generation metric as a single series instead of plotting the alias next to the raw source it was built from.
DynamoMetricsSampler.snapshotresolves each entry ofCANONICAL_LOGGER_ALIASESto the first source name present in the sample dict, then deletes only that one:A managed Dynamo worker exports the Dynamo gauge and the vLLM twin underneath it, so an alias usually has more than one source present. Only the winner was removed, so each losing twin stayed in the dict and got logged beside the alias it had just fed:
inflight_batch_sizesnext tovllm_num_requests_running,num_pending_samplesnext tovllm_num_requests_waiting,kv_cache_usage_percnext tovllm_kv_cache_usage_perc.logger.pythen plots every surviving key twice, once asper_worker_and once asaverage_.The fix is to consume every source that is present and keep the first one as the value, so precedence is unchanged.
Two strays called out in LE-21 fall out of the same mechanism rather than needing new machinery:
vllm:num_requests_waitingis a curated prefix, so it also matchesvllm:num_requests_waiting_by_reason. That metric exists in vLLM 0.23.0 (v1/metrics/loggers.py), which is what the Dynamo venv gets fromai-dynamo[vllm]==1.3.0.post1. Summed over itsreasonlabels it is the same waiting count, so I listed it as a source ofnum_pending_samplesand let it be consumed instead of surfacing on its own.vllm_gpu_cache_usage_percwas unreachable as an alias source. No curated include prefix collectsvllm:gpu_cache_usage_perc, and the existingtest_metrics_parser_and_sampler_aliasesalready asserts"vllm_gpu_cache_usage_perc" not in parsed. It is the older vLLM spelling of whatvllm_kv_cache_usage_percalready covers, so I dropped it instead of adding the prefix back. Happy to go the other way if you would rather keep a fallback for older vLLM reachable.One thing I deliberately left alone: LE-21 also notes that
kv_cache_usage_percresolves todynamo_component_gpu_cache_usage_percenthere but tovllm:kv_cache_usage_percon the vLLM backend, and that Dynamo's own docs disagree on the scale (prometheus_names.rssays 0.0-1.0,DASHBOARD_METRICS.mdsays 0-100). Normalizing that changes the meaning of a logged value across backends, so it seemed like your call rather than mine. Can follow up once you decide which scale is canonical.Issues
Addresses LE-21 of #3459. Leaving that issue open since the other items are untouched.
Usage
No config or API change. Existing runs with
policy.generation.backend: dynamojust stop emitting the duplicate series. For one scrape carrying both sources of all four aliases:generation_tokens,inflight_batch_sizes,kv_cache_usage_perc,num_pending_samples,vllm_kv_cache_usage_perc,vllm_num_requests_running,vllm_num_requests_waiting,vllm_num_requests_waiting_by_reasongeneration_tokens,inflight_batch_sizes,kv_cache_usage_perc,num_pending_samplesBefore your PR is "Ready for review"
Pre checks:
Additional Information
Three tests added next to the existing one in
tests/unit/models/generation/test_dynamo_generation.py:test_snapshot_consumes_every_source_of_an_alias- Dynamo gauge and vLLM twin both scraped; the alias keeps the Dynamo value and neither source survives.test_snapshot_folds_vllm_waiting_by_reason_into_the_pending_alias- theby_reasonvariant is absorbed instead of plotted.test_curated_snapshot_leaves_no_alias_source_beside_its_alias- the general invariant, that no alias source name is left in a snapshot.All three fail on
mainand pass with the change. Reverting justmetrics.pyto its pre-fix version:The existing
test_metrics_parser_and_sampler_aliasespasses either way. It only ever has one source present per alias, which is why this went unnoticed.With the change, the five Dynamo unit files pass together:
That is CPU only. I don't have a GPU box, so the vLLM-backed modules in
tests/unit/models/generation/don't import for me and I could not run the functional suite; this change doesn't touch either path, but a CI run would be good to confirm.ruff 0.9.9check, import sort and format are clean on both files.No docs change:
docs/guides/dynamo-generation.mddoes not enumerate the metric series, so there is nothing there that goes stale. Say the word if you want the canonical four listed in the guide and I will add them here.