[AMD][MI355X] DSv4.1-Flash vLLM: match ATOM's fixed 16384 prefill chunk / DSv4.1-Flash vLLM:预填充块大小与 ATOM 一致固定为 16384 - #3451
Draft
Fangzhou-Ai wants to merge 2 commits into
Draft
Fangzhou-Ai wants to merge 2 commits into
Fangzhou-Ai wants to merge 2 commits into
Conversation
…nk / DSv4.1-Flash vLLM:预填充块大小与 ATOM 一致固定为 16384 Replace the TP=2/concurrency-dependent 4096/8192 --max-num-batched-tokens reduction with ATOM's unconditional 16384 (dsv41flash_fp4_mi355x_atom_mtp.sh), so both engines run the same prefill chunk size. Unpin the image to TBD: the prior pin predates this change and the new chunk size needs a repin once its KV headroom is validated at TP=2 c64+/c128+ and TP=4 c128 on this SKU, where the smaller chunk was previously carrying KV/prefix-cache pressure (run 35574132719). Draft PR pending that validation. 将 TP=2 及并发相关的 4096/8192 --max-num-batched-tokens 缩减，替换为与 ATOM (dsv41flash_fp4_mi355x_atom_mtp.sh) 一致的固定值 16384，使两个引擎使用相同的 预填充块大小。将镜像取消固定为 TBD：此前的固定镜像早于本次改动，新的块大小 需要在该 SKU 上验证 TP=2 c64+/c128+ 与 TP=4 c128 处的 KV 显存余量后再重新固定， 这些并发点此前依赖较小的块大小来缓解 KV/前缀缓存压力（run 35574132719）。 该 PR 为草稿，待验证完成。 Co-authored-by: Cursor Agent (Claude Sonnet 5) <cursoragent@cursor.com> Signed-off-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com>
Contributor
|
Thanks for the contribution!
中文感谢你的贡献！
|
…log.yaml 的 pr-link 更新为实际 PR #3451 Co-authored-by: Cursor Agent (Claude Sonnet 5) <cursoragent@cursor.com> Signed-off-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com>
Collaborator
|
InferenceX has switched away from unmaintainable bash scripts to YAML files that don't repeat the same stuff over and over again. Please merge the latest |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Match
dsv41flash-fp4-mi355x-vllm-agentic-dspark's prefill chunk size toATOM's fixed value for the same model/SKU (
dsv41flash_fp4_mi355x_atom_mtp.shuses
--max-num-batched-tokens 16384 --attn-prefill-chunk-size 16384unconditionally). Today this recipe reduces
--max-num-batched-tokensto4096/8192 at TP=2 c64+ and TP=4/TP=2 c128+ to free indexer/KV headroom (see
the comment this replaces, and run
35574132719). This PR removes thatreduction so both engines run the identical prefill chunk, and unpins
imagetoTBDsince the prior pin predates and is unrelated to thischange — a maintainer should pick a validated image when repinning.
This is a draft. The KV-headroom regression this previously guarded
against (TP=2 c64+/c128+, TP=4 c128) has not been re-validated at 16384
on this SKU. Before taking this out of draft, a full sweep needs to confirm
prefix-cache hit rate / TTFT / decode throughput don't regress at those
points the way they did before this recipe added the reduction.
Duplicate-check: searched open PRs/issues for
dsv41flashandprefillonthis repo; no open PR changes this recipe's prefill chunk size or image pin.
AI model disclosure
vLLM and ATOM for this model (see the companion vLLM draft PR), identified
that ATOM also runs a larger, unconditional prefill chunk, read both
engines' recipe scripts and this repo's
perf-changelog.yaml/CONTRIBUTING.mdconventions, and authored this diff end to end. Thesubmitting human reviewed the script/config changes and is responsible for
validating the flagged KV-headroom risk before merge.
Related Issue
Fixes #
Type of Change
Checklist
perf-changelog.yamland have not edited historical entriesOWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR中文
将
dsv41flash-fp4-mi355x-vllm-agentic-dspark的预填充块大小改为与 ATOM 在同一模型/SKU 上的固定值一致（
dsv41flash_fp4_mi355x_atom_mtp.sh在所有并发下都无条件使用
--max-num-batched-tokens 16384 --attn-prefill-chunk-size 16384）。目前该配方在 TP=2 c64+ 以及 TP=4/TP=2 c128+ 时会将--max-num-batched-tokens降至 4096/8192，以腾出 indexer/KV 显存余量（见本次替换的注释及 run
35574132719）。本 PR 移除该缩减逻辑，使两个引擎使用完全相同的预填充块大小，并将
image取消固定为TBD：此前的固定镜像与本次改动无关且早于本次改动，重新固定镜像需由维护者选择经过验证的镜像。
本 PR 为草稿。 此前用于规避 KV 显存不足的场景（TP=2 c64+/c128+、TP=4
c128）尚未在该 SKU 上以 16384 重新验证。在将其从草稿状态转正之前，需要一次
完整 sweep 来确认这些并发点的前缀缓存命中率、TTFT 与解码吞吐不会像此前那样
回退。
重复性检查：已在本仓库搜索涉及
dsv41flash与prefill的未关闭 PR/issue，未发现任何未关闭 PR 修改该配方的预填充块大小或镜像固定值。
AI 模型使用说明：Claude Sonnet 5（Cursor）负责调研 vLLM 与 ATOM 在该模型上
a8w4 与 a4w4 MoE 激活精度的差异（见配套的 vLLM 草稿 PR）、发现 ATOM 同时采用
更大且无条件的预填充块大小、阅读两个引擎的配方脚本以及本仓库
perf-changelog.yaml/CONTRIBUTING.md的约定，并端到端完成本次改动。提交人已审阅脚本/配置改动，并负责在合并前验证上述 KV 显存余量风险。