Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
|
Claude finished @xinli-sw's task in 4m 3s —— View job Review of PR #3396
One 🔴 BLOCKING issue (inline on Other checks passed: |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # Sparse indexer logits are opt-in in the upstream Blackwell recipe. This | ||
| # shared script also serves H200, where these SM100 backends are unsupported. | ||
| # TP2 uses FlashInfer; TP4 uses MegaAttention. fp8 resolves backend KV layout. | ||
| ATTENTION_ARGS=() | ||
| if (( GPU_MEM_MIB >= 150000 )); then | ||
| if (( TP == 2 )); then | ||
| ATTENTION_CONFIG='{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | ||
| else | ||
| ATTENTION_CONFIG='{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | ||
| fi | ||
| ATTENTION_ARGS=(--attention-config "$ATTENTION_CONFIG" --kv-cache-dtype fp8) | ||
| fi |
There was a problem hiding this comment.
🔴 BLOCKING: This gate is not GB300-specific. dsv41flash_fp4_b200_vllm_mtp.sh and dsv41flash_fp4_gb200_vllm_mtp.sh are symlinks to this file, and both B200 (~183 GB) and GB200 pass GPU_MEM_MIB >= 150000, so dsv41flash-fp4-b200-vllm-agentic-dspark and dsv41flash-fp4-gb200-vllm-agentic-dspark also start receiving --attention-config {FLASHINFER_MLA_SPARSE_DSV41 | FLASHMLA_MEGA_ATTN_DSV41, indexer_kv_dtype=mxfp4, indexer_sparse_logits=true} --kv-cache-dtype fp8. The comment above only accounts for H200.
Why it matters: Those two arms stay pinned to vllm/vllm-openai:nightly-cd10ed6f (2026-09-15), while the PR describes these as newly merged Blackwell settings landing in the 2026-09-22 nightly. vLLM's AttentionConfig rejects unknown keys and resolves backend through the AttentionBackendEnum name, so an older image that lacks indexer_sparse_logits, the mxfp4 indexer dtype, or the *_DSV41 backend names fails at vllm serve argument parsing and the whole B200/GB200 grid goes red. Even if the old nightly did accept them, this silently changes the B200/GB200 recipe (fp8 KV cache, MXFP4 indexer KV, different attention backends) while perf-changelog.yaml only lists the GB300 config key, which violates the AGENTS.md rule that every perf-affecting recipe change needs a changelog entry for the affected configs. (I could not reach the upstream vLLM source from this session to confirm which nightly introduced the keys, so please verify against cd10ed6f if you intend to keep the memory gate.)
Fix: Opt in from the GB300 launcher instead of inferring from GPU memory, following the existing VLLM_SAFETENSORS_LOAD_STRATEGY pattern in this script. Add export DSV41_SPARSE_INDEXER_LOGITS=1 next to export VLLM_ENGINE_READY_TIMEOUT_S=7200 in runners/launch_gb300-nv.sh (line 130), and gate here on that variable:
| # Sparse indexer logits are opt-in in the upstream Blackwell recipe. This | |
| # shared script also serves H200, where these SM100 backends are unsupported. | |
| # TP2 uses FlashInfer; TP4 uses MegaAttention. fp8 resolves backend KV layout. | |
| ATTENTION_ARGS=() | |
| if (( GPU_MEM_MIB >= 150000 )); then | |
| if (( TP == 2 )); then | |
| ATTENTION_CONFIG='{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | |
| else | |
| ATTENTION_CONFIG='{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | |
| fi | |
| ATTENTION_ARGS=(--attention-config "$ATTENTION_CONFIG" --kv-cache-dtype fp8) | |
| fi | |
| # Sparse indexer logits are opt-in in the upstream Blackwell recipe. This | |
| # shared script also serves H200, B200 and GB200; H200 lacks these SM100 | |
| # backends and the B200/GB200 arms pin an older nightly, so the GB300 | |
| # launcher opts in explicitly via DSV41_SPARSE_INDEXER_LOGITS=1. | |
| # TP2 uses FlashInfer; TP4 uses MegaAttention. fp8 resolves backend KV layout. | |
| ATTENTION_ARGS=() | |
| if [[ "${DSV41_SPARSE_INDEXER_LOGITS:-0}" == "1" ]]; then | |
| if (( TP == 2 )); then | |
| ATTENTION_CONFIG='{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | |
| else | |
| ATTENTION_CONFIG='{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | |
| fi | |
| ATTENTION_ARGS=(--attention-config "$ATTENTION_CONFIG" --kv-cache-dtype fp8) | |
| fi |
Alternatively, if the intent is for B200 and GB200 to adopt the same settings, re-pin both to nightly-0961bbae in configs/nvidia-master.yaml and add their config keys and the recipe change to the perf-changelog.yaml entry.
| ATTENTION_ARGS=() | ||
| if (( GPU_MEM_MIB >= 150000 )); then | ||
| if (( TP == 2 )); then | ||
| ATTENTION_CONFIG='{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | ||
| else | ||
| ATTENTION_CONFIG='{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | ||
| fi | ||
| ATTENTION_ARGS=(--attention-config "$ATTENTION_CONFIG" --kv-cache-dtype fp8) |
There was a problem hiding this comment.
🔴 This script is symlinked to by dsv41flash_fp4_b200_vllm_mtp.sh and dsv41flash_fp4_gb200_vllm_mtp.sh too, so the new ATTENTION_ARGS block also fires for those arms, not just GB300. GPU_MEM_MIB>=150000 is true on B200 (180GB) and GB200 (256GB) as well as GB300, but only the gb300-vllm-agentic-dspark image was bumped; b200 and gb200 still pin vllm/vllm-openai:nightly-cd10ed6f9f6b37a8ace9cf380007e66fe12ec0c3, the same build documented as lacking these SM100 sparse-indexer backends on H200. Passing --attention-config with FLASHINFER_MLA_SPARSE_DSV41/FLASHMLA_MEGA_ATTN_DSV41 and --kv-cache-dtype fp8 to that older image will likely fail vllm serve startup, breaking the b200 and gb200 agentic-coding benchmark runs. …
Why this was flagged
…Fix: gate ATTENTION_ARGS on the image/nightly actually shipping this feature (e.g. per-config-key or image-tag check), not solely on GPU_MEM_MIB, and add matching perf-changelog entries if those arms' behavior does change too.
dsv41flash_fp4_b200_vllm_mtp.sh and dsv41flash_fp4_gb200_vllm_mtp.sh are symlinks to this file (benchmarks/single_node/agentic/dsv41flash_fp4_vllm_mtp.sh), confirmed via ls -la showing both -> dsv41flash_fp4_vllm_mtp.sh. Their configs (configs/nvidia-master.yaml lines ~8092 and ~8155, dsv41flash-fp4-gb200-vllm-agentic-dspark and dsv41flash-fp4-b200-vllm-agentic-dspark) still use image vllm/vllm-openai:nightly-cd10ed6f9f6b37a8ace9cf380007e66fe12ec0c3, unchanged by this diff. Both are Blackwell so GPU_MEM_MIB>=150000 at line 84 is true, so the new --attention-config/--kv-cache-dtype fp8 args at line 90 get added to their vllm serve invocation. The PR/changelog only pins and documents this feature for dsv41flash-fp4-gb300-vllm-agentic-dspark on the new nightly-0961bbae image; the older shared nightly is the same one the script's own comment says lacks SM100…
Verification: normal. The new block at benchmarks/single_node/agentic/dsv41flash_fp4_vllm_mtp.sh:83-91 is gated only on hardware memory (if (( GPU_MEM_MIB >= 150000 ))), and GPU_MEM_MIB comes from nvidia-smi --query-gpu=memory.total (line 59). The script's own comment at line 51 enumerates the Blackwell arms sharing this file — "B200 180 GB, GB200 256 GB, GB300 277 GB" — all far above 150000 MiB; the…
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=36248534298 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=36248534298 |
3494e65 to
b4fa176
Compare
|
InferenceX has switched away from unmaintainable bash scripts to YAML files that don't repeat the same stuff over and over again. Please merge the latest |
1456b41 to
5e51844
Compare
将 GB300 的批处理与 CUDA graph 档位对齐 B300，并保留 GB300 独有的 TP2 c1 设置。
5e51844 to
70cf20f
Compare
This branch has not been deployed
2 participants
Pin GB300 DeepSeek-V4.1-Flash vLLM to nightly
ddd6fbca148a867aad1fcab7ec72f582b9977db4on the native srt-slurm AgentX recipe. Use FlashInfer sparse attention at TP2 and TP4 with MXFP4 indexer KV, sparse logits, fp8 KV cache, and Engram CPU offload. Topology and concurrency remain unchanged. Shared TP2 and TP4 points now use B300's batching and CUDA graph tiers; GB300's extra TP2 c1 point retains its existing settings.中文
在原生 srt-slurm AgentX 配方中，将 GB300 DeepSeek-V4.1-Flash vLLM 固定到 nightly
ddd6fbca148a867aad1fcab7ec72f582b9977db4。TP2 和 TP4 均使用 FlashInfer 稀疏注意力，并启用 MXFP4 索引器 KV、稀疏 logits、fp8 KV 缓存及 Engram CPU 卸载。 拓扑与并发设置不变。共有的 TP2 和 TP4 测试点现采用 B300 的批处理与 CUDA graph 档位；GB300 独有的 TP2 c1 测试点保持原设置。