From 4e5b9f7107f042055581a6e9f53de6423007afa1 Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Sat, 26 Sep 2026 02:22:22 -0400 Subject: [PATCH 1/6] [Klaud Cold] Remove everything that pointed at the deleted benchmarks/single_node/agentic reached it: - Drop the 17 search-space entries without srt-recipe: six configs that were entirely bash-based (both B300 Qwen3.5 power A/B keys, qwen3.8next H100, Kimi-K3 and MiniMax-M3 MI355X vLLM, DSv4.1 Flash MI355X ATOM) and the ATOM LMCache DRAM-offload arms of Kimi-K3, MiniMax-M3 and GLM-5.2 on MI355X. - Remove the dead dsv41flash bash branches in the GB200/GB300 launchers and the MiniMax-M3 ATOM script special case in the MI355X launcher. - Restore the DSv4.1 Flash H100/H200 block32 installers and kernel configs under benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/ and repoint the setup scripts, which the live single-node H100/H200 SGLang AgentX recipes run and #3461 broke. - Point docs and comments at the srt-slurm recipes and srt_agentic.sh. Co-Authored-By: Claude Opus 5.5 (1M context) --- .github/codeowner-signoff-verify-prompt.md | 2 +- KLAUD_DEBUG.md | 2 +- MODELS.md | 2 +- MODELS_zh.md | 2 +- .../multi_node/amd_utils/server_sglang.sh | 4 +- .../install_h100_block32_configs.py | 48 +++++++ .../install_h200_block32_configs.py | 43 ++++++ ...3,dtype=fp8_w8a8,block_shape=[32, 32].json | 114 ++++++++++++++++ ...3,dtype=fp8_w8a8,block_shape=[32, 32].json | 50 +++++++ ...3,dtype=fp8_w8a8,block_shape=[32, 32].json | 98 ++++++++++++++ ...3,dtype=fp8_w8a8,block_shape=[32, 32].json | 66 ++++++++++ ...3,dtype=fp8_w8a8,block_shape=[32, 32].json | 66 ++++++++++ ...3,dtype=fp8_w8a8,block_shape=[32, 32].json | 122 ++++++++++++++++++ .../h100_dsv41_block32/provenance.json | 71 ++++++++++ ...0,dtype=fp8_w8a8,block_shape=[32, 32].json | 58 +++++++++ ...0,dtype=fp8_w8a8,block_shape=[32, 32].json | 66 ++++++++++ ...0,dtype=fp8_w8a8,block_shape=[32, 32].json | 66 ++++++++++ ...0,dtype=fp8_w8a8,block_shape=[32, 32].json | 122 ++++++++++++++++++ .../h200_dsv41_block32/provenance.json | 16 +++ .../configs/dsv41flash-h100-block32.sh | 4 +- .../configs/dsv41flash-h200-block32.sh | 4 +- .../glm5.2/atom/mi355x-fp4-mtp/agentic.yaml | 4 +- .../kimik3/atom/mi355x-fp4-mtp/agentic.yaml | 5 +- .../kimik3/vllm/mi355x-fp4-mtp/agentic.yaml | 5 +- .../atom/mi355x-fp4-mtp/agentic.yaml | 5 +- configs/amd-master.yaml | 55 -------- configs/nvidia-master.yaml | 51 -------- docs/DOCUMENTATION_PLAN.md | 4 +- docs/DOCUMENTATION_PLAN_zh.md | 4 +- docs/index.md | 2 +- docs/index_zh.md | 2 +- docs/waiver/2508.md | 5 +- runners/launch_gb200-nv.sh | 35 ----- runners/launch_gb300-nv.sh | 33 ----- runners/launch_mi355x-amds.sh | 7 - 35 files changed, 1036 insertions(+), 207 deletions(-) create mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/install_h100_block32_configs.py create mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/install_h200_block32_configs.py create mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=1280,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json create mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=1792,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json create mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=512,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json create mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=5120,K=1024,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json create mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=5120,K=288,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json create mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=576,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json create mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/provenance.json create mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=1792,K=5120,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json create mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=5120,K=1024,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json create mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=5120,K=288,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json create mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=576,K=5120,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json create mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/provenance.json diff --git a/.github/codeowner-signoff-verify-prompt.md b/.github/codeowner-signoff-verify-prompt.md index e2780d4217..d29eaf8bd7 100644 --- a/.github/codeowner-signoff-verify-prompt.md +++ b/.github/codeowner-signoff-verify-prompt.md @@ -316,7 +316,7 @@ The rule covers the SERVING stack that produces the numbers. ## Check 11 — Agentic spec-decode configs use the golden simulated acceptance length APPLICABILITY: this check covers AGENTIC-workload benchmark changes that enable speculative decoding. From the PR diff, identify configs that are BOTH: -- agentic scripts under `benchmarks/single_node/agentic/**`, multi-node recipes +- single-node agentic srt-slurm recipes (`benchmarks/single_node/srt-slurm-recipes/**/agentic.yaml`), multi-node recipes under an `agentic/` directory (e.g. `benchmarks/multi_node/srt-slurm-recipes/**/agentic/**`), or master-config entries whose name/recipe path marks them agentic, AND - speculative-decoding with MTP / EAGLE / draft-model flags such as diff --git a/KLAUD_DEBUG.md b/KLAUD_DEBUG.md index 76be01cb97..0f0fb21043 100644 --- a/KLAUD_DEBUG.md +++ b/KLAUD_DEBUG.md @@ -77,7 +77,7 @@ changing the benchmark recipe. **Fix:** in `benchmarks/single_node/.sh`, either: 1. **Lower `--gpu-memory-utilization`** (`0.95 → 0.90`, sometimes 0.85). Matches the H100/H200/B200 NVIDIA pattern. Smallest blast radius. -2. **Disable the profiler entirely** for cases where lowering isn't enough: `export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` before `vllm serve`. Matches `benchmarks/single_node/agentic/deprecated/kimik2.5_fp4_b200.sh:65`. +2. **Disable the profiler entirely** for cases where lowering isn't enough: `export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` before `vllm serve`. Matches `benchmarks/single_node/fixed_seq_len/deprecated/kimik2.5_fp4_b200.sh`. Seen on: #1395 (kimik2.5-fp4-b200-vllm, needed env var), #1403 (gptoss-fp4-mi300x-vllm, needed 0.90), #1461 (dsv4-fp8-h200-vllm, needed 0.90). diff --git a/MODELS.md b/MODELS.md index c9fc1adc3b..ce8e6f83f0 100644 --- a/MODELS.md +++ b/MODELS.md @@ -64,7 +64,7 @@ Rationale: `dsv4` carries the largest single-turn footprint in the repository. 4 | Scenario | ISL/OSL | Status | |---|---|---| -| Agentic coding | Long Context, Multi Turn Realistic traffic trace replay with sub agents | Active. This is the trace-replay agentic-coding benchmark (see [`benchmarks/single_node/agentic/`](benchmarks/single_node/agentic/)). Going forward, new models will likely be onboarded with agentic coding only. Speculative decoding may be enabled or disabled to produce the best Pareto points; a separate non-spec-decode A/B baseline is not required (see [Deprecation Notice](#deprecation-notice)). | +| Agentic coding | Long Context, Multi Turn Realistic traffic trace replay with sub agents | Active. This is the trace-replay agentic-coding benchmark (see the srt-slurm recipes under [`benchmarks/single_node/srt-slurm-recipes/`](benchmarks/single_node/srt-slurm-recipes/) and the shared client [`benchmarks/srt_agentic.sh`](benchmarks/srt_agentic.sh)). Going forward, new models will likely be onboarded with agentic coding only. Speculative decoding may be enabled or disabled to produce the best Pareto points; a separate non-spec-decode A/B baseline is not required (see [Deprecation Notice](#deprecation-notice)). | | Single-turn 8k1k | 8192 / 1024 | Active. This is the primary fixed-sequence-length scenario. | | Single-turn 1k1k | 1024 / 1024 | Deprecated since 2026-07-17 ([#2263](https://github.com/SemiAnalysisAI/InferenceX/pull/2263)), to save GPU cluster time for higher-priority real-world agentic-coding benchmarks and new frontier models. Archived configs live in [`configs/deprecated/`](configs/deprecated/). The GLM-5.1 B200 TileRT point added later in [#2533](https://github.com/SemiAnalysisAI/InferenceX/pull/2533) remains active. | | Single-turn 1k8k | 1024 / 8192 | **Deprecated for all models** since 2026-03-27 ([#911](https://github.com/SemiAnalysisAI/InferenceX/pull/911)), to save GPU cluster time for higher-priority real-world agentic-coding benchmarks and new frontier models. Configs were removed, not archived. | diff --git a/MODELS_zh.md b/MODELS_zh.md index f26bc9b947..e2794deb5b 100644 --- a/MODELS_zh.md +++ b/MODELS_zh.md @@ -64,7 +64,7 @@ InferenceX-e2e 运行在数量固定且有限的 GPU 资源池上,并由一支 | 场景 | ISL/OSL | 状态 | |---|---|---| -| 智能体编码(agentic coding) | 长上下文、多轮真实流量的轨迹回放,含子智能体(sub agents) | 启用。此场景采用基于轨迹回放的智能体编码基准测试(见 [`benchmarks/single_node/agentic/`](benchmarks/single_node/agentic/))。今后新模型预计将仅以智能体编码场景接入。可开启或关闭投机解码以获得最优帕累托点;不要求独立的非投机解码 A/B 基线(见[弃用公告](#弃用公告))。 | +| 智能体编码(agentic coding) | 长上下文、多轮真实流量的轨迹回放,含子智能体(sub agents) | 启用。此场景采用基于轨迹回放的智能体编码基准测试(见 [`benchmarks/single_node/srt-slurm-recipes/`](benchmarks/single_node/srt-slurm-recipes/) 下的 srt-slurm 配方与共享客户端 [`benchmarks/srt_agentic.sh`](benchmarks/srt_agentic.sh))。今后新模型预计将仅以智能体编码场景接入。可开启或关闭投机解码以获得最优帕累托点;不要求独立的非投机解码 A/B 基线(见[弃用公告](#弃用公告))。 | | 单轮 8k1k | 8192 / 1024 | 启用。当前主要的固定序列长度(fixed-seq-len)场景。 | | 单轮 1k1k | 1024 / 1024 | 自 2026-07-17 起弃用([#2263](https://github.com/SemiAnalysisAI/InferenceX/pull/2263)),以便将 GPU 集群时间留给优先级更高的真实场景智能体编码基准测试与新的前沿模型。归档配置位于 [`configs/deprecated/`](configs/deprecated/)。后续由 [#2533](https://github.com/SemiAnalysisAI/InferenceX/pull/2533) 加入的 GLM-5.1 B200 TileRT 测试点仍启用。 | | 单轮 1k8k | 1024 / 8192 | **对所有模型均已弃用**,自 2026-03-27 起([#911](https://github.com/SemiAnalysisAI/InferenceX/pull/911)),以便将 GPU 集群时间留给优先级更高的真实场景智能体编码基准测试与新的前沿模型。相关配置已删除,未归档。 | diff --git a/benchmarks/multi_node/amd_utils/server_sglang.sh b/benchmarks/multi_node/amd_utils/server_sglang.sh index cae7c9d25f..ffb18695e9 100755 --- a/benchmarks/multi_node/amd_utils/server_sglang.sh +++ b/benchmarks/multi_node/amd_utils/server_sglang.sh @@ -580,8 +580,8 @@ elif [[ "$KV_OFFLOADING" != "none" && "$KV_OFFLOAD_BACKEND" == umbp-linker* ]]; # ========================================================================= # UMBP as a DIRECT external store for the unified radix tree (PD disagg). # - # Ported from benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang_mtp.sh, - # which is where this arm exists today. It is a SEPARATE sglang code path + # Ported from the single-node DSv4 MI355X SGLang AgentX bash script (since + # removed in #3460). It is a SEPARATE sglang code path # from the HiCache branch above, not a variation of it: the tree loads and # offloads pages against UMBP with NO host cache tier in between, and # sglang rejects the combination outright (server_args.py::_handle_hicache diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/install_h100_block32_configs.py b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/install_h100_block32_configs.py new file mode 100644 index 0000000000..88c3fc97f6 --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/install_h100_block32_configs.py @@ -0,0 +1,48 @@ +"""Install the H100 V4.1 tiling configs into the imported SGLang kernel package.""" # noqa: INP001 -- Standalone recipe executable. + +import hashlib +import json +import shutil +import sys +from pathlib import Path + +from sglang.kernels.ops.quantization import fp8_kernel + + +def main() -> None: + source = Path(sys.argv[1]) + artifacts = Path(sys.argv[2]) / "fp8_kernel_configs" + device = fp8_kernel.get_device_name().replace(" ", "_") + if device != "NVIDIA_H100_80GB_HBM3": + raise RuntimeError(f"H100 tiling configs cannot be installed on {device}") + destination = Path(fp8_kernel.__file__).resolve().parent / "configs" + destination.mkdir(exist_ok=True) + artifacts.mkdir(parents=True, exist_ok=True) + for n, k in [ + (5120, 1024), + (5120, 288), + (576, 5120), + (1792, 5120), + (512, 5120), + (1280, 5120), + ]: + name = ( + f"N={n},K={k},device_name={device},dtype=fp8_w8a8,block_shape=[32, 32].json" + ) + path = source / name + expected = { + int(m): config for m, config in json.loads(path.read_text()).items() + } + shutil.copyfile(path, destination / name) + shutil.copyfile(path, artifacts / name) + fp8_kernel.get_w8a8_block_fp8_configs.cache_clear() + actual = fp8_kernel.get_w8a8_block_fp8_configs(n, k, 32, 32) + if actual != expected: + raise RuntimeError(f"SGLang did not resolve the installed config: {name}") + print( + f"Installed {destination / name}: sha256={hashlib.sha256(path.read_bytes()).hexdigest()}" + ) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/install_h200_block32_configs.py b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/install_h200_block32_configs.py new file mode 100644 index 0000000000..3d34a865e2 --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/install_h200_block32_configs.py @@ -0,0 +1,43 @@ +"""Install the H200 V4.1 tiling configs into the imported SGLang kernel package.""" # noqa: INP001 -- Standalone recipe executable. + +import hashlib +import json +import shutil +import sys +from pathlib import Path + +from sglang.kernels.ops.quantization import fp8_kernel + + +def main() -> None: + source = Path(sys.argv[1]) + artifacts = Path(sys.argv[2]) / "fp8_kernel_configs" + tp = int(sys.argv[3]) + if tp not in (4, 8): + raise ValueError(f"H200 tiling configs support TP4 or TP8, got TP{tp}") + device = fp8_kernel.get_device_name().replace(" ", "_") + if device != "NVIDIA_H200": + raise RuntimeError(f"H200 tiling configs cannot be installed on {device}") + destination = Path(fp8_kernel.__file__).resolve().parent / "configs" + destination.mkdir(exist_ok=True) + artifacts.mkdir(parents=True, exist_ok=True) + shapes = [(1792, 5120)] + if tp == 8: + shapes += [(5120, 1024), (5120, 288), (576, 5120)] + for n, k in shapes: + name = f"N={n},K={k},device_name={device},dtype=fp8_w8a8,block_shape=[32, 32].json" + path = source / name + expected = {int(m): config for m, config in json.loads(path.read_text()).items()} + shutil.copyfile(path, destination / name) + shutil.copyfile(path, artifacts / name) + fp8_kernel.get_w8a8_block_fp8_configs.cache_clear() + actual = fp8_kernel.get_w8a8_block_fp8_configs(n, k, 32, 32) + if actual != expected: + raise RuntimeError(f"SGLang did not resolve the installed config: {name}") + print( + f"Installed {destination / name}: sha256={hashlib.sha256(path.read_bytes()).hexdigest()}" + ) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=1280,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=1280,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json new file mode 100644 index 0000000000..971110754f --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=1280,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json @@ -0,0 +1,114 @@ +{ + "0": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3 + }, + "1": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3, + "SPLIT_K": 16, + "SWAP_AB": false + }, + "5": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3, + "SPLIT_K": 16, + "SWAP_AB": false + }, + "10": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3, + "SPLIT_K": 16, + "SWAP_AB": true + }, + "20": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3, + "SPLIT_K": 8, + "SWAP_AB": true + }, + "40": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3, + "SPLIT_K": 8, + "SWAP_AB": true + }, + "80": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3, + "SPLIT_K": 16, + "SWAP_AB": true + }, + "160": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3, + "SPLIT_K": 8, + "SWAP_AB": true + }, + "190": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3, + "SPLIT_K": 8, + "SWAP_AB": true + }, + "191": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3 + }, + "320": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3 + }, + "321": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3 + } +} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=1792,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=1792,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json new file mode 100644 index 0000000000..986ff1953b --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=1792,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json @@ -0,0 +1,50 @@ +{ + "1": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 16, + "SWAP_AB": false, + "num_stages": 3, + "num_warps": 4 + }, + "6": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 16, + "SWAP_AB": true, + "num_stages": 3, + "num_warps": 4 + }, + "24": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "GROUP_SIZE_M": 32, + "SPLIT_K": 8, + "SWAP_AB": true, + "num_stages": 4, + "num_warps": 4 + }, + "96": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "GROUP_SIZE_M": 32, + "SPLIT_K": 8, + "SWAP_AB": true, + "num_stages": 4, + "num_warps": 4 + }, + "128": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "num_stages": 3, + "num_warps": 4 + } +} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=512,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=512,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json new file mode 100644 index 0000000000..3c4f7326b0 --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=512,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json @@ -0,0 +1,98 @@ +{ + "0": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3 + }, + "1": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3, + "SPLIT_K": 16, + "SWAP_AB": false + }, + "5": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3, + "SPLIT_K": 16, + "SWAP_AB": false + }, + "10": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3, + "SPLIT_K": 16, + "SWAP_AB": false + }, + "20": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3, + "SPLIT_K": 16, + "SWAP_AB": true + }, + "40": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3, + "SPLIT_K": 16, + "SWAP_AB": true + }, + "80": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3, + "SPLIT_K": 8, + "SWAP_AB": true + }, + "160": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3, + "SPLIT_K": 4, + "SWAP_AB": true + }, + "320": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3, + "SPLIT_K": 4, + "SWAP_AB": true + }, + "321": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3 + } +} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=5120,K=1024,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=5120,K=1024,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json new file mode 100644 index 0000000000..fece264886 --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=5120,K=1024,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json @@ -0,0 +1,66 @@ +{ + "1": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 4, + "num_stages": 5 + }, + "4": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 64, + "num_warps": 4, + "num_stages": 4 + }, + "16": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 1, + "num_warps": 4, + "num_stages": 4 + }, + "32": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 4, + "num_stages": 5 + }, + "64": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 1, + "num_warps": 4, + "num_stages": 5 + }, + "256": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 4, + "num_stages": 5 + }, + "1024": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 1, + "num_warps": 4, + "num_stages": 5 + }, + "4096": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 1, + "num_warps": 4, + "num_stages": 4 + } +} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=5120,K=288,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=5120,K=288,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json new file mode 100644 index 0000000000..ccf51c6e0a --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=5120,K=288,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json @@ -0,0 +1,66 @@ +{ + "1": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 4, + "num_stages": 2 + }, + "4": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 2 + }, + "16": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 8, + "num_stages": 2 + }, + "32": { + "BLOCK_SIZE_M": 32, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 64, + "num_warps": 4, + "num_stages": 5 + }, + "64": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3 + }, + "256": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 64, + "num_warps": 4, + "num_stages": 5 + }, + "1024": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 4, + "num_stages": 2 + }, + "4096": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 4, + "num_stages": 3 + } +} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=576,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=576,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json new file mode 100644 index 0000000000..4e45c66e6d --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=576,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json @@ -0,0 +1,122 @@ +{ + "1": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 16, + "SWAP_AB": false, + "num_stages": 3, + "num_warps": 4 + }, + "4": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 16, + "SWAP_AB": true, + "num_stages": 3, + "num_warps": 4 + }, + "6": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 16, + "SWAP_AB": true, + "num_stages": 3, + "num_warps": 4 + }, + "16": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 16, + "SWAP_AB": true, + "num_stages": 3, + "num_warps": 4 + }, + "24": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 16, + "SWAP_AB": true, + "num_stages": 3, + "num_warps": 4 + }, + "32": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 16, + "SWAP_AB": true, + "num_stages": 3, + "num_warps": 4 + }, + "64": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "GROUP_SIZE_M": 32, + "SPLIT_K": 8, + "SWAP_AB": true, + "num_stages": 4, + "num_warps": 4 + }, + "96": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "GROUP_SIZE_M": 32, + "SPLIT_K": 8, + "SWAP_AB": true, + "num_stages": 4, + "num_warps": 4 + }, + "128": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "GROUP_SIZE_M": 16, + "num_stages": 5, + "num_warps": 4 + }, + "193": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "GROUP_SIZE_M": 32, + "num_stages": 4, + "num_warps": 4 + }, + "256": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "GROUP_SIZE_M": 32, + "num_stages": 4, + "num_warps": 4 + }, + "1024": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "GROUP_SIZE_M": 1, + "num_stages": 4, + "num_warps": 4 + }, + "4096": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "GROUP_SIZE_M": 1, + "num_stages": 5, + "num_warps": 4 + } +} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/provenance.json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/provenance.json new file mode 100644 index 0000000000..9c4fff096c --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/provenance.json @@ -0,0 +1,71 @@ +{ + "sglang_commit": "0f6761b54facebb47f2068f87ecccd8f14da3a0e", + "image": "lmsysorg/sglang:nightly-dev-cu13-20260921-0f6761b5", + "hardware": "NVIDIA H100 80GB HBM3", + "tuner": "benchmark/kernels/quantization/tuning_block_wise_kernel.py", + "search_change": "Use BLOCK_SIZE_K=[32] for checkpoint block32 scales; upstream CUDA candidates [64,128] otherwise filter to an empty search.", + "tuning_slurm_job": 18452, + "selected_audit_slurm_job": 18457, + "resolver_install_slurm_job": 18458, + "validation": "39 cases: FP32 dequantized reference rtol=0.02/atol=0.05; exact CUDA graph/eager equality; alternating timing order. BF16 rounding may differ from default tiling.", + "selection": "N5120/K288 batch64 restored to upstream default after timing regression.", + "precision": "Stock FP8 inputs and scales, BF16 output; no quantization or model changes.", + "split_k_audit": { + "slurm_job": 18479, + "cases": 80, + "mechanism": "Pinned upstream Hopper SPLIT_K and SWAP_AB launch configuration, FP32 partial reduction, unchanged FP8 inputs/scales and BF16 output", + "shapes": [ + "N1792/K5120", + "N576/K5120" + ], + "boundary_validation": "Slurm18480:56/56 FP32-reference and exact CUDA-graph replay checks passed, batch boundariesM1..1024; changed cases all faster in alternating-order microtimings", + "large_batch_policy": "Retain original fallback selection for M129..8192 using explicit boundary anchors" + }, + "draft_projection_audit": { + "device": "NVIDIA H100 80GB HBM3", + "image": "nightly-dev-cu13-20260922-582389ce", + "search": { + "job": 18492, + "cases": 144, + "status": "COMPLETED0:0" + }, + "boundary": { + "job": 18493, + "cases": 206, + "status": "COMPLETED0:0" + }, + "trimmed_boundary": { + "job": 18494, + "cases": 20, + "status": "COMPLETED0:0" + }, + "reference": "FP32 dequant reference rtol.02/atol.05; exact eager/graph replay; no quantization or kernel code changes", + "ranges": { + "N512": "M1..320 tuned; M0 and M>=321 stock", + "N1280": "M1..190 tuned; M0 and M>=191 stock" + }, + "retained_speedup_range": [ + 1.1861693666195017, + 6.770688638312714 + ], + "baseline_l2_max": 0.0016977186314761639, + "candidate_l2_max": 0.0016970767173916101, + "hashes": { + "N=1280,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json": "635f684b6ed8b538709249baa5c474bf8583f952bcee330e01fd0f9e0ce714a3", + "N=512,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json": "193fc012550dce39bb4de0a4bde28eed5cee79f2900ed95841fe18ccb6d8c187" + }, + "serving_qualification": { + "candidate_run": 35683668032, + "baseline_run": 35683669703, + "protocol": "Matched canonical C1, 3600 seconds, normal warmup; zero profiled errors and valid power in both runs", + "candidate_total_tps_per_gpu": 1239.57314, + "baseline_total_tps_per_gpu": 958.51774, + "candidate_p90_interactivity": 90.89516, + "baseline_p90_interactivity": 60.93631, + "full_gsm8k_run": 35683671358, + "full_gsm8k_examples": 1319, + "full_gsm8k_strict_accuracy": 0.9696739954510993, + "final_sweep": "Required after integration" + } + } +} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=1792,K=5120,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=1792,K=5120,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json new file mode 100644 index 0000000000..83620a6b10 --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=1792,K=5120,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json @@ -0,0 +1,58 @@ +{ + "1": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "GROUP_SIZE_M": 32, + "SPLIT_K": 8, + "SWAP_AB": true, + "num_stages": 4, + "num_warps": 4 + }, + "2": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "num_stages": 3, + "num_warps": 4 + }, + "6": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 16, + "SWAP_AB": true, + "num_stages": 3, + "num_warps": 4 + }, + "24": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 8, + "SWAP_AB": true, + "num_stages": 3, + "num_warps": 4 + }, + "96": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 16, + "SWAP_AB": true, + "num_stages": 3, + "num_warps": 4 + }, + "128": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "num_stages": 3, + "num_warps": 4 + } +} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=5120,K=1024,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=5120,K=1024,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json new file mode 100644 index 0000000000..850f61947e --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=5120,K=1024,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json @@ -0,0 +1,66 @@ +{ + "1": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 64, + "num_warps": 4, + "num_stages": 3 + }, + "4": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 64, + "num_warps": 4, + "num_stages": 4 + }, + "16": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 4, + "num_stages": 5 + }, + "32": { + "BLOCK_SIZE_M": 32, + "BLOCK_SIZE_N": 64, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 4 + }, + "64": { + "BLOCK_SIZE_M": 32, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 4, + "num_stages": 3 + }, + "256": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 4, + "num_stages": 5 + }, + "1024": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3 + }, + "4096": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 1, + "num_warps": 4, + "num_stages": 4 + } +} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=5120,K=288,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=5120,K=288,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json new file mode 100644 index 0000000000..94b8a3fccf --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=5120,K=288,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json @@ -0,0 +1,66 @@ +{ + "1": { + "BLOCK_SIZE_M": 32, + "BLOCK_SIZE_N": 64, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 8, + "num_stages": 2 + }, + "4": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 1, + "num_warps": 4, + "num_stages": 4 + }, + "16": { + "BLOCK_SIZE_M": 32, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 64, + "num_warps": 4, + "num_stages": 2 + }, + "32": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 3 + }, + "64": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 4, + "num_stages": 4 + }, + "256": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 64, + "num_warps": 4, + "num_stages": 3 + }, + "1024": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 32, + "num_warps": 4, + "num_stages": 4 + }, + "4096": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 64, + "num_warps": 4, + "num_stages": 4 + } +} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=576,K=5120,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=576,K=5120,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json new file mode 100644 index 0000000000..4359ec7f93 --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=576,K=5120,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json @@ -0,0 +1,122 @@ +{ + "1": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 16, + "SWAP_AB": false, + "num_stages": 3, + "num_warps": 4 + }, + "4": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 16, + "SWAP_AB": false, + "num_stages": 3, + "num_warps": 4 + }, + "6": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 16, + "SWAP_AB": false, + "num_stages": 3, + "num_warps": 4 + }, + "16": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 16, + "SWAP_AB": true, + "num_stages": 3, + "num_warps": 4 + }, + "24": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 16, + "SWAP_AB": true, + "num_stages": 3, + "num_warps": 4 + }, + "32": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 16, + "SWAP_AB": true, + "num_stages": 3, + "num_warps": 4 + }, + "64": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 8, + "SWAP_AB": true, + "num_stages": 3, + "num_warps": 4 + }, + "96": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "GROUP_SIZE_M": 32, + "SPLIT_K": 8, + "SWAP_AB": true, + "num_stages": 3, + "num_warps": 4 + }, + "128": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "GROUP_SIZE_M": 16, + "num_stages": 5, + "num_warps": 4 + }, + "193": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "GROUP_SIZE_M": 64, + "num_stages": 3, + "num_warps": 4 + }, + "256": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "GROUP_SIZE_M": 64, + "num_stages": 3, + "num_warps": 4 + }, + "1024": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "GROUP_SIZE_M": 1, + "num_stages": 5, + "num_warps": 4 + }, + "4096": { + "BLOCK_SIZE_K": 32, + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "GROUP_SIZE_M": 1, + "num_stages": 5, + "num_warps": 4 + } +} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/provenance.json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/provenance.json new file mode 100644 index 0000000000..074329f645 --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/provenance.json @@ -0,0 +1,16 @@ +{ + "sglang_commit": "0f6761b54facebb47f2068f87ecccd8f14da3a0e", + "image": "lmsysorg/sglang:nightly-dev-cu13-20260921-0f6761b5", + "hardware": "NVIDIA H200", + "tuner": "benchmark/kernels/quantization/tuning_block_wise_kernel.py", + "search_change": "Use BLOCK_SIZE_K=[32] for checkpoint block32 scales; upstream CUDA candidates [64,128] otherwise filter to an empty search.", + "tuning_slurm_job": 88009, + "selected_audit_and_resolver_slurm_job": 88030, + "validation": "39 cases against FP32 dequantized reference at rtol=0.02/atol=0.05; exact CUDA graph/eager equality; alternating timing order. BF16 rounding may differ from default tiling.", + "selection": "N5120/K288 batch32 restored to upstream default after timing regression.", + "precision": "Stock FP8 inputs and scales, BF16 output; no quantization or model changes.", + "splitk_tuning_slurm_job": 88079, + "splitk_selected_boundary_validation_slurm_job": 88084, + "splitk_scope": "Supported upstream SPLIT_K/SWAP_AB launch settings for N1792K5120 and N576K5120 at small verification batches; unchanged prior config selections for all M129..8192.", + "splitk_validation": "72/72 exploratory cases and56/56 selected boundary cases passed FP32reference rtol=.02/atol=.05 plus exact eager/graph equality; alternating timing order, selected changed cases improved1.37\u20136.22x." +} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-h100-block32.sh b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-h100-block32.sh index 4332962a39..b5ae175fc1 100755 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-h100-block32.sh +++ b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-h100-block32.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash # Install the measured H100 DeepSeek-V4.1-Flash block-32 tilings into the worker's SGLang. set -euo pipefail -agentic=/infmax-workspace/benchmarks/single_node/agentic -python3 "$agentic/install_h100_block32_configs.py" "$agentic/kernel_configs/h100_dsv41_block32" /logs +assets=/infmax-workspace/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32 +python3 "$assets/install_h100_block32_configs.py" "$assets/kernel_configs/h100_dsv41_block32" /logs diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-h200-block32.sh b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-h200-block32.sh index 3d311f530c..6d80c7a180 100755 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-h200-block32.sh +++ b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-h200-block32.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash # Install the measured H200 DeepSeek-V4.1-Flash block-32 tilings into the worker's SGLang. set -euo pipefail -agentic=/infmax-workspace/benchmarks/single_node/agentic -python3 "$agentic/install_h200_block32_configs.py" "$agentic/kernel_configs/h200_dsv41_block32" /logs "$DSV41_BLOCK32_TP" +assets=/infmax-workspace/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32 +python3 "$assets/install_h200_block32_configs.py" "$assets/kernel_configs/h200_dsv41_block32" /logs "$DSV41_BLOCK32_TP" diff --git a/benchmarks/single_node/srt-slurm-recipes/glm5.2/atom/mi355x-fp4-mtp/agentic.yaml b/benchmarks/single_node/srt-slurm-recipes/glm5.2/atom/mi355x-fp4-mtp/agentic.yaml index e5672aa6f3..5c6f598100 100644 --- a/benchmarks/single_node/srt-slurm-recipes/glm5.2/atom/mi355x-fp4-mtp/agentic.yaml +++ b/benchmarks/single_node/srt-slurm-recipes/glm5.2/atom/mi355x-fp4-mtp/agentic.yaml @@ -1,6 +1,6 @@ # GLM-5.2 MXFP4 AgentX on MI355X with ATOM MTP, TP-only with GPU-resident KV. -# The DCP4 LMCache band stays on the legacy script: srtctl reserves ATOM's -# kv-transfer-config for disaggregated workers. +# The DCP4 LMCache band was removed with the legacy script (#3461): srtctl +# reserves ATOM's kv-transfer-config for disaggregated workers. base: schema: 2 name: glm5.2-fp4-mi355x-atom-agentic diff --git a/benchmarks/single_node/srt-slurm-recipes/kimik3/atom/mi355x-fp4-mtp/agentic.yaml b/benchmarks/single_node/srt-slurm-recipes/kimik3/atom/mi355x-fp4-mtp/agentic.yaml index ac5021e444..4646809ea3 100644 --- a/benchmarks/single_node/srt-slurm-recipes/kimik3/atom/mi355x-fp4-mtp/agentic.yaml +++ b/benchmarks/single_node/srt-slurm-recipes/kimik3/atom/mi355x-fp4-mtp/agentic.yaml @@ -1,7 +1,8 @@ # Kimi-K3 MXFP4 AgentX on MI355X with ATOM DSpark: the interactive band, TP8 # with GPU-resident KV and the deepest published draft (seven tokens). The -# 1.56 TB checkpoint only fits at TP8. The DCP8 LMCache bands stay on the legacy -# script: srtctl reserves ATOM's kv-transfer-config for disaggregated workers. +# 1.56 TB checkpoint only fits at TP8. The DCP8 LMCache bands were removed with +# the legacy script (#3461): srtctl reserves ATOM's kv-transfer-config for +# disaggregated workers. base: schema: 2 name: kimik3-fp4-mi355x-atom-agentic diff --git a/benchmarks/single_node/srt-slurm-recipes/kimik3/vllm/mi355x-fp4-mtp/agentic.yaml b/benchmarks/single_node/srt-slurm-recipes/kimik3/vllm/mi355x-fp4-mtp/agentic.yaml index 0621caa8e2..470956663a 100644 --- a/benchmarks/single_node/srt-slurm-recipes/kimik3/vllm/mi355x-fp4-mtp/agentic.yaml +++ b/benchmarks/single_node/srt-slurm-recipes/kimik3/vllm/mi355x-fp4-mtp/agentic.yaml @@ -1,8 +1,9 @@ # Kimi-K3 MXFP4 AgentX on MI355X with vLLM DSpark # (https://recipes.vllm.ai/moonshotai/Kimi-K3). TP8 only: the 1.56 TB checkpoint # is ~195 GB per GPU. The KV cache is GPU-resident through c4 and backed by -# vLLM's SimpleCPUOffloadConnector from c8. The DCP8 arm (c44-c70) runs without -# a draft model and stays on the legacy script. +# vLLM's SimpleCPUOffloadConnector from c8. The DCP8 arm (c44-c70) ran without a +# draft model on the legacy script and was removed with it (#3461). No master +# config uses this recipe yet. base: schema: 2 name: kimik3-fp4-mi355x-vllm-agentic diff --git a/benchmarks/single_node/srt-slurm-recipes/minimaxm3/atom/mi355x-fp4-mtp/agentic.yaml b/benchmarks/single_node/srt-slurm-recipes/minimaxm3/atom/mi355x-fp4-mtp/agentic.yaml index df6b8d5165..41ad42a575 100644 --- a/benchmarks/single_node/srt-slurm-recipes/minimaxm3/atom/mi355x-fp4-mtp/agentic.yaml +++ b/benchmarks/single_node/srt-slurm-recipes/minimaxm3/atom/mi355x-fp4-mtp/agentic.yaml @@ -1,6 +1,7 @@ # MiniMax-M3 MXFP4 AgentX on MI355X with ATOM EAGLE3 (GQA draft, three tokens) -# and GPU-resident KV at TP2 and TP4. The LMCache bands stay on the legacy -# script: srtctl reserves ATOM's kv-transfer-config for disaggregated workers. +# and GPU-resident KV at TP2 and TP4. The LMCache bands were removed with the +# legacy script (#3461): srtctl reserves ATOM's kv-transfer-config for +# disaggregated workers. base: schema: 2 name: minimaxm3-fp4-mi355x-atom-agentic diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 6eeba9f6d1..e08e74208f 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -652,22 +652,6 @@ dsr1-fp8-mi355x-sglang-disagg-mtp: ep: 1 dp-attn: false -kimik3-fp4-mi355x-vllm-agentic-mtp: - image: vllm/vllm-openai-rocm:nightly-rocm100-7f1a5398e9610d96c473931a26c0e12bbe0d0423 - model: moonshotai/Kimi-K3 - model-prefix: kimik3 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.60 - search-space: - - { tp: 8, ep: 1, dcp-size: 1, kv-offloading: none, conc-list: [1, 4] , spec-decoding: mtp } - - { tp: 8, ep: 1, dcp-size: 1, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [8, 10, 12, 14], spec-decoding: mtp } - - { tp: 8, ep: 1, dcp-size: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [44, 48, 70], spec-decoding: mtp } - # Kimi-K3 MXFP4 agentic-coding benchmark on MI355X via ATOM with DSpark # speculative decoding. Acceptance is pinned to the committed golden curve in # golden_al_distribution/kimik3_dspark_probabilistic_sample_method_block_rejection_sample_method.yaml @@ -702,10 +686,6 @@ kimik3-fp4-mi355x-atom-agentic-mtp: - dram-utilization: 0.343 search-space: - { tp: 8, kv-offloading: none, conc-list: [1, 4], spec-decoding: mtp, srt-recipe: benchmarks/single_node/srt-slurm-recipes/kimik3/atom/mi355x-fp4-mtp/agentic.yaml } - - { tp: 8, dcp-size: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.4.5" }, conc-list: [14, 16, 48], spec-decoding: mtp } - - dram-utilization: 0.513 - search-space: - - { tp: 8, dcp-size: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.4.5" }, conc-list: [56, 72], spec-decoding: mtp } minimaxm3-fp4-mi355x-atom-agentic-mtp: image: rocm/atom-dev:nightly_202609171455 @@ -723,8 +703,6 @@ minimaxm3-fp4-mi355x-atom-agentic-mtp: search-space: - { tp: 4, kv-offloading: none, conc-list: [1, 2, 4, 5, 8, 10, 12, 15, 20, 24, 28, 32], spec-decoding: mtp, srt-recipe: benchmarks/single_node/srt-slurm-recipes/minimaxm3/atom/mi355x-fp4-mtp/agentic.yaml } - { tp: 2, kv-offloading: none, conc-list: [1, 2], spec-decoding: mtp, srt-recipe: benchmarks/single_node/srt-slurm-recipes/minimaxm3/atom/mi355x-fp4-mtp/agentic.yaml } - - { tp: 2, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.5rc3+rocm7.2.4" }, conc-list: [20, 25, 30], spec-decoding: mtp } - - { tp: 4, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.5rc3+rocm7.2.4" }, conc-list: [40, 48], spec-decoding: mtp } dsr1-fp4-mi355x-sglang-disagg: image: lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809 @@ -1305,21 +1283,6 @@ dsv4-fp4-mi355x-atom-disagg-agentic-lmcache-dspark: - "DECODE_NODES=1" - "DECODE_MTP_SIZE=3" -minimaxm3-fp4-mi355x-vllm-agentic-mtp: - image: vllm/vllm-openai-rocm:nightly-2a02f6efe319c885e3ccbcecde402e0028f9ec1e - model: amd/MiniMax-M3-MXFP4 - model-prefix: minimaxm3 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.20 - search-space: - - { tp: 4, kv-offloading: none, conc-list: [1, 4, 5, 8, 10, 12, 15, 20, 24, 32], spec-decoding: mtp } - - { tp: 2, kv-offloading: none, conc-list: [1, 2], spec-decoding: mtp } - # GLM-5.2 FP4 agentic-coding benchmark on MI355X via SGLang with MTP speculative # decoding. Two arms: # (1) TP4/EP4 with HiCache KV offloading to DRAM at concurrency [4, 8, 10, 12, 14, 16]; @@ -1392,11 +1355,9 @@ glm5.2-fp4-mi355x-atom-agentic-mtp: agentic-coding: # dram-utilization 0.171 -> TOTAL_CPU_DRAM_GB=256 per TP4 rank (recipe's # LMCACHE_MAX_LOCAL_CPU_SIZE=256): min(3095781,2861022)MiB*0.171*4/8 = 256. - # Only the DCP4 arm offloads; the other two are GPU-resident. - dram-utilization: 0.171 search-space: - { tp: 4, kv-offloading: none, conc-list: [2, 4, 8, 10], spec-decoding: mtp, srt-recipe: benchmarks/single_node/srt-slurm-recipes/glm5.2/atom/mi355x-fp4-mtp/agentic.yaml } - - { tp: 4, dcp-size: 4, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.4.5" }, conc-list: [16, 24, 32, 40, 48], spec-decoding: mtp } - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4], spec-decoding: mtp, srt-recipe: benchmarks/single_node/srt-slurm-recipes/glm5.2/atom/mi355x-fp4-mtp/agentic.yaml } dsv4-fp4-mi355x-sglang-agentic-mtp: @@ -1445,22 +1406,6 @@ dsv41flash-fp4-mi355x-vllm-agentic-dspark: - { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64, 128], srt-recipe: benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml } - { tp: 2, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64, 128], srt-recipe: benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml } -# DeepSeek-V4.1-Flash on ATOM, following the upstream TP2 / TP4 AgentX recipe. -# Five-token DSpark uses golden AL 3.51 for throughput and real acceptance for eval. -dsv41flash-fp4-mi355x-atom-agentic-dspark: - image: rocm/atom-dev:nightly_202609250902 - model: deepseek-ai/DeepSeek-V4.1-Flash - model-prefix: dsv41flash - runner: cluster:mi355x-amds - precision: fp4 - framework: atom - multinode: false - scenarios: - agentic-coding: - - search-space: - - { tp: 2, ep: 1, dp-attn: false, kv-offloading: none, spec-decoding: draft_model, conc-list: [1, 2, 8, 16, 32, 64] } - - { tp: 4, ep: 1, dp-attn: false, kv-offloading: none, spec-decoding: draft_model, conc-list: [2, 8, 16, 32, 64] } - # Speculative decoding on an agentic scenario must run with simulated # synthetic acceptance at the committed golden AL for this model, thinking mode # and draft length (docs/PR_REVIEW_CHECKLIST.md), and a submission may not diff --git a/configs/nvidia-master.yaml b/configs/nvidia-master.yaml index 30ce89afcd..6d0ec1c375 100644 --- a/configs/nvidia-master.yaml +++ b/configs/nvidia-master.yaml @@ -5090,38 +5090,6 @@ qwen3.8next-fp4-b300-sglang-agentic-mtp: - dram-utilization: 0.8 search-space: - { tp: 1, ep: 1, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 12, 16], srt-recipe: benchmarks/single_node/srt-slurm-recipes/qwen3.8next/sglang/b300-fp4-mtp/agentic.yaml } -# Controlled AgentX power A/B: identical software, topology, MTP settings, -# concurrency, and memory tier across FP8 and FP4. The HBM-only rows measure -# the natural AgentX prefix-cache workload; HiCache isolates host-tier effects. -qwen3.5-fp8-b300-sglang-agentic-power-ab: - image: lmsysorg/sglang:v0.5.16-cu130 - model: Qwen/Qwen3.5-397B-A17B-FP8 - model-prefix: qwen3.5 - runner: cluster:b300-nv - precision: fp8 - framework: sglang - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 2, ep: 2, spec-decoding: mtp, kv-offloading: none, conc-list: [16, 32] } - - { tp: 2, ep: 2, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [16, 32] } - -qwen3.5-fp4-b300-sglang-agentic-power-ab: - image: lmsysorg/sglang:v0.5.16-cu130 - model: nvidia/Qwen3.5-397B-A17B-NVFP4 - model-prefix: qwen3.5 - runner: cluster:b300-nv - precision: fp4 - framework: sglang - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 2, ep: 2, spec-decoding: mtp, kv-offloading: none, conc-list: [16, 32] } - - { tp: 2, ep: 2, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [16, 32] } # CONC range conservative for H100's 80 GB HBM3 under the long-ISL with- # subagents corpus. hicache arm capped at conc 16 since high-conc + hicache @@ -5291,25 +5259,6 @@ qwen3.5-fp8-h100-sglang-agentic-mtp: search-space: - { tp: 8, ep: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 12, 16], srt-recipe: benchmarks/single_node/srt-slurm-recipes/qwen3.5/sglang/h100-fp8-mtp/agentic.yaml } - { tp: 8, ep: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [4, 8, 12, 16], srt-recipe: benchmarks/single_node/srt-slurm-recipes/qwen3.5/sglang/h100-fp8-mtp/agentic.yaml } - -# Qwen3.8-Flash-Next FP8 AgentX on H100 via SGLang with native NEXTN MTP. -# Day-zero recipe. H100 is Hopper, so FP8: NVFP4 needs SM100 tensor cores. The -# SGLang cookbook does not list H100, so this mirrors the H200 arm adjusted for -# the smaller part: TP8/EP8 rather than the cookbook's TP4/EP4, since 172.8 GiB -# at TP4 leaves too little of an 80 GB card for the 256k-capped traces. -qwen3.8next-fp8-h100-sglang-agentic-mtp: - image: lmsysorg/sglang:qwen38flashnext - model: Qwen/Qwen3.8-Flash-Next-FP8 - model-prefix: qwen3.8next - runner: cluster:h100-dgxc - precision: fp8 - framework: sglang - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.8 - search-space: - - { tp: 8, ep: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 12, 16] } qwen3.5-fp4-b200-trt: image: nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc18 model: nvidia/Qwen3.5-397B-A17B-NVFP4 diff --git a/docs/DOCUMENTATION_PLAN.md b/docs/DOCUMENTATION_PLAN.md index dee862d4e7..d5b91427a8 100644 --- a/docs/DOCUMENTATION_PLAN.md +++ b/docs/DOCUMENTATION_PLAN.md @@ -24,7 +24,7 @@ The main risks are: - Operational recovery knowledge remains trapped in a long agent-instruction file or historical debugging note. - New contributor-facing pages violate the bilingual documentation rule or omit the matching Chinese link. - Existing domain references are not consistently discoverable from `docs/`, and many are not paired with `_zh.md`. The migration must distinguish contributor-facing pages from internal implementation notes. -- AgentX documentation is inconsistent: `benchmarks/single_node/agentic/README.md` describes an unpublished experimental MVP while `MODELS.md` and active master configs expose agentic-coding coverage. +- AgentX documentation is scattered: the legacy `benchmarks/single_node/agentic/` README was removed with the bash scripts (#3461), and single-node AgentX now lives in srt-slurm recipes plus `benchmarks/srt_agentic.sh` with no maintained guide. ## Baseline knowledge inventory @@ -41,7 +41,7 @@ This table records the repository state before the staged consolidation below. | Multi-node recipes | `benchmarks/multi_node/srt-slurm-recipes/RECIPES.md` | Recipe/master-config coupling should be visible before editing either file | | Runner setup | `utils/runner_setup/RUNNER_SETUP.md`, `runners/` | Provisioning and runtime launcher concerns are separated | | Model and hardware catalog | `MODELS.md`, `configs/*-master.yaml` | Public model list and runnable config list serve different audiences | -| AgentX and agentic coding | `benchmarks/single_node/agentic/README.md`, `MODELS.md`, `configs/*-master.yaml` | Status and publication claims conflict. The official trace-to-result path needs one maintained guide | +| AgentX and agentic coding | `benchmarks/single_node/srt-slurm-recipes/**/agentic.yaml`, `benchmarks/srt_agentic.sh`, `MODELS.md`, `configs/*-master.yaml` | Status and publication claims conflict. The official trace-to-result path needs one maintained guide | | Artifact schemas and app handoff | `utils/process_result.py`, `utils/collect_*.py`, `../InferenceX-app/.github/workflows/ingest-results.yml`, `../InferenceX-app/packages/db/src/etl/*` | Artifact identities, JSON contracts, and the boundary into InferenceX-app are not documented in this repository | ## Target information architecture diff --git a/docs/DOCUMENTATION_PLAN_zh.md b/docs/DOCUMENTATION_PLAN_zh.md index ee3428005f..95fa40fb59 100644 --- a/docs/DOCUMENTATION_PLAN_zh.md +++ b/docs/DOCUMENTATION_PLAN_zh.md @@ -24,7 +24,7 @@ - 运维恢复知识被困在很长的 Agent 指令或历史调试记录中。 - 新的贡献者文档违反双语规则,或没有提供对应的中文链接。 - 现有领域参考文档没有统一从 `docs/` 发现,且许多页面没有 `_zh.md` 对应版本;迁移时必须区分贡献者文档与内部实现说明。 -- AgentX 文档存在不一致:`benchmarks/single_node/agentic/README.md` 仍描述未发布的实验性 MVP,但 `MODELS.md` 与主配置已经包含有效的 `agentic-coding` 覆盖。 +- AgentX 文档较为分散:旧的 `benchmarks/single_node/agentic/` README 已随 bash 脚本一并删除(#3461),单节点 AgentX 现由 srt-slurm 配方与 `benchmarks/srt_agentic.sh` 承载,尚无持续维护的指南。 ## 基线知识清单 @@ -41,7 +41,7 @@ | 多节点 Recipe | `benchmarks/multi_node/srt-slurm-recipes/RECIPES.md` | 编辑任一文件前应明确 Recipe 与主配置的耦合关系 | | Runner 初始化 | `utils/runner_setup/RUNNER_SETUP.md`、`runners/` | 部署与运行时启动器关注点分离 | | 模型与硬件目录 | `MODELS.md`、`configs/*-master.yaml` | 面向用户的模型列表与可运行配置列表服务于不同读者 | -| AgentX 与 Agentic Coding | `benchmarks/single_node/agentic/README.md`、`MODELS.md`、`configs/*-master.yaml` | 状态与发布说明冲突,需要一份持续维护的官方 Trace 到结果指南 | +| AgentX 与 Agentic Coding | `benchmarks/single_node/srt-slurm-recipes/**/agentic.yaml`、`benchmarks/srt_agentic.sh`、`MODELS.md`、`configs/*-master.yaml` | 状态与发布说明冲突,需要一份持续维护的官方 Trace 到结果指南 | | 产物 Schema 与 App 交接 | `utils/process_result.py`、`utils/collect_*.py`、`../InferenceX-app/.github/workflows/ingest-results.yml`、`../InferenceX-app/packages/db/src/etl/*` | 本仓库没有说明产物身份、JSON 契约以及进入 InferenceX-app 的边界 | ## 目标信息架构 diff --git a/docs/index.md b/docs/index.md index a57795adc7..d0548aa27e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -43,7 +43,7 @@ This is the mandatory low-context router for InferenceX work. Pick the one page | [`KLAUD_DEBUG.md`](../KLAUD_DEBUG.md) | Historical Klaud-Cold, CI, image, cluster, and GitHub CLI failure signatures | | [`klaud.md`](./klaud.md) / [`klaud_zh.md`](./klaud_zh.md) | Klaud Cold selection, ownership, validation and recovery | | [`klaud-reporting.md`](./klaud-reporting.md) / [`klaud-reporting_zh.md`](./klaud-reporting_zh.md) | Klaud PR body, progress comments, numeric comparisons and final preflight | -| [`benchmarks/single_node/agentic/README.md`](../benchmarks/single_node/agentic/README.md) | AgentX trace benchmark implementation | +| [`benchmarks/srt_agentic.sh`](../benchmarks/srt_agentic.sh) | AgentX trace replay client shared by single- and multi-node srt-slurm recipes | ## Context rules diff --git a/docs/index_zh.md b/docs/index_zh.md index 5c1b3da8bb..d8c6026163 100644 --- a/docs/index_zh.md +++ b/docs/index_zh.md @@ -43,7 +43,7 @@ | [`KLAUD_DEBUG.md`](../KLAUD_DEBUG.md) | Klaud-Cold、CI、镜像、集群与 GitHub CLI 的历史故障特征 | | [`klaud.md`](./klaud.md) / [`klaud_zh.md`](./klaud_zh.md) | Klaud Cold 候选选择、所有权、验证与恢复 | | [`klaud-reporting.md`](./klaud-reporting.md) / [`klaud-reporting_zh.md`](./klaud-reporting_zh.md) | Klaud PR 正文、进度评论、数值对比与最终预检 | -| [`benchmarks/single_node/agentic/README.md`](../benchmarks/single_node/agentic/README.md) | AgentX Trace 基准测试实现 | +| [`benchmarks/srt_agentic.sh`](../benchmarks/srt_agentic.sh) | 单节点与多节点 srt-slurm 配方共用的 AgentX Trace 回放客户端 | ## 上下文规则 diff --git a/docs/waiver/2508.md b/docs/waiver/2508.md index 6fdc3e7c64..66e9e9afa4 100644 --- a/docs/waiver/2508.md +++ b/docs/waiver/2508.md @@ -9,8 +9,9 @@ prohibits unless covered by a filled-out waiver. - **Master config entry:** `kimik3-fp4-mi355x-vllm-agentic-mtp` in [`configs/amd-master.yaml`](../../configs/amd-master.yaml) - **Pinned image:** `vllm/vllm-openai-rocm:nightly-cb8104839c141609d99f1254459ef3a4f1bd4263` - **Patch entrypoint:** `bash "$(dirname "$0")/apply_k3_container_patches.sh"` invoked from - [`benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh`](../../benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh) -- **Patch script:** [`benchmarks/single_node/agentic/apply_k3_container_patches.sh`](../../benchmarks/single_node/agentic/apply_k3_container_patches.sh) + the legacy `kimik3_fp4_mi355x_mtp.sh` AgentX script +- **Patch script:** `apply_k3_container_patches.sh` +- **Status:** both scripts, and the `kimik3-fp4-mi355x-vllm-agentic-mtp` config that used them, were removed with the legacy single-node AgentX bash path (#3460, #3461). ## What is patched diff --git a/runners/launch_gb200-nv.sh b/runners/launch_gb200-nv.sh index 66a0996ba3..c217d42f0a 100755 --- a/runners/launch_gb200-nv.sh +++ b/runners/launch_gb200-nv.sh @@ -106,41 +106,6 @@ if [[ "$IS_MULTINODE" != true && -n "${SRT_RECIPE:-}" ]]; then exit $? fi -# Direct single-tray AgentX uses the existing shared image and HF caches. -if [[ "$MODEL_PREFIX" == "dsv41flash" && ( "$FRAMEWORK" == "vllm" || "$FRAMEWORK" == "sglang" ) && "${IS_MULTINODE}" != "true" ]]; then - check_env_vars SPEC_DECODING - BENCH_SCRIPT="benchmarks/single_node/agentic/${MODEL_PREFIX}_${PRECISION}_gb200_${FRAMEWORK}" - case "$SPEC_DECODING" in - mtp) BENCH_SCRIPT+="_mtp.sh" ;; - none) - [[ "$FRAMEWORK" == "sglang" ]] || { echo "Native STP requires the SGLang recipe" >&2; exit 1; } - BENCH_SCRIPT+=".sh" - ;; - *) echo "Unsupported SPEC_DECODING=$SPEC_DECODING" >&2; exit 1 ;; - esac - # Cover DSpark5 verification for concurrent AgentX subagents at c1/c2/c4. - export DSV41_MIN_CUDAGRAPH_CAPTURE_SIZE=64 - [[ "${IS_AGENTIC}" == "1" && -f "$BENCH_SCRIPT" ]] || { - echo "Unsupported single-node recipe: $BENCH_SCRIPT" >&2 - exit 1 - } - HF_HUB_CACHE_HOST_PATH="/mnt/lustre01/users-public/sa-shared/hf-hub-cache" - mkdir -p "$HF_HUB_CACHE_HOST_PATH" - export MODEL_PATH="$MODEL" HF_HUB_CACHE=/hf-cache - export INFMAX_CONTAINER_WORKSPACE=/ix RESULT_DIR=/ix/results - SQUASH_FILE="$SQUASH_DIR/$(echo "$IMAGE" | sed 's/[\/:@#]/_/g').sqsh" - import_squash "$SQUASH_FILE" "$IMAGE" - srun --account="$SLURM_ACCOUNT" --partition="$SLURM_PARTITION" \ - --nodes=1 --ntasks=1 --gpus="${TP:?}" --exclusive --mem=0 \ - --time="${SALLOC_TIME_LIMIT}" --job-name="$RUNNER_NAME" \ - --mpi=none --container-image="$SQUASH_FILE" \ - --container-mounts="$GITHUB_WORKSPACE:/ix,$HF_HUB_CACHE_HOST_PATH:/hf-cache" \ - --no-container-mount-home --container-remap-root \ - --container-workdir=/ix --no-container-entrypoint \ - --export=ALL,PORT=8888 bash "$BENCH_SCRIPT" - exit $? -fi - if [[ "$FRAMEWORK" == "llmd-vllm" ]]; then if [[ "$MODEL_PREFIX" == "dsv4" && "$PRECISION" == "fp4" ]]; then export MODEL_PATH="/mnt/numa1/models/DeepSeek-V4-Pro" diff --git a/runners/launch_gb300-nv.sh b/runners/launch_gb300-nv.sh index 7d579742b0..a09a238e40 100644 --- a/runners/launch_gb300-nv.sh +++ b/runners/launch_gb300-nv.sh @@ -117,39 +117,6 @@ if [[ "$IS_MULTINODE" != true && -n "${SRT_RECIPE:-}" ]]; then --var DYNAMO_WHEELS_CACHE_HOST_PATH "$DYNAMO_WHEELS_CACHE_HOST_PATH" exit $? fi -# Keep this branch before the nginx import and srtctl setup. -if [[ "$MODEL_PREFIX" == "dsv41flash" && ( "$FRAMEWORK" == "vllm" || "$FRAMEWORK" == "sglang" ) && "${IS_MULTINODE}" != "true" ]]; then - check_env_vars SPEC_DECODING - BENCH_SCRIPT="benchmarks/single_node/agentic/${MODEL_PREFIX}_${PRECISION}_gb300_${FRAMEWORK}" - case "$SPEC_DECODING" in - mtp) BENCH_SCRIPT+="_mtp.sh" ;; - none) - [[ "$FRAMEWORK" == "sglang" ]] || { echo "Native STP requires the SGLang recipe" >&2; exit 1; } - BENCH_SCRIPT+=".sh" - ;; - *) echo "Unsupported SPEC_DECODING=$SPEC_DECODING" >&2; exit 1 ;; - esac - # Cover DSpark5 verification for concurrent AgentX subagents at c1/c2/c4. - export DSV41_MIN_CUDAGRAPH_CAPTURE_SIZE=64 - [[ "${IS_AGENTIC}" == "1" && -f "$BENCH_SCRIPT" ]] || { - echo "Unsupported single-node recipe: $BENCH_SCRIPT" >&2 - exit 1 - } - export HF_HUB_CACHE=/hf-cache - export INFMAX_CONTAINER_WORKSPACE=/ix - export RESULT_DIR=/ix/results - # Cold model loading and graph capture exceeded the one-hour frontend deadline. - export VLLM_ENGINE_READY_TIMEOUT_S=7200 - srun --account="$SLURM_ACCOUNT" --partition="$SLURM_PARTITION" \ - --nodes=1 --ntasks=1 --gpus="${TP:?}" --cpus-per-task=144 --exclusive --mem=0 \ - --time="${SALLOC_TIME_LIMIT}" --job-name="$RUNNER_NAME" \ - --mpi=none --container-image="$SQUASH_FILE" \ - --container-mounts="$GITHUB_WORKSPACE:/ix,$HF_HUB_CACHE_HOST_PATH:/hf-cache" \ - --no-container-mount-home --container-remap-root \ - --container-workdir=/ix --no-container-entrypoint \ - --export=ALL,PORT=8888 bash "$BENCH_SCRIPT" - exit $? -fi import_squash "$NGINX_SQUASH_FILE" "$NGINX_IMAGE" diff --git a/runners/launch_mi355x-amds.sh b/runners/launch_mi355x-amds.sh index 29282f9cc8..7f313f8fb9 100644 --- a/runners/launch_mi355x-amds.sh +++ b/runners/launch_mi355x-amds.sh @@ -369,13 +369,6 @@ else BENCHMARK_SCRIPT="$SCRIPT_FALLBACK" fi - if [[ "$BENCHMARK_SCRIPT" == "benchmarks/single_node/agentic/minimaxm3_fp4_mi355x_atom_mtp.sh" ]]; then - export MODEL_PATH="$MODEL" - export ENABLE_PREFIX_CACHING=true - export AITER_LOG_LEVEL=WARNING - export EVAL_TASKS_DIR=infx/evals/gsm8k.yaml - fi - srun --jobid=$JOB_ID \ --container-image=$SQUASH_FILE \ --container-mounts=$GITHUB_WORKSPACE:$CONTAINER_REPO/,$HF_HUB_CACHE_MOUNT:$HF_HUB_CACHE,$AIPERF_MMAP_CACHE_HOST_PATH:/aiperf_mmap_cache \ From f45d6bff02f51c43ae9c686817ce58d0b483ed42 Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Sat, 26 Sep 2026 02:29:41 -0400 Subject: [PATCH 2/6] Drop the DSv4.1 Flash H100/H200 block32 kernel tilings instead of restoring them Per review: do not re-add the block32 installers and kernel configs. Remove the restored dsv41flash-block32 assets, the two block32 setup scripts, the recipes' setup_script entries and the H200 DSV41_BLOCK32_TP env vars; the H100/H200 SGLang AgentX recipes now run with SGLang's default tilings. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../install_h100_block32_configs.py | 48 ------- .../install_h200_block32_configs.py | 43 ------ ...3,dtype=fp8_w8a8,block_shape=[32, 32].json | 114 ---------------- ...3,dtype=fp8_w8a8,block_shape=[32, 32].json | 50 ------- ...3,dtype=fp8_w8a8,block_shape=[32, 32].json | 98 -------------- ...3,dtype=fp8_w8a8,block_shape=[32, 32].json | 66 ---------- ...3,dtype=fp8_w8a8,block_shape=[32, 32].json | 66 ---------- ...3,dtype=fp8_w8a8,block_shape=[32, 32].json | 122 ------------------ .../h100_dsv41_block32/provenance.json | 71 ---------- ...0,dtype=fp8_w8a8,block_shape=[32, 32].json | 58 --------- ...0,dtype=fp8_w8a8,block_shape=[32, 32].json | 66 ---------- ...0,dtype=fp8_w8a8,block_shape=[32, 32].json | 66 ---------- ...0,dtype=fp8_w8a8,block_shape=[32, 32].json | 122 ------------------ .../h200_dsv41_block32/provenance.json | 16 --- .../configs/dsv41flash-h100-block32.sh | 5 - .../configs/dsv41flash-h200-block32.sh | 5 - .../sglang/h100-fp4-mtp/agentic.yaml | 2 - .../sglang/h200-fp4-mtp/agentic.yaml | 35 ----- 18 files changed, 1053 deletions(-) delete mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/install_h100_block32_configs.py delete mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/install_h200_block32_configs.py delete mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=1280,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json delete mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=1792,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json delete mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=512,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json delete mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=5120,K=1024,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json delete mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=5120,K=288,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json delete mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=576,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json delete mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/provenance.json delete mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=1792,K=5120,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json delete mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=5120,K=1024,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json delete mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=5120,K=288,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json delete mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=576,K=5120,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json delete mode 100644 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/provenance.json delete mode 100755 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-h100-block32.sh delete mode 100755 benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-h200-block32.sh diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/install_h100_block32_configs.py b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/install_h100_block32_configs.py deleted file mode 100644 index 88c3fc97f6..0000000000 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/install_h100_block32_configs.py +++ /dev/null @@ -1,48 +0,0 @@ -"""Install the H100 V4.1 tiling configs into the imported SGLang kernel package.""" # noqa: INP001 -- Standalone recipe executable. - -import hashlib -import json -import shutil -import sys -from pathlib import Path - -from sglang.kernels.ops.quantization import fp8_kernel - - -def main() -> None: - source = Path(sys.argv[1]) - artifacts = Path(sys.argv[2]) / "fp8_kernel_configs" - device = fp8_kernel.get_device_name().replace(" ", "_") - if device != "NVIDIA_H100_80GB_HBM3": - raise RuntimeError(f"H100 tiling configs cannot be installed on {device}") - destination = Path(fp8_kernel.__file__).resolve().parent / "configs" - destination.mkdir(exist_ok=True) - artifacts.mkdir(parents=True, exist_ok=True) - for n, k in [ - (5120, 1024), - (5120, 288), - (576, 5120), - (1792, 5120), - (512, 5120), - (1280, 5120), - ]: - name = ( - f"N={n},K={k},device_name={device},dtype=fp8_w8a8,block_shape=[32, 32].json" - ) - path = source / name - expected = { - int(m): config for m, config in json.loads(path.read_text()).items() - } - shutil.copyfile(path, destination / name) - shutil.copyfile(path, artifacts / name) - fp8_kernel.get_w8a8_block_fp8_configs.cache_clear() - actual = fp8_kernel.get_w8a8_block_fp8_configs(n, k, 32, 32) - if actual != expected: - raise RuntimeError(f"SGLang did not resolve the installed config: {name}") - print( - f"Installed {destination / name}: sha256={hashlib.sha256(path.read_bytes()).hexdigest()}" - ) - - -if __name__ == "__main__": - main() diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/install_h200_block32_configs.py b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/install_h200_block32_configs.py deleted file mode 100644 index 3d34a865e2..0000000000 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/install_h200_block32_configs.py +++ /dev/null @@ -1,43 +0,0 @@ -"""Install the H200 V4.1 tiling configs into the imported SGLang kernel package.""" # noqa: INP001 -- Standalone recipe executable. - -import hashlib -import json -import shutil -import sys -from pathlib import Path - -from sglang.kernels.ops.quantization import fp8_kernel - - -def main() -> None: - source = Path(sys.argv[1]) - artifacts = Path(sys.argv[2]) / "fp8_kernel_configs" - tp = int(sys.argv[3]) - if tp not in (4, 8): - raise ValueError(f"H200 tiling configs support TP4 or TP8, got TP{tp}") - device = fp8_kernel.get_device_name().replace(" ", "_") - if device != "NVIDIA_H200": - raise RuntimeError(f"H200 tiling configs cannot be installed on {device}") - destination = Path(fp8_kernel.__file__).resolve().parent / "configs" - destination.mkdir(exist_ok=True) - artifacts.mkdir(parents=True, exist_ok=True) - shapes = [(1792, 5120)] - if tp == 8: - shapes += [(5120, 1024), (5120, 288), (576, 5120)] - for n, k in shapes: - name = f"N={n},K={k},device_name={device},dtype=fp8_w8a8,block_shape=[32, 32].json" - path = source / name - expected = {int(m): config for m, config in json.loads(path.read_text()).items()} - shutil.copyfile(path, destination / name) - shutil.copyfile(path, artifacts / name) - fp8_kernel.get_w8a8_block_fp8_configs.cache_clear() - actual = fp8_kernel.get_w8a8_block_fp8_configs(n, k, 32, 32) - if actual != expected: - raise RuntimeError(f"SGLang did not resolve the installed config: {name}") - print( - f"Installed {destination / name}: sha256={hashlib.sha256(path.read_bytes()).hexdigest()}" - ) - - -if __name__ == "__main__": - main() diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=1280,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=1280,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json deleted file mode 100644 index 971110754f..0000000000 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=1280,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json +++ /dev/null @@ -1,114 +0,0 @@ -{ - "0": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3 - }, - "1": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3, - "SPLIT_K": 16, - "SWAP_AB": false - }, - "5": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3, - "SPLIT_K": 16, - "SWAP_AB": false - }, - "10": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3, - "SPLIT_K": 16, - "SWAP_AB": true - }, - "20": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3, - "SPLIT_K": 8, - "SWAP_AB": true - }, - "40": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3, - "SPLIT_K": 8, - "SWAP_AB": true - }, - "80": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3, - "SPLIT_K": 16, - "SWAP_AB": true - }, - "160": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3, - "SPLIT_K": 8, - "SWAP_AB": true - }, - "190": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3, - "SPLIT_K": 8, - "SWAP_AB": true - }, - "191": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3 - }, - "320": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3 - }, - "321": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3 - } -} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=1792,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=1792,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json deleted file mode 100644 index 986ff1953b..0000000000 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=1792,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json +++ /dev/null @@ -1,50 +0,0 @@ -{ - "1": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 16, - "SWAP_AB": false, - "num_stages": 3, - "num_warps": 4 - }, - "6": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 16, - "SWAP_AB": true, - "num_stages": 3, - "num_warps": 4 - }, - "24": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "GROUP_SIZE_M": 32, - "SPLIT_K": 8, - "SWAP_AB": true, - "num_stages": 4, - "num_warps": 4 - }, - "96": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "GROUP_SIZE_M": 32, - "SPLIT_K": 8, - "SWAP_AB": true, - "num_stages": 4, - "num_warps": 4 - }, - "128": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "num_stages": 3, - "num_warps": 4 - } -} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=512,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=512,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json deleted file mode 100644 index 3c4f7326b0..0000000000 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=512,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json +++ /dev/null @@ -1,98 +0,0 @@ -{ - "0": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3 - }, - "1": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3, - "SPLIT_K": 16, - "SWAP_AB": false - }, - "5": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3, - "SPLIT_K": 16, - "SWAP_AB": false - }, - "10": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3, - "SPLIT_K": 16, - "SWAP_AB": false - }, - "20": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3, - "SPLIT_K": 16, - "SWAP_AB": true - }, - "40": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3, - "SPLIT_K": 16, - "SWAP_AB": true - }, - "80": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3, - "SPLIT_K": 8, - "SWAP_AB": true - }, - "160": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3, - "SPLIT_K": 4, - "SWAP_AB": true - }, - "320": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3, - "SPLIT_K": 4, - "SWAP_AB": true - }, - "321": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3 - } -} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=5120,K=1024,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=5120,K=1024,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json deleted file mode 100644 index fece264886..0000000000 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=5120,K=1024,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json +++ /dev/null @@ -1,66 +0,0 @@ -{ - "1": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 16, - "num_warps": 4, - "num_stages": 5 - }, - "4": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 64, - "num_warps": 4, - "num_stages": 4 - }, - "16": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 1, - "num_warps": 4, - "num_stages": 4 - }, - "32": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 16, - "num_warps": 4, - "num_stages": 5 - }, - "64": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 1, - "num_warps": 4, - "num_stages": 5 - }, - "256": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 16, - "num_warps": 4, - "num_stages": 5 - }, - "1024": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 1, - "num_warps": 4, - "num_stages": 5 - }, - "4096": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 1, - "num_warps": 4, - "num_stages": 4 - } -} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=5120,K=288,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=5120,K=288,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json deleted file mode 100644 index ccf51c6e0a..0000000000 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=5120,K=288,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json +++ /dev/null @@ -1,66 +0,0 @@ -{ - "1": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 16, - "num_warps": 4, - "num_stages": 2 - }, - "4": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 2 - }, - "16": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 8, - "num_stages": 2 - }, - "32": { - "BLOCK_SIZE_M": 32, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 64, - "num_warps": 4, - "num_stages": 5 - }, - "64": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3 - }, - "256": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 64, - "num_warps": 4, - "num_stages": 5 - }, - "1024": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 16, - "num_warps": 4, - "num_stages": 2 - }, - "4096": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 16, - "num_warps": 4, - "num_stages": 3 - } -} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=576,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=576,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json deleted file mode 100644 index 4e45c66e6d..0000000000 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/N=576,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json +++ /dev/null @@ -1,122 +0,0 @@ -{ - "1": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 16, - "SWAP_AB": false, - "num_stages": 3, - "num_warps": 4 - }, - "4": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 16, - "SWAP_AB": true, - "num_stages": 3, - "num_warps": 4 - }, - "6": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 16, - "SWAP_AB": true, - "num_stages": 3, - "num_warps": 4 - }, - "16": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 16, - "SWAP_AB": true, - "num_stages": 3, - "num_warps": 4 - }, - "24": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 16, - "SWAP_AB": true, - "num_stages": 3, - "num_warps": 4 - }, - "32": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 16, - "SWAP_AB": true, - "num_stages": 3, - "num_warps": 4 - }, - "64": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "GROUP_SIZE_M": 32, - "SPLIT_K": 8, - "SWAP_AB": true, - "num_stages": 4, - "num_warps": 4 - }, - "96": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "GROUP_SIZE_M": 32, - "SPLIT_K": 8, - "SWAP_AB": true, - "num_stages": 4, - "num_warps": 4 - }, - "128": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "GROUP_SIZE_M": 16, - "num_stages": 5, - "num_warps": 4 - }, - "193": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "GROUP_SIZE_M": 32, - "num_stages": 4, - "num_warps": 4 - }, - "256": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "GROUP_SIZE_M": 32, - "num_stages": 4, - "num_warps": 4 - }, - "1024": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "GROUP_SIZE_M": 1, - "num_stages": 4, - "num_warps": 4 - }, - "4096": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "GROUP_SIZE_M": 1, - "num_stages": 5, - "num_warps": 4 - } -} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/provenance.json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/provenance.json deleted file mode 100644 index 9c4fff096c..0000000000 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h100_dsv41_block32/provenance.json +++ /dev/null @@ -1,71 +0,0 @@ -{ - "sglang_commit": "0f6761b54facebb47f2068f87ecccd8f14da3a0e", - "image": "lmsysorg/sglang:nightly-dev-cu13-20260921-0f6761b5", - "hardware": "NVIDIA H100 80GB HBM3", - "tuner": "benchmark/kernels/quantization/tuning_block_wise_kernel.py", - "search_change": "Use BLOCK_SIZE_K=[32] for checkpoint block32 scales; upstream CUDA candidates [64,128] otherwise filter to an empty search.", - "tuning_slurm_job": 18452, - "selected_audit_slurm_job": 18457, - "resolver_install_slurm_job": 18458, - "validation": "39 cases: FP32 dequantized reference rtol=0.02/atol=0.05; exact CUDA graph/eager equality; alternating timing order. BF16 rounding may differ from default tiling.", - "selection": "N5120/K288 batch64 restored to upstream default after timing regression.", - "precision": "Stock FP8 inputs and scales, BF16 output; no quantization or model changes.", - "split_k_audit": { - "slurm_job": 18479, - "cases": 80, - "mechanism": "Pinned upstream Hopper SPLIT_K and SWAP_AB launch configuration, FP32 partial reduction, unchanged FP8 inputs/scales and BF16 output", - "shapes": [ - "N1792/K5120", - "N576/K5120" - ], - "boundary_validation": "Slurm18480:56/56 FP32-reference and exact CUDA-graph replay checks passed, batch boundariesM1..1024; changed cases all faster in alternating-order microtimings", - "large_batch_policy": "Retain original fallback selection for M129..8192 using explicit boundary anchors" - }, - "draft_projection_audit": { - "device": "NVIDIA H100 80GB HBM3", - "image": "nightly-dev-cu13-20260922-582389ce", - "search": { - "job": 18492, - "cases": 144, - "status": "COMPLETED0:0" - }, - "boundary": { - "job": 18493, - "cases": 206, - "status": "COMPLETED0:0" - }, - "trimmed_boundary": { - "job": 18494, - "cases": 20, - "status": "COMPLETED0:0" - }, - "reference": "FP32 dequant reference rtol.02/atol.05; exact eager/graph replay; no quantization or kernel code changes", - "ranges": { - "N512": "M1..320 tuned; M0 and M>=321 stock", - "N1280": "M1..190 tuned; M0 and M>=191 stock" - }, - "retained_speedup_range": [ - 1.1861693666195017, - 6.770688638312714 - ], - "baseline_l2_max": 0.0016977186314761639, - "candidate_l2_max": 0.0016970767173916101, - "hashes": { - "N=1280,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json": "635f684b6ed8b538709249baa5c474bf8583f952bcee330e01fd0f9e0ce714a3", - "N=512,K=5120,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[32, 32].json": "193fc012550dce39bb4de0a4bde28eed5cee79f2900ed95841fe18ccb6d8c187" - }, - "serving_qualification": { - "candidate_run": 35683668032, - "baseline_run": 35683669703, - "protocol": "Matched canonical C1, 3600 seconds, normal warmup; zero profiled errors and valid power in both runs", - "candidate_total_tps_per_gpu": 1239.57314, - "baseline_total_tps_per_gpu": 958.51774, - "candidate_p90_interactivity": 90.89516, - "baseline_p90_interactivity": 60.93631, - "full_gsm8k_run": 35683671358, - "full_gsm8k_examples": 1319, - "full_gsm8k_strict_accuracy": 0.9696739954510993, - "final_sweep": "Required after integration" - } - } -} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=1792,K=5120,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=1792,K=5120,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json deleted file mode 100644 index 83620a6b10..0000000000 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=1792,K=5120,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json +++ /dev/null @@ -1,58 +0,0 @@ -{ - "1": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "GROUP_SIZE_M": 32, - "SPLIT_K": 8, - "SWAP_AB": true, - "num_stages": 4, - "num_warps": 4 - }, - "2": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "num_stages": 3, - "num_warps": 4 - }, - "6": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 16, - "SWAP_AB": true, - "num_stages": 3, - "num_warps": 4 - }, - "24": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 8, - "SWAP_AB": true, - "num_stages": 3, - "num_warps": 4 - }, - "96": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 16, - "SWAP_AB": true, - "num_stages": 3, - "num_warps": 4 - }, - "128": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "num_stages": 3, - "num_warps": 4 - } -} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=5120,K=1024,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=5120,K=1024,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json deleted file mode 100644 index 850f61947e..0000000000 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=5120,K=1024,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json +++ /dev/null @@ -1,66 +0,0 @@ -{ - "1": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 64, - "num_warps": 4, - "num_stages": 3 - }, - "4": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 64, - "num_warps": 4, - "num_stages": 4 - }, - "16": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 16, - "num_warps": 4, - "num_stages": 5 - }, - "32": { - "BLOCK_SIZE_M": 32, - "BLOCK_SIZE_N": 64, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 4 - }, - "64": { - "BLOCK_SIZE_M": 32, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 16, - "num_warps": 4, - "num_stages": 3 - }, - "256": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 16, - "num_warps": 4, - "num_stages": 5 - }, - "1024": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3 - }, - "4096": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 1, - "num_warps": 4, - "num_stages": 4 - } -} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=5120,K=288,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=5120,K=288,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json deleted file mode 100644 index 94b8a3fccf..0000000000 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=5120,K=288,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json +++ /dev/null @@ -1,66 +0,0 @@ -{ - "1": { - "BLOCK_SIZE_M": 32, - "BLOCK_SIZE_N": 64, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 8, - "num_stages": 2 - }, - "4": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 1, - "num_warps": 4, - "num_stages": 4 - }, - "16": { - "BLOCK_SIZE_M": 32, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 64, - "num_warps": 4, - "num_stages": 2 - }, - "32": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 32, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 3 - }, - "64": { - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 16, - "num_warps": 4, - "num_stages": 4 - }, - "256": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 64, - "num_warps": 4, - "num_stages": 3 - }, - "1024": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 32, - "num_warps": 4, - "num_stages": 4 - }, - "4096": { - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "BLOCK_SIZE_K": 32, - "GROUP_SIZE_M": 64, - "num_warps": 4, - "num_stages": 4 - } -} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=576,K=5120,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=576,K=5120,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json deleted file mode 100644 index 4359ec7f93..0000000000 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/N=576,K=5120,device_name=NVIDIA_H200,dtype=fp8_w8a8,block_shape=[32, 32].json +++ /dev/null @@ -1,122 +0,0 @@ -{ - "1": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 16, - "SWAP_AB": false, - "num_stages": 3, - "num_warps": 4 - }, - "4": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 16, - "SWAP_AB": false, - "num_stages": 3, - "num_warps": 4 - }, - "6": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 16, - "SWAP_AB": false, - "num_stages": 3, - "num_warps": 4 - }, - "16": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 16, - "SWAP_AB": true, - "num_stages": 3, - "num_warps": 4 - }, - "24": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 16, - "SWAP_AB": true, - "num_stages": 3, - "num_warps": 4 - }, - "32": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 16, - "SWAP_AB": true, - "num_stages": 3, - "num_warps": 4 - }, - "64": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 8, - "SWAP_AB": true, - "num_stages": 3, - "num_warps": 4 - }, - "96": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 32, - "GROUP_SIZE_M": 32, - "SPLIT_K": 8, - "SWAP_AB": true, - "num_stages": 3, - "num_warps": 4 - }, - "128": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "GROUP_SIZE_M": 16, - "num_stages": 5, - "num_warps": 4 - }, - "193": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "GROUP_SIZE_M": 64, - "num_stages": 3, - "num_warps": 4 - }, - "256": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 16, - "BLOCK_SIZE_N": 64, - "GROUP_SIZE_M": 64, - "num_stages": 3, - "num_warps": 4 - }, - "1024": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "GROUP_SIZE_M": 1, - "num_stages": 5, - "num_warps": 4 - }, - "4096": { - "BLOCK_SIZE_K": 32, - "BLOCK_SIZE_M": 64, - "BLOCK_SIZE_N": 128, - "GROUP_SIZE_M": 1, - "num_stages": 5, - "num_warps": 4 - } -} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/provenance.json b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/provenance.json deleted file mode 100644 index 074329f645..0000000000 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32/kernel_configs/h200_dsv41_block32/provenance.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "sglang_commit": "0f6761b54facebb47f2068f87ecccd8f14da3a0e", - "image": "lmsysorg/sglang:nightly-dev-cu13-20260921-0f6761b5", - "hardware": "NVIDIA H200", - "tuner": "benchmark/kernels/quantization/tuning_block_wise_kernel.py", - "search_change": "Use BLOCK_SIZE_K=[32] for checkpoint block32 scales; upstream CUDA candidates [64,128] otherwise filter to an empty search.", - "tuning_slurm_job": 88009, - "selected_audit_and_resolver_slurm_job": 88030, - "validation": "39 cases against FP32 dequantized reference at rtol=0.02/atol=0.05; exact CUDA graph/eager equality; alternating timing order. BF16 rounding may differ from default tiling.", - "selection": "N5120/K288 batch32 restored to upstream default after timing regression.", - "precision": "Stock FP8 inputs and scales, BF16 output; no quantization or model changes.", - "splitk_tuning_slurm_job": 88079, - "splitk_selected_boundary_validation_slurm_job": 88084, - "splitk_scope": "Supported upstream SPLIT_K/SWAP_AB launch settings for N1792K5120 and N576K5120 at small verification batches; unchanged prior config selections for all M129..8192.", - "splitk_validation": "72/72 exploratory cases and56/56 selected boundary cases passed FP32reference rtol=.02/atol=.05 plus exact eager/graph equality; alternating timing order, selected changed cases improved1.37\u20136.22x." -} diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-h100-block32.sh b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-h100-block32.sh deleted file mode 100755 index b5ae175fc1..0000000000 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-h100-block32.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/usr/bin/env bash -# Install the measured H100 DeepSeek-V4.1-Flash block-32 tilings into the worker's SGLang. -set -euo pipefail -assets=/infmax-workspace/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32 -python3 "$assets/install_h100_block32_configs.py" "$assets/kernel_configs/h100_dsv41_block32" /logs diff --git a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-h200-block32.sh b/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-h200-block32.sh deleted file mode 100755 index 6d80c7a180..0000000000 --- a/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-h200-block32.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/usr/bin/env bash -# Install the measured H200 DeepSeek-V4.1-Flash block-32 tilings into the worker's SGLang. -set -euo pipefail -assets=/infmax-workspace/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-block32 -python3 "$assets/install_h200_block32_configs.py" "$assets/kernel_configs/h200_dsv41_block32" /logs "$DSV41_BLOCK32_TP" diff --git a/benchmarks/single_node/srt-slurm-recipes/dsv41flash/sglang/h100-fp4-mtp/agentic.yaml b/benchmarks/single_node/srt-slurm-recipes/dsv41flash/sglang/h100-fp4-mtp/agentic.yaml index 6f4767f0ce..041fb4849f 100644 --- a/benchmarks/single_node/srt-slurm-recipes/dsv41flash/sglang/h100-fp4-mtp/agentic.yaml +++ b/benchmarks/single_node/srt-slurm-recipes/dsv41flash/sglang/h100-fp4-mtp/agentic.yaml @@ -29,8 +29,6 @@ base: health_check: interval_seconds: 10 max_attempts: 360 - # Hardware-specific tiling only; checkpoint data, scales and dtypes are unchanged. - setup_script: dsv41flash-h100-block32.sh roles: agg: nodes: 1 diff --git a/benchmarks/single_node/srt-slurm-recipes/dsv41flash/sglang/h200-fp4-mtp/agentic.yaml b/benchmarks/single_node/srt-slurm-recipes/dsv41flash/sglang/h200-fp4-mtp/agentic.yaml index 0b55113278..a5d86251de 100644 --- a/benchmarks/single_node/srt-slurm-recipes/dsv41flash/sglang/h200-fp4-mtp/agentic.yaml +++ b/benchmarks/single_node/srt-slurm-recipes/dsv41flash/sglang/h200-fp4-mtp/agentic.yaml @@ -19,9 +19,6 @@ base: tachometer: enabled: false engine: sglang - # Measured H200 launch configurations: the replicated projection on TP4 and - # TP8, plus the qualified TP8 sharded shapes. - setup_script: dsv41flash-h200-block32.sh roles: agg: nodes: 1 @@ -90,8 +87,6 @@ override_tp4_c1: args: tensor-parallel-size: 4 max-running-requests: 2 - env: - DSV41_BLOCK32_TP: '4' benchmark: env: CONC: '1' @@ -104,8 +99,6 @@ override_tp4_c2: args: tensor-parallel-size: 4 max-running-requests: 4 - env: - DSV41_BLOCK32_TP: '4' benchmark: env: CONC: '2' @@ -118,8 +111,6 @@ override_tp4_c4: args: tensor-parallel-size: 4 max-running-requests: 8 - env: - DSV41_BLOCK32_TP: '4' benchmark: env: CONC: '4' @@ -132,8 +123,6 @@ override_tp4_c8: args: tensor-parallel-size: 4 max-running-requests: 16 - env: - DSV41_BLOCK32_TP: '4' benchmark: env: CONC: '8' @@ -146,8 +135,6 @@ override_tp4_c16: args: tensor-parallel-size: 4 max-running-requests: 32 - env: - DSV41_BLOCK32_TP: '4' benchmark: env: CONC: '16' @@ -160,8 +147,6 @@ override_tp4_c32: args: tensor-parallel-size: 4 max-running-requests: 64 - env: - DSV41_BLOCK32_TP: '4' benchmark: env: CONC: '32' @@ -174,8 +159,6 @@ override_tp4_c64: args: tensor-parallel-size: 4 max-running-requests: 64 - env: - DSV41_BLOCK32_TP: '4' benchmark: env: CONC: '64' @@ -188,8 +171,6 @@ override_tp4_c128: args: tensor-parallel-size: 4 max-running-requests: 64 - env: - DSV41_BLOCK32_TP: '4' benchmark: env: CONC: '128' @@ -202,8 +183,6 @@ override_tp8_c1: args: tensor-parallel-size: 8 max-running-requests: 2 - env: - DSV41_BLOCK32_TP: '8' benchmark: env: CONC: '1' @@ -216,8 +195,6 @@ override_tp8_c2: args: tensor-parallel-size: 8 max-running-requests: 4 - env: - DSV41_BLOCK32_TP: '8' benchmark: env: CONC: '2' @@ -230,8 +207,6 @@ override_tp8_c4: args: tensor-parallel-size: 8 max-running-requests: 8 - env: - DSV41_BLOCK32_TP: '8' benchmark: env: CONC: '4' @@ -244,8 +219,6 @@ override_tp8_c8: args: tensor-parallel-size: 8 max-running-requests: 16 - env: - DSV41_BLOCK32_TP: '8' benchmark: env: CONC: '8' @@ -258,8 +231,6 @@ override_tp8_c16: args: tensor-parallel-size: 8 max-running-requests: 32 - env: - DSV41_BLOCK32_TP: '8' benchmark: env: CONC: '16' @@ -272,8 +243,6 @@ override_tp8_c32: args: tensor-parallel-size: 8 max-running-requests: 64 - env: - DSV41_BLOCK32_TP: '8' benchmark: env: CONC: '32' @@ -286,8 +255,6 @@ override_tp8_c64: args: tensor-parallel-size: 8 max-running-requests: 64 - env: - DSV41_BLOCK32_TP: '8' benchmark: env: CONC: '64' @@ -300,8 +267,6 @@ override_tp8_c128: args: tensor-parallel-size: 8 max-running-requests: 64 - env: - DSV41_BLOCK32_TP: '8' benchmark: env: CONC: '128' From 3fdd26d2d64b42ca597fc70990e5a054effe1cfa Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Sat, 26 Sep 2026 02:35:28 -0400 Subject: [PATCH 3/6] Drop the KLAUD_DEBUG script pointer (fixed_seq_len is deleted in #3464) Co-Authored-By: Claude Opus 5.5 (1M context) --- KLAUD_DEBUG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/KLAUD_DEBUG.md b/KLAUD_DEBUG.md index 0f0fb21043..7d50415bc7 100644 --- a/KLAUD_DEBUG.md +++ b/KLAUD_DEBUG.md @@ -77,7 +77,7 @@ changing the benchmark recipe. **Fix:** in `benchmarks/single_node/.sh`, either: 1. **Lower `--gpu-memory-utilization`** (`0.95 → 0.90`, sometimes 0.85). Matches the H100/H200/B200 NVIDIA pattern. Smallest blast radius. -2. **Disable the profiler entirely** for cases where lowering isn't enough: `export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` before `vllm serve`. Matches `benchmarks/single_node/fixed_seq_len/deprecated/kimik2.5_fp4_b200.sh`. +2. **Disable the profiler entirely** for cases where lowering isn't enough: `export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` before `vllm serve`. Seen on: #1395 (kimik2.5-fp4-b200-vllm, needed env var), #1403 (gptoss-fp4-mi300x-vllm, needed 0.90), #1461 (dsv4-fp8-h200-vllm, needed 0.90). From 43cc8af7b5e0e21bd7d7d25e8baaab307022215b Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Sat, 26 Sep 2026 02:34:42 -0400 Subject: [PATCH 4/6] Delete configs/deprecated and drop remaining fixed_seq_len pointers - Delete configs/deprecated/{amd,nvidia}-master.yaml; nothing reads them. AGENTS.md, docs/configuration-procedures and the Claude review prompt now say retired entries are deleted, with git history and perf-changelog.yaml as the record. - Rewrite the add-model-hardware playbook and configuration procedures for srt-slurm recipes instead of bash scripts under fixed_seq_len/. - Point MODELS links, the KLAUD_DEBUG example and a deprecated multi-node comment away from the deleted folders. Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/commands/add-model-hardware.md | 76 +- .github/workflows/claude.yml | 2 +- AGENTS.md | 6 +- MODELS.md | 12 +- MODELS_zh.md | 12 +- .../amd_utils/deprecated/models_vllm.yaml | 4 +- configs/deprecated/amd-master.yaml | 3464 ---- configs/deprecated/nvidia-master.yaml | 16588 ---------------- docs/configuration-procedures.md | 10 +- docs/configuration-procedures_zh.md | 10 +- 10 files changed, 61 insertions(+), 20123 deletions(-) delete mode 100644 configs/deprecated/amd-master.yaml delete mode 100644 configs/deprecated/nvidia-master.yaml diff --git a/.claude/commands/add-model-hardware.md b/.claude/commands/add-model-hardware.md index 0100361404..02c1efc043 100644 --- a/.claude/commands/add-model-hardware.md +++ b/.claude/commands/add-model-hardware.md @@ -1,5 +1,5 @@ --- -description: Add a new model+hardware single-node benchmark recipe (script + master-config entry + perf-changelog + launcher routing), open a [Klaud Cold] PR, label full-sweep-fail-fast, and monitor CI +description: Add a new model+hardware single-node benchmark recipe (srt-slurm recipe + master-config entry + perf-changelog), open a [Klaud Cold] PR, label full-sweep-fail-fast, and monitor CI argument-hint: [recipes-link] [draft-model-link] [mtp] --- @@ -40,45 +40,43 @@ Check `MODELS.md` before choosing a model, scenario, or precision. Do not reintr **A. In-codebase research (primary because this repo is the source of truth):** ```bash -# similar benchmark scripts: same model on other SKUs, AND same SKU on other models -ls benchmarks/single_node/fixed_seq_len/_*.sh benchmarks/single_node/fixed_seq_len/*_*.sh +# similar srt-slurm recipes: same model on other SKUs, AND same SKU on other models +ls -d benchmarks/single_node/srt-slurm-recipes//*/* benchmarks/single_node/srt-slurm-recipes/*/*/-* # similar master-config entries (search spaces, image, parallelism), this model + analogues grep -nE "-|.*--" configs/{nvidia,amd}-master.yaml -# the runner launcher for this SKU (script-name routing, env, mounts, MODEL_PATH rewrite) -sed -n '1,80p' runners/launch_*.sh -# shared helpers the scripts rely on -grep -nE "run_benchmark_serving|setup_eval_context|wait_for_server_ready|start_gpu_monitor" benchmarks/benchmark_lib.sh +# how a matrix point selects exactly one recipe variant (TP, GPUs, CONC, KV_OFFLOADING, image) +sed -n '/def select_recipe/,/^def runtime_arguments/p' infx/srt_slurm/single_node.py ``` -- **Read multiple sibling scripts** end-to-end for the exact env vars and serve shape (`VLLM_*`, +- **Read multiple sibling recipes** end-to-end for the exact engine args, env vars and serve shape (`VLLM_*`, `SGLANG_*`, device mapping, download/cache handling, `--enforce-eager` vs graph capture, - KV-cache dtype, attention/MoE backend, parsers). These are the truth for each runner. + KV-cache dtype, attention/MoE backend, parsers, `setup_script`). These are the truth for each runner. - **Compare several master-config search spaces** (e.g. `dsr1`, `qwen3.5`, the same model on a sibling SKU) to choose `{tp, ep, dp-attn} × concurrency` combos that fit *this* hardware's memory. Small-memory SKUs like h100/mi300x go TP8-only, while bigger SKUs add tp4/tp2/DEP. - **Internalize the fixed-seq-len nuances from the existing configs**: `8k1k` runs do **not** need the full `MAX_MODEL_LEN` (the matrix supplies `isl + osl + slack`), and graph-capture batch sizes are scaled to concurrency/scenario (and spec-token count for MTP), not maxed. - Copy how sibling scripts/configs already do it. + Copy how sibling recipes/configs already do it. **B. External research (confirm against upstream guidance):** - **`WebFetch` the model-link card + its `config.json`** → confirm `model:` id, precision, max context, architecture, spec-decode fields (`num_mtp_modules`, etc.). - **`WebFetch` the recipes-link** (if given) → canonical `vllm serve` flags + troubleshooting. - Reconcile with what the sibling scripts do. If they conflict, follow the repo and note why. + Reconcile with what the sibling recipes do. If they conflict, follow the repo and note why. - If a **draft-model-link** is given, note its id for `--speculative-config` and check the card for method (`eagle3` vs native `mtp`) and recommended token count. - Pick the **image tag** from the sibling's master-config entry (or recipes page) and **verify it exists** on the registry before using it. -This research directly feeds Step 2 (script flags/env) and Step 3 (search space). +This research directly feeds Step 2 (recipe args/env) and Step 3 (search space). -## What you're producing (4–5 files) +## What you're producing (3 files) -1. `benchmarks/single_node/fixed_seq_len/__[_][_mtp].sh` +1. `benchmarks/single_node/srt-slurm-recipes///-[-mtp]/8k1k.yaml` + (an `agentic.yaml` beside it for AgentX) 2. an entry in either master config, **`configs/nvidia-master.yaml`** (b*/h*/gb* SKUs) or - **`configs/amd-master.yaml`** (mi* SKUs) + **`configs/amd-master.yaml`** (mi* SKUs), with `srt-recipe:` on every search-space row 3. a `perf-changelog.yaml` entry (this diff vs main is what selects the sweep) -4. (if missing) `SPEC_SUFFIX`/framework-suffix routing in `runners/launch_*.sh` ## Step 1 — branch + find the sibling to copy @@ -86,17 +84,20 @@ This research directly feeds Step 2 (script flags/env) and Step 3 (search space) git checkout main && git pull origin main git checkout -b feat/-[-mtp]-dayzero # nearest sibling: same model other SKU, or same SKU other model -ls benchmarks/single_node/fixed_seq_len/_* # same model, other hardware -ls benchmarks/single_node/fixed_seq_len/*_*.sh # same hardware, other model +ls -d benchmarks/single_node/srt-slurm-recipes//*/* # same model, other hardware +ls -d benchmarks/single_node/srt-slurm-recipes/*/*/-* # same hardware, other model grep -n "--" configs/{nvidia,amd}-master.yaml ``` -Read the closest sibling script **and** its master-config entry. Copy their flag shapes and +Read the closest sibling recipe **and** its master-config entry. Copy their flag shapes and search-space structure rather than inventing. The right model is "same model on a sibling SKU, adjusted for this hardware's quirks." -## Step 2 — write the benchmark script +## Step 2 — write the srt-slurm recipe -Copy the sibling script and adjust. Things that vary and must be checked against the sibling / +Copy the sibling recipe (`base:` plus one `override_*` variant per matrix point) and adjust. +Engine flags are `roles.agg.args` keys without the leading `--`; env vars go in +`roles.agg.env`; each variant names its `CONC` (and `KV_OFFLOADING` for AgentX) in +`benchmark.env`, and `model.container` must equal the master-config `image`. Things that vary and must be checked against the sibling / the model's `recipes.vllm.ai` page: - **Mandatory model flags** (carry from the sibling): block size, parser flags (`--tool-call-parser` / `--reasoning-parser`), `--language-model-only` for text-only sweeps, @@ -113,7 +114,7 @@ the model's `recipes.vllm.ai` page: - **Memory headroom.** Bigger checkpoints constrain TP/EP. If the sibling on a smaller-memory SKU is TP8-only (e.g. h100), match that. -Validate as you go: `bash -n