feat(srt): run AgentX on native srt-slurm - #3428
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
|
Claude encountered an error after 0s —— View job I'll analyze this and get back to you. |
eb81460 to
e151d6f
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline findings, I also checked the -n "${SRT_RECIPE:-}" pattern added to all 12 launchers against AGENTS.md's "no fallback defaults" rule — it's only used as an existence test in a boolean condition (not a silently-substituted default value), and each branch still calls check_env_vars SRT_RECIPE before use, so it doesn't violate that convention.
Extended reasoning...
The diff wires AgentX single-node jobs through the native srt-slurm connector (matrix schema/generator, infx/srt_slurm/single_node.py branching, 12 launcher routing conditions, a new recipe/config pair, and a new bash replay client) with no auth/crypto/permissions surface, but it does touch the shared connector and recipe pairing logic across many files. Two confirmed findings (a duplicated --server-metrics flag corrupting metrics collection in the new AgentX client, and a missing perf-changelog.yaml entry for the new recipe) are being posted as inline comments, so this is a findings-present run where defer is used only to record one additional item I checked and ruled out.
Findings marked 🟡 are optional suggestions and need no follow-up push.
Additional findings (no inline location):
-
🟡
perf-changelog.yaml— This PR adds a new recipe (benchmarks/single_node/srt-slurm-recipes/dsv41flash/sglang/h200-fp4-mtp/agentic.yaml) and repoints dsv41flash-fp4-h200-sglang-agentic-dspark in configs/nvidia-master.yaml at it, but perf-changelog.yaml has no new entry. AGENTS.md (via CLAUDE.md) requires "every recipe addition or modification" to get a new append-only perf-changelog.yaml entry, and this PR's changed-file list omits it. Fix: append a perf-changelog.yaml entry (preserving existing bytes, appending only at the tail) documenting the new agentic.yaml recipe and its TP4/TP8 admission and warmup-grace-period tuning before merge.Why this was flagged
AGENTS.md:124 states every recipe addition or modification requires a new perf-changelog.yaml entry, appended at the tail without disturbing existing bytes. This PR (configs/nvidia-master.yaml:8547-8548, benchmarks/single_node/srt-slurm-recipes/dsv41flash/sglang/h200-fp4-mtp/agentic.yaml) adds a brand-new recipe and search-space entry pointing to it. perf-changelog.yaml is not among the 19 changed files, so no entry documents this recipe's admission/warmup tuning choices. On base, every prior recipe change (e.g. PR #3256, #3406) carries a matching changelog entry; maintainers and future tuners lose the documented rationale trail for this recipe's specific settings.
Verification: nit. The candidate correctly identifies a missing required deliverable. AGENTS.md (delegated from the repo-root CLAUDE.md) states as a non-negotiable benchmark invariant: "Every change that can affect benchmark performance and every recipe addition or modification requires a new
perf-changelog.yamlentry. The file is append-only and byte-sensitive." This PR adds a brand-new recipe file…
| export AGENTIC_OUTPUT_DIR="$(dirname "$RESULT_DIR")" | ||
| # Worker metrics, not the router's, carry the engine series AIPerf must record. | ||
| AIPERF_SERVER_METRICS_URLS=$(sed -E 's#([^,]+)#http://\1/metrics#g' <<< "$SRT_AGG_ENDPOINTS") | ||
| export AIPERF_SERVER_METRICS_URLS | ||
|
|
||
| mkdir -p "$RESULT_DIR" | ||
| resolve_trace_source | ||
| install_agentic_deps | ||
| build_replay_cmd "$RESULT_DIR" | ||
| REPLAY_CMD+=" --server-metrics ${AIPERF_SERVER_METRICS_URLS}" |
There was a problem hiding this comment.
🔴 Single-node AgentX runs on native srt-slurm get a corrupted --server-metrics list, an outcome fixed-sequence and multi-node AgentX jobs do not hit. srt_agentic.sh exports AIPERF_SERVER_METRICS_URLS before calling build_replay_cmd (line 25), so benchmark_lib.sh:3340-3352 already appends a correctly space-split '--server-metrics url1 url2 ...'. Line 26 then appends a second, redundant '--server-metrics' with the raw comma-joined string as one unsplit argument. Fix: drop the manual append at line 26 and rely on build_replay_cmd's existing AIPERF_SERVER_METRICS_URLS handling, which already splits on commas for every caller.
Why this was flagged
Trigger: any single-node AgentX point routed to native srt-slurm (SRT_RECIPE set) with SRT_AGG_ENDPOINTS containing one or more worker endpoints, reached via runners/*.sh -> slurm_utils.sh apply_srt_recipe -> srt_agentic.sh. build_replay_cmd (benchmark_lib.sh:3340-3352) already appends '--server-metrics' followed by each URL as a separate token when AIPERF_SERVER_METRICS_URLS is non-empty. srt_agentic.sh:26 appends a second '--server-metrics' flag whose value is the whole comma-joined string as one token (no spaces to split on). Depending on how the aiperf CLI parses a repeated flag, the final effective value becomes either a single malformed URL like 'http://a/metrics,http://b/metrics' or a duplicated/garbled list, breaking the worker engine-metrics capture the comment at line 18 says is required. No existing check catches this since REPLAY_CMD is an unquoted string executed via 'run_server_client $REPLAY_CMD' (benchmark_lib.sh:3486).
Verification: normal (for the multi-endpoint case; benign redundancy for a single endpoint). The candidate's core code claim holds. In benchmarks/single_node/srt_agentic.sh, AIPERF_SERVER_METRICS_URLS is exported at line 20 (built from SRT_AGG_ENDPOINTS via the sed at line 19) BEFORE build_replay_cmd is called at line 25. build_replay_cmd (benchmark_lib.sh:3340-3352) sees the non-empty var and appends a…
090ca53 to
169d12e
Compare
169d12e to
1235ab6
Compare
Single-node AgentX points with an srt-recipe now take the native single-node path fixed-sequence points already use. The adapter accepts AgentX points and DSpark speculation. Single- and multi-node AgentX share one client, benchmarks/srt_agentic.sh (moved from benchmarks/multi_node/agentic_srt.sh), which replays one CONC point or a CONC_LIST batch against the srt-slurm frontend and reads engine metrics from the workers behind a router frontend. The H200 DeepSeek-V4.1-Flash SGLang config is the first one ported.
1235ab6 to
4fa2a2f
Compare
…en3.5 fixed-sequence recipe srt-slurm parses recipes strictly and rejected the repeated disable-cuda-graph key. Removing the unported MI355X Qwen3.5 AgentX recipe had also deleted the fixed-sequence recipe beside it.
vLLM points validate their topology as tensor x data parallel GPUs, with DP attention as data-parallel ranks and expert parallelism as enable-expert-parallel, and eval-only runs set max-model-len.
…ient-side Several legacy AMD AgentX scripts appended --apply-chat-template to the replay command. AIPERF_APPLY_CHAT_TEMPLATE=true in a recipe's benchmark env keeps that behavior on the shared srt-slurm client.
…V4.1 Flash SGLang there GB launchers submit recipe points through launch_srt_single_node with an aarch64 srt-slurm setup; the squash is imported on a compute tray first.
…benchmarks/single_node/agentic (#3460) #3428 moved single-node AgentX onto native srt-slurm recipes. Delete every script in benchmarks/single_node/agentic that no master config reaches any more. Keep the nine scripts still reached by search-space entries without an srt-recipe, plus the H100/H200 block32 installers and kernel_configs that the multi-node DeepSeek-V4.1-Flash recipes use. The README now marks the folder as legacy. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
3 participants
Step 3 of the srt-slurm migration: run all AgentX, single- and multi-node, on native srt-slurm with the design used for fixed-sequence (#3352, #2542, #3413, #3415). TileRT is out of scope.
Draft: 58 of the 62 active single-node AgentX configs now run natively, the multi-node recipes are consolidated, and MI355X DSV4 multi-node (#2555) is ported. Some smoke runs are still in flight; see Testing.
Background
srt-recipe,infx.srt_slurm.single_nodeselects the one recipe variant that matches the matrix point, and srt-slurm owns the server.benchmarks/multi_node/agentic_srt.sh.Design
The same principles as fixed-sequence:
override_*variants (one per point), not separate files or zip variants. Existing multi-node AgentX recipes split into one file per variation (for exampleqwen3.5/sglang/gb300-fp4/agentx/agg-tp2-c24-...-jid2530012.yaml) get consolidated the same way;setup_script, never cluster hooks.Changes
benchmarks/srt_agentic.sh(moved frombenchmarks/multi_node/agentic_srt.sh) serves both paths:CONCpoint, or each point of a multi-nodeCONC_LISTbatch with the existing drain between points;SRT_*_ENDPOINTS); Dynamo is unchanged;srt-recipe, AgentX included. Configs without one keep the legacy script, so they move over one at a time.srt-recipe, and the generator passes it through as it does for fixed-sequence.infx.srt_slurm.single_nodeaccepts AgentX points and DSpark speculation:/logs.synthetic_acceptance, as for multi-node AgentX.logs/agentictoresults/, where the workflow uploads the raw replay artifacts and the power window.SRT_MODEL_PATH, as the other launchers do. On B200 a model that isn't staged locally resolves ashf:<model>. GB200 and GB300 gain the native single-node path: they import the squash on a compute tray, then calllaunch_srt_single_nodewith an aarch64 srt-slurm setup (SRT_SETUP_ARCH).tensor-parallel-sizexdata-parallel-size= TP, DP attention asdata-parallel-size> 1, and expert parallelism asenable-expert-parallel. Eval-only runs setmax-model-len.AIPERF_APPLY_CHAT_TEMPLATE: 'true'where the legacy script passed--apply-chat-templateto the replay.provider/default_frequencytelemetry fields; they now usecollect_interval_ms: 1000;collector_join_timeout_seconds: 10, which the validator rejects; srt-slurm now derives it.dsv41flash-fp4-h200-sglang-agentic-dspark,qwen3.5-fp8-h200-sglang-agentic-mtp,qwen3.8next-fp8-h200-sglang-agentic-mtp, and DSV4.1 Flash DSpark on B200, B300, GB200 and GB300 (dsv41flash-fp4-{b200,b300,gb200,gb300}-sglang-agentic-dspark). The first is described below; the others follow the same pattern. The adapter also checksKV_OFFLOADINGandTOTAL_CPU_DRAM_GBwhen a variant declares them, which is how the Qwen3.5 GPU-KV and HiCache variants are selected.dsv41flash-fp4-h200-sglang-agentic-dspark(TP4 and TP8) moves todsv41flash/sglang/h200-fp4-mtp/agentic.yaml, translated fromdsv41flash_fp4_h200_sglang_mtp.sh.setup_script.Status of single-node AgentX configs
srt-recipeand binds through the adapter at every matrix point (perf and eval), except the legacy entries listed next.glm5.2-fp4-mi355x-atom(DCP4),kimik3-fp4-mi355x-atom(DCP8) andminimaxm3-fp4-mi355x-atom: srtctl's ATOM backend reserveskv-transfer-config, so an aggregate worker can't set the LMCache connector without an srt-slurm patch.kimik3-fp4-mi355x-vllmandminimaxm3-fp4-mi355x-vllm: their nightly images were pruned from Docker Hub. The legacy path still runs from node-local squashes. The native path can use a squash staged on/it-share(the launcher already looks there); the recipes are ready.qwen3.5-{fp4,fp8}-b300-sglang-agentic-power-abtargetcluster:b300-nv, which has no runners.uv python install(File existsat the runner's~/.cache/uv);minimaxm3-fp4-{b200,b300}-vllm: image pruned, no squash cache, and the legacy path fails the same way.Testing
All runs use
agentx-fast, single concurrency,--no-evals.Single-node, native (new path)
power_valid: 1Multi-node, existing srt-slurm path (shared client)
Single-node, ported batches. Each ported config was smoke-tested on its cluster with
agentx-fastat points covering each variant regime, and at least one eval-only point per engine family. Runs are listed in the batch comments on this PR.--time; the profile now takes the workflow limit.MAX_MODEL_LENrequirement, then double artifact staging); it now evaluates at native context with the workflow's framework, and artifacts upload./it-sharefor the MI355X native path.override_*variants; every selector resolves identically to the file it replaced (0 diffs), and smoke passed on B200, B300, GB200 and GB300.Known issues
telemetry requires infra.etcd_nats_dedicated_node: false).RECIPES.mdalready records this as a power compatibility blocker that should not be fixed by changing topology, so this PR leaves the placement as is.ProfileMetricCoverageError: at c1 with the 20-minuteagentx-fastprofile, only 5 to 7 long requests (median E2E 113 s) completed and none ran into the last 60 s. A full-duration rerun is pending to separate that from a server stall.Remaining work
amd_utilsAgentX path.qwen3.5-{fp4,fp8}-b300-sglang-agentic-power-abtargetcluster:b300-nv, which has no runners inconfigs/runners.yaml; they are left as is.