feat: validate BFCL through native APIs and guard eval parity / 通过原生 API 验证 BFCL 并保障跨硬件评估一致性

[adibarra]

Summary

Keep BFCL working through native APIs on supported backends, remove runtime framework-source patches, and validate AMD/NVIDIA evaluation coverage. TRT BFCL remains disabled while MiniMax B200/B300/GB200 moves to the official 1.3.0rc28.dev202609220000 nightly; vendor validators remain enabled and GPU qualification of the new image is pending. Draft: selected suites pass across both vendors, but complete framework, offload, concurrency and multi-node parity remains unfinished.

Changes

• Remove TRT chat-schema/executor-metrics patches and both MiniMax vLLM CPU-offload patch callers plus their shared helper. Use native tool parsers; preserve separate Chat Completions and Responses identities. Do not rewrite requests or synthesize token usage.
• Add stock BFCL Responses smoke, exact selected-case manifests, failure artifacts and API provenance. Retain upstream Kimi's 20-step limit. Request budgets are 180 seconds for MiniMax/smoke and 600 seconds for full Kimi, with two SDK retries.
• Upgrade MiniMax B200/B300 vLLM to official af1c0149, containing native mixed-layout CPU-offload and MSA fixes. Use native FlashInfer draft attention on both SKUs. Correct CPU budget conversion from decimal GB to bytes; preserve the production grid.
• Update MiniMax B200/B300 TRT and GB200 Dynamo-TRT from 1.3.0rc23.post1 to official nightly 1.3.0rc28.dev202609220000. Synchronize all seven GB200 recipes, use native block_reuse_config.policy: per_conversation, and remove the unsupported fuse_qkv_index_projection field. BFCL is disabled for trt and dynamo-trt, including explicit overrides: eval-only jobs skip before GPU allocation, and throughput jobs omit BFCL. Vendor validators stay enabled. Remove the experimental TRT nightly probe; runtime patches remain removed. The adapters and suites stay intact, and utils/evals/EVALS.md documents the small workflow/runtime guard removal needed to re-enable TRT after validating a stock image.
• Preserve main’s Kimi MI355X nightly-rocm100-af1c0149 image, native vllm-simple production offload, ROCM_AITER_MLA draft attention and concurrency settings. Keep the branch’s native BFCL diagnostics and remove runtime dependency installation from the optional LMCache path. Historical LMCache and old-image BFCL failures do not establish results for this updated recipe.
• Add experimental resident Kimi B300 and MiniMax MI300X probes. Production MI300X remains unchanged because the replacement's LMCache arm is unvalidated.
• Add a native Kimi CPU-cache restoration check. Add a separate bfcl_kimi_diagnostic for multiple_0–multiple_15: temperature 0.001, 16 threads, 60-second request timeout, two retries, 600-second suite bound. Only this diagnostic enables native NaN counters and archives final metrics, including after failure. Full-suite sampling remains unchanged.
• Use the job token for public checkout/run statistics after shared credential failures. Merge current main while preserving every historical changelog byte.

Historical TRT skip validation

On source 63e450258: 129 eval-dispatch tests pass, including explicit TRT/Dynamo-TRT skips and continued vendor/non-TRT evaluation dispatch. Both reusable workflows pass actionlint; shell syntax, changelog generation and diff checks pass. The 22 generated TRT eval rows retain rc23.post1 and the full MiniMax vendor suite. Workflow guard verification passed: both explicit TRT BFCL jobs were skipped before GPU allocation, and the workflow completed successfully. This verifies skip behavior, not GPU evaluation or throughput.

Verified BFCL coverage

All rows below completed every expected case with no integration error. The MiniMax vLLM rows use 8 BFCL threads and requested temperature 0.001. c identifies the serving recipe's concurrency. These are selected BFCL suites, not the complete leaderboard; zero thresholds establish diagnostic completion, not a quality acceptance baseline.

Model / framework / SKU / recipe	Result	Run
MiniMax-M3-NVFP4 / vLLM af1 / B200 / TP4 resident c1	898/1,000	35154685442
MiniMax-M3-NVFP4 / vLLM af1 / B200 / TP4 native lazy CPU offload c15, corrected 1.082 TB budget	895/1,000	35244609202
MiniMax-M3-NVFP4 / vLLM af1 / B300 / TP4 native lazy CPU offload c30	898/1,000	35171313858
MiniMax-M3-MXFP8 / vLLM v0.27.1 / H100 / TP8 c1, CPU weight offload, resident KV	899/1,000	35173594641
MiniMax-M3-MXFP8 / vLLM d910 / H200 / TP8 resident c1	895/1,000	35171252969
MiniMax-M3-MXFP8 / vLLM af1 / MI300X / TP8 resident c2, experimental image	899/1,000	35171259984
MiniMax-M3-MXFP8 / vLLM v0.27.1 / MI325X / TP8 resident c1	892/1,000	35168911491
MiniMax-M3-MXFP4 / vLLM 2a02 / MI355X / TP4 resident c4	877/1,000	35158177585
MiniMax-M3 / ATOM / MI355X / TP4 resident c8	703/1,000	35031849044
Kimi-K3 / vLLM ROCm100 af1 / MI355X / TP8 DCP1 resident c1, AITER DSpark7	1,039/1,240	35253995642
Kimi-K3 / vLLM 3696c77 / B300 / TP8 DCP8 resident c8, DSpark7	1,042/1,240	35176168999
Kimi-K3 / ATOM / MI355X / TP8 resident c4	1,038/1,240	35134517859

Kimi B300: 916/1,000 single-turn and 126/240 multi-turn; ATOM: 911/1,000 and 127/240. Both use 16 BFCL threads, temperature 0.001 and the stock 20-step limit. Checkpoint formats, images, backends and topologies differ; score differences are not hardware-only comparisons.

The updated AMD Kimi production recipe completed 1,240/1,240 unique records matching the exact case manifest: 910/1,000 single-turn + 129/240 multi-turn = 1,039/1,240 (83.79%). Source e2213f5c2, official nightly-rocm100-af1c0149, TP8 DCP1 resident c1, native AITER DSpark7; server max-num-seqs=2, 16 BFCL threads. No recorded inference/integration failures, request-timeout or SDK-retry messages, server ERROR-level lines, tracebacks or OOM. Requested temperature 0.001 is clamped by the stock server to 0.01. Generation took 92m06s; the GPU job including startup/cleanup took 99m44s, excluding queue. There are 201 quality failures; 881 per-turn decode messages are not 881 unique failed cases. This clears completion for the updated resident recipe, not native CPU offload or a hardware-only timing/quality comparison.

The new B200 offload run contains exactly 1,000 unique records with no inference/integration failures: simple 361/400, multiple 181/200, parallel 177/200, parallel-multiple 176/200. The stock server logged one MiniMax tool-parser exception; this is not an entirely error-free quality run. External-cache-hit logs stayed at zero, so this BFCL result alone does not prove CPU restoration.

Additional native evidence:

• Kimi MI355X/vLLM bounded resident diagnostic: 12/16 correct, all expected cases completed without integration errors; native corrupted-request counter was zero and no server error/OOM was logged. This short old-image diagnostic does not clear the earlier 1,240-case failure or establish its cause.
• Kimi MI355X/vLLM TP8 DCP1 eager CPU offload c4: 4/4 smoke plus restoration check. An 8,464-token prompt had zero cached tokens cold and 6,144 after clearing GPU cache; external-hit delta was also 6,144. Identical 32-token greedy output and finish reason. This validates one prefix, not all cache behavior.
• MiniMax B200/vLLM TP4 lazy CPU offload c15: 4/4 smoke with corrected 1.082 TB aggregate budget. Native served-model name is /scratch/models/MiniMax-M3-NVFP4; no checkpoint-hash equivalence claim.
• MiniMax B300/vLLM TP4 lazy CPU offload c30 performance canary: 1,542 profiled completed requests, zero recorded request errors, 4,460,672 native external-cache-hit tokens, and valid power telemetry on all four GPUs. Output throughput was 1,147.85 tok/s total (286.96/GPU). Limitation: harness teardown logged a grace-period timeout, one cancelled credit and remaining session-tree state. This 20-minute agentx-fast run proves native lazy CPU reads, but does not establish clean teardown, a performance gain, or official artifact-reuse eligibility.

Failures and remaining blockers

• Historical Kimi MI355X/vLLM full-suite failures: both resident and native eager CPU offload hit the four-hour bound. Each saved 114 cases, including 112 request timeouts. Engines kept decoding without a logged crash/OOM. There is no valid quality score. An earlier identical resident recipe completed nearly the entire suite, so neither temperature nor offload alone is an established cause. The 16-case native-counter probe subsequently completed with 12 correct and zero recorded NaNs. The merged ROCm100/AITER resident recipe now completes all 1,240 cases as reported above. This verifies the updated recipe without establishing which earlier difference caused the failures; the full native CPU-offload path remains unvalidated.
• TRT BFCL is intentionally excluded from this release. The official rc28 nightly experiment passed image import but failed during PyTorch compilation of a native MiniMax KV-cache call before BFCL started (0 cases). The current branch now pins official nightly 1.3.0rc28.dev202609220000, whose GPU behavior remains unverified. The separate upstream source build passed four Responses smoke cases only with eager prefill and EAGLE3 disabled; that is not production EAGLE3 or full-suite qualification. No framework-source patch is added to force BFCL compatibility. Generic Responses adapter support is retained for other backends. Native performance-metrics behavior still requires fresh throughput validation.
• Optional Kimi LMCache (historical failure; current main uses native vllm-simple): native failure is Unsupported kv_layout: none, matching upstream issue. The bundled revision lacks the upstream fix; published ROCm wheels target a different PyTorch ABI. DCP8 is unvalidated.
• Kimi B300 Mooncake: no active RDMA rail on two nodes. Resident B300 success does not validate Mooncake.

Active validation

Run	Scope / immutable source
35244612071	MiniMax B300 native offload c30, standard warmup and one-hour AgentX profile; 57ce9acdf.

Pending runs are not passing evidence. No production ingestion, PR merge or official sweep reuse is claimed.

Local validation and follow-up

Main is merged through b43fe5e10 in 905d1d624. The latest sync resolved overlapping CI token changes by retaining main’s native job-token behavior and this branch’s BFCL suite options. The complete main changelog is preserved byte-for-byte, with the existing branch entry appended unchanged. Local CI selection: 1,804 passed, 2 skipped, 187 subtests passed; changelog/matrix validation and diff checks passed. Actionlint passed with the pre-existing unsupported $/ workflow-call syntax warning excluded. GitHub CI, Zizmor, changelog and reuse-gate all passed on 905d1d624. The PR remains draft and mergeable. GPU results above retain their original immutable source revisions; the sync adds no Kimi/MiniMax recipe changes.

Remaining work: inspect active artifacts, validate other framework/concurrency/multi-node paths and changed production throughput against a matched baseline. Historical AMD vendor-result publication needs provenance review. Other SPEED-Bench/lm-eval/SWE-bench runtime modifications remain separate cleanup; the repository is not yet entirely patch-free.

Vendor coverage follow-up: full Kimi and MiniMax vendor selection is now merged into main and included here. Three live cells completed with quality misses; NVIDIA/Kimi remains queued. Vendor suites are separate from the BFCL results above.

Historical smoke interpretation: the Kimi 1.0 result represents one schema case × streaming/nonstreaming = 2/2 checks, using required strict tool calling. Full vendor coverage is 204 unique cases / 408 checks. Current frontend source preserves n_eff in ingestion but drops counts in the chart projection/table. Display unique cases, check count and smoke/full identity without adding duplicate cases across retries. No frontend or production-data edits have been made.

AI model disclosure

Codex assisted with the implementation, diagnostics, and branch synchronization. The exact underlying model/version is not exposed in the current runtime; the model versions of earlier delegated agents have not been verified.

中文

概述与改动

保留 BFCL，通过原生 API 验证 AMD/NVIDIA 评估路径，移除框架运行时源码补丁。PR 保持草稿：两家硬件上的选定套件已经完成，但完整框架、offload、并发及多节点覆盖仍未完成。

• 删除 TRT 请求结构、执行器指标补丁，以及 MiniMax 两个 vLLM CPU offload 补丁调用和共享脚本；使用原生工具解析器，区分 Chat Completions 与 Responses，不改写请求或补造用量。
• 新增上游 Responses smoke、精确用例清单、失败产物及 API 来源信息；Kimi 保留上游 20 步限制。MiniMax/smoke 请求时限为 180 秒，完整 Kimi 为 600 秒，均最多两次 SDK 重试。
• MiniMax B200/B300 vLLM 升级官方 af1c0149，采用原生混合布局 CPU offload、MSA 修复及 FlashInfer 草稿 attention。CPU 预算按十进制 GB 换算为字节，正式扫描网格不变。
• MiniMax B200/B300 TRT 与 GB200 Dynamo-TRT 从 1.3.0rc23.post1 更新至官方 nightly 1.3.0rc28.dev202609220000，同步全部七份 GB200 配方，使用原生 block_reuse_config.policy: per_conversation，移除已不支持的 fuse_qkv_index_projection 字段。新镜像的 GPU 验证尚未完成。trt 和 dynamo-trt 禁用 BFCL，包括显式覆盖：仅评估任务在分配 GPU 前跳过，吞吐量任务不执行 BFCL；厂商评估继续启用。移除 TRT nightly 实验配置，不恢复运行时补丁。现有适配器和套件完整保留；utils/evals/EVALS.md 说明原生镜像验证通过后，重新启用 TRT 时需移除的少量工作流和运行时分发限制。Kimi MI355X 保留 main 的 nightly-rocm100-af1c0149 镜像、正式配置中的原生 vllm-simple offload、ROCM_AITER_MLA 草稿 attention 及并发设置，同时保留本分支的原生 BFCL 诊断，并移除可选 LMCache 路径的运行时依赖安装。旧镜像和 LMCache 的历史失败不能代表此更新配置的结果。
• 新增 Kimi B300、MiniMax MI300X 实验配置；MI300X 正式配置暂未变更，其 LMCache 路径尚待验证。
• Kimi CPU smoke 加入原生缓存恢复检查。独立的 bfcl_kimi_diagnostic 选择 multiple_0 至 multiple_15，temperature 0.001、16 个线程、60 秒请求时限、两次重试、600 秒总时限；仅该诊断启用原生 NaN 计数，失败后也保存指标，完整套件的采样不变。
• 公共仓库检出和只读统计改用 GitHub 任务令牌；合入最新 main，完整保留历史性能日志字节。

TRT 跳过行为的历史验证

源码 63e450258 的 129 项评估分发测试通过，覆盖 TRT/Dynamo-TRT 显式跳过 BFCL，以及厂商评估和其他后端 BFCL 继续执行。两个可复用工作流的 actionlint、Shell 语法、性能日志矩阵生成和差异检查通过。生成的 22 条 TRT 评估配置均保留 rc23.post1 与完整 MiniMax 厂商套件。工作流跳过验证通过：两条显式 TRT BFCL 任务均在分配 GPU 前跳过，工作流整体成功。这验证的是跳过行为，不是 GPU 评估或吞吐量。

已核验结果

上表各行全部预期用例均完成且无集成错误。七种 MiniMax vLLM SKU 的选定 1,000 用例结果分别为：B200 NVFP4 898、B300 NVFP4 原生 CPU offload 898、H100 MXFP8 899、H200 MXFP8 895、MI300X MXFP8 899、MI325X MXFP8 892、MI355X MXFP4 877。这些 vLLM 套件使用 8 个 BFCL 线程、temperature 0.001，表中的 c 表示服务配方并发。MiniMax MI355X/ATOM 为 703/1,000。

新增 B200 原生 lazy CPU offload c15 结果为 895/1,000，已修正为 1.082 TB 总预算。原始产物包含 1,000 条唯一记录，无请求/集成失败；四类分别为 361/400、181/200、177/200、176/200。原生服务器记录了一次工具解析异常，因此不能称为完全无错误的质量验证。外部缓存命中日志为零，该 BFCL 运行本身不证明 CPU 缓存恢复。Kimi MI355X 的旧镜像 16 用例原生计数诊断完成，12/16 正确、无集成错误、NaN 计数为零，服务器未记录崩溃/OOM；该小规模诊断不能证明完整套件已修复。

更新后的 AMD Kimi 正式配方已完成 1,240/1,240 条唯一记录，与用例清单逐项一致：单轮 910/1,000，多轮 129/240，合计 1,039/1,240（83.79%）。源码为 e2213f5c2，官方 nightly-rocm100-af1c0149 镜像，TP8 DCP1 驻留 c1、原生 AITER DSpark7，服务器 max-num-seqs=2，16 个 BFCL 线程。未记录推理或集成失败、请求超时、SDK 重试、服务器 ERROR 级日志、异常堆栈或 OOM。请求 temperature 0.001 被原生服务器调整至 0.01。生成耗时 92 分 06 秒，GPU 任务含启动和清理为 99 分 44 秒，均不含排队。仍有 201 个质量未通过用例；881 条逐轮解码提示不等于 881 个独立失败用例。本结果只验证更新后的驻留配方，不证明原生 CPU offload 完整覆盖，也不能将耗时或分数差异单独归因于硬件。

Kimi B300/vLLM TP8 DCP8 驻留 c8 为 1,042/1,240（单轮 916/1,000，多轮 126/240）；MI355X/ATOM TP8 c4 为 1,038/1,240（单轮 911/1,000，多轮 127/240）。均使用 16 个 BFCL 线程、temperature 0.001 和上游 20 步限制。模型格式、镜像、后端和拓扑不同，不能将差异单独归因于硬件。选定套件不是整个 BFCL；零阈值用于诊断完成，不代表质量基线验收。

Kimi MI355X 原生 eager CPU offload c4 的 4/4 smoke 和一个前缀恢复检查通过：8,464 个输入 token，冷请求缓存为零，清空 GPU 缓存后命中 6,144 token，外部命中增量相同，32-token 贪心输出与终止原因一致。B200 原生 lazy CPU offload c15 在修正为 1.082 TB 总预算后 4/4 smoke 通过；服务使用本地模型路径，未验证 checkpoint 哈希等价性。

B300 20 分钟性能 canary 完成 1,542 个测量请求，记录中无请求错误，原生 CPU 缓存命中 4,460,672 token，四个 GPU 的功耗遥测均有效，总输出吞吐量 1,147.85 tok/s（286.96/GPU）。但 harness 收尾出现宽限期超时、一个取消的 credit 和剩余会话树状态，因此只能证明原生 lazy CPU 读取已实际发生，不能宣称收尾完全正常、性能提升或正式产物可复用。

失败与活跃验证

历史 Kimi MI355X/vLLM 驻留和原生 CPU offload 的完整套件均触及四小时上限，各保存 114 条记录，其中 112 条请求超时。引擎仍在解码，未记录崩溃/OOM，没有有效质量分数。此前相同驻留配方几乎完成全部用例，因此尚不能归因于 temperature 或 offload；16 用例诊断随后完成且 NaN 计数为零。合并后的 ROCm100/AITER 驻留配方现已完成全部 1,240 个用例，结果见上文；这证明新配方可完成评估，但尚未定位旧配方失败的具体原因，原生 CPU offload 完整路径仍待验证。

本次发布明确排除 TRT BFCL。官方 rc28 nightly 实验通过镜像导入，但在 PyTorch 编译原生 MiniMax KV-cache 调用时失败，BFCL 尚未启动（0 个用例）。当前分支改用官方 nightly 1.3.0rc28.dev202609220000，该镜像的 GPU 行为仍待验证。独立的上游源码构建仅在 eager 预填充且关闭 EAGLE3 时通过四用例 Responses smoke，不能证明正式 EAGLE3 配方或完整套件已通过；不通过框架源码补丁强行兼容 BFCL。通用 Responses 适配器保留供其他后端使用。移除指标补丁后的原生性能行为仍需重新验证。可选 Kimi LMCache 的历史失败来自不兼容 API，已发布外部 ROCm wheel 的 PyTorch ABI 不匹配，DCP8 未验证；当前 main 正式配置已使用原生 vllm-simple。B300 Mooncake 在两个节点没有活动 RDMA rail，驻留路径通过不能替代此项验证。相关失败与上游源码链接见英文部分。

活跃任务：35244612071 为 B300 标准预热、一小时性能验证，源码为 57ce9acdf。运行中不代表通过，未合并 PR、导入生产数据或宣称正式扫描复用。

验证与后续

main 已合入至 b43fe5e10，合并提交为 905d1d624。本次同步解决 CI 令牌配置的重叠改动，保留 main 的原生任务令牌行为及本分支的 BFCL 套件选项。完整保留 main 性能日志的所有字节，并将本分支原有条目原样追加至末尾。1,804 项测试、187 项子测试通过，2 项跳过；性能日志/矩阵验证与差异检查通过。Actionlint 在排除其对既有 $/ 工作流调用语法的兼容性告警后通过。905d1d624 的 GitHub Tests、Lint、Zizmor、性能日志及复用检查全部通过，PR 仍为无冲突草稿。上文 GPU 结果保留各自实际运行的不可变源码版本；本次同步没有新增 Kimi/MiniMax 配方变更。

继续核验活跃任务、其他框架/并发/多节点路径，以及相同条件下的正式吞吐量基线。历史 AMD 厂商结果恢复仍需来源核验；其他 SPEED-Bench、lm-eval、SWE-bench 运行时修改需另行清理，不能宣称仓库已完全无补丁。

Kimi 和 MiniMax 默认完整厂商评估的独立改动已合入 main，本分支同步包含该改动。四个实机组合中三个已完成，但有质量未通过项，NVIDIA/Kimi 仍在排队。厂商套件与上面的 BFCL 结果分别报告。

Kimi 厂商评估的 1.0 是 一个 schema 用例 × 流式/非流式 = 2/2 检查，采用强制严格工具调用。完整套件有 204 个独立用例、408 次检查。当前前端入库保留 n_eff，但图表投影和表格丢失数量；需显示独立用例、检查数及 smoke/full 身份，避免重试重复累加。尚未修改前端或生产数据。

AI 模型说明

Codex 参与了实现、诊断及分支同步。当前运行环境未提供底层模型的准确名称和版本；此前参与工作的子代理模型版本尚未核实。