Repository navigation
fix(benchmark): default heartbeat and Turn sessions to resume - #7400
Conversation
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · gpt-6.1-sol · OpenAI · declaration_source=runtime_reported · reasoning_effort=max.
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
评审对象:#7400,完整 head 301c1e2e6ab974215a8db7be14c642de66753e58;不可变对照基线 cdbdc14ea3963e6f6aec4d574a1b88baff965132。
未发现本次变更的阻塞问题,结论 APPROVE。 此账号也是 PR 作者,故用 COMMENTED 发布完整独立评审结论;不冒充 GitHub 正式自批准。
动机
受影响的是以 heartbeat 或 Turn 运行多轮任务、希望规划和后续执行共享同一上下文的基准实验维护者。 旧版省略会话配置时每次另开对话,刚完成的规划不能自然出现在下一次执行的历史中;本次让新 heartbeat、Turn 默认复用同一 Goal/Agent 会话,并保留显式 fresh。 同一真实 Codex 回环脚本中,旧版省略配置的两次调用使用不同会话且第二次没有规划历史;候选复用同一会话并带回规划。显式 fresh 在两侧均产生独立会话。 本 PR 不启动付费模型或实验、不改评分和冻结试验,不启用差分上下文或 TurnEnvelope,也不把会话连通性当成模型采用、净收益或已安装资格。
共享历史会增加后续请求可能读取的上下文,因此这里证明的是连续性与隔离,不宣称更省 token。要比较新旧策略,仍须新建并固定独立试验、显式记录 fresh/resume,把规划、恢复和全部用量计入原预算。
改动思路
把省略配置的解析留在已有 Execution,Harbor、direct worker、LHTB renderer/preflight和shell复用同一owner;会话绑定、拒绝悄悄替换及恢复仍由既有 CodexAgentSession 负责。 本 PR 只改变新 heartbeat/Turn 的省略值,保留显式 fresh、其它模式、现有失败拒绝与固定试验边界;不另建会话存储、resume协议或控制循环。
省略值在 dataclass 完成解析,调用方交出重复的 fresh 默认。LHTB shell 也询问同一个 owner,而非另写一份模式判断。绑定和实际进程仍复用既有产品 owner;配置可接受不等于已有会话可恢复,损坏、身份或 profile 变化继续拒绝。
| 策略 | 基线实际 Codex | 当前 head 实际 Codex |
|---|---|---|
| 省略 context,规划后再执行 | 两个会话,执行看不到规划历史 | 同一会话,执行保留规划历史 |
| 显式 fresh | 两个独立会话 | 两个独立会话 |
| 其他三个模式省略 context | fresh | fresh |
具体改动
完整 base-to-head 差异共 13 个文件、+58/−18。
生产改动覆盖 Execution、Harbor、worker、session适配、LHTB shell、renderer、preflight和示例 YAML;两份文档公开新默认,三份现有测试扩展省略配置。只在 None 时推导;显式值、现有 token 集、会话存储格式和其它模式没有加入隐式兼容分支。
在读全 diff 前先读已接受的 docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.md,固定 spec_revision cdbdc14ea3963e6f6aec4d574a1b88baff965132。本 PR 新文档或对 RUNTIME 的修改属于待评审提案,不用它们证明自身正确:
- 5.3 Fairness and treatment integrity:已覆盖。Pin runner/model/budget and do not change implementations during a run; disclose changed harness behavior. 24-row immutable base/head comparison; explicitfresh parity; frozen input files remain unchanged; source-pinned new-trial boundary
- 7 Architecture and Ownership:已覆盖。Host harness owns model/tool loop; LoopX owns provider-neutral Goal/Todo/effect/authority and settlement. Real native Codex continuation, existing scoped store/rejection/retry tests and unchanged canonical owner blobs
- 6.2 Efficiency:本次不作资格。Measure full cost and coverage separately from useful results; preserve failures and unknowns. Default continuity is verified, but no real model/solver or complete cost comparison was run; longer retained history is a disclosed possible cost, not a proven efficiency gain.
- 11.2 Engineering readiness stages:已覆盖。Keep synthetic runtime integration separate from matched-study and product qualification. Real transport uses a local scripted endpoint; no Harbor job/Docker study/installed promotion claimed;4 Linux process-group cases remain unrun.
关键代码讲解
- class Execution::唯一解析模式与省略会话配置;fresh统一默认 -> heartbeat/turn省略时resume。只解析None,显式fresh/resume仍验证;其它模式保持fresh。调用方为 四条既有生产入口,结果进入 Harbor/worker/renderer/preflight;失败由 非法mode/context或Turn缺独立validator先拒绝。
- def run_once:通过真实driver运行解析后的策略;raw env省略不再自行补fresh。Execution先校验;session/phase/deadline仍由原owner约束。调用方为 scheduler/direct worker; planning and execute,结果进入 per-wake receipt/native session日志;失败由 不静默换ID;原超时/重试/cleanup路径。
- class BenchmarkCodex:把Harbor省略值交给公共解析器;iteration_context参数默认None。_worker_env写解析后的明确context字符串。调用方为 Harbor配置entry,结果进入 同一trial worker;失败由 原配置/预算拒绝;无真实job launch。
- def main:把LHTB CLI省略值解析并冻结到新配置;argparse默认fresh -> None -> Execution。同一46题模板;新输出明确写入解析后的context,原模板不改。调用方为 run.sh现有renderer调用,结果进入 preflight/Harbor;失败由 非法参数及旧非空planning_timeout拒绝。
- class BenchmarkSessionWake:复用产品会话owner;增加解析完成断言,未另写绑定规则。context不可再是None;原产品scope/profile检查保留。调用方为 worker and checkpoint continuation,结果进入 worker真实会话/receipt;失败由 原owner处理损坏、错scope及显式fresh恢复。
正向读回:真实构造器、Harbor worker环境和render_config CLI的24行对照只有heartbeat/Turn省略值的4处变化;真实Codex默认恢复保留规划历史,显式fresh在base/head各保持独立会话。
反向与恢复:真实源码套件覆盖损坏/变化的绑定拒绝、启动前复核、错Goal/Agent隔离、超时保留观察ID及显式fresh恢复;本次无修改这些既有owner规则。原缺省行为在独立连续性oracle下失败、候选通过。
对主干的风险
最强风险是默认复用错误会话,或为了恢复而悄悄换新会话、把显式fresh也变成resume。独立真实Codex回环保留请求、会话ID和完整历史:同一oracle在不可变基线只因省略配置的连续性断言失败,显式fresh仍过;当前head两项都过。另一组24行真实构造器、Harbor环境和renderer对照仅有4处预期默认变化,显式fresh、其他模式和off/false选项一致。
正确的Python3.12、SForge 9d447b5829c17bb342b69a404864117b8aaf3e8d、Harbor0.20环境中,同8模块套件为head701通过/base690通过;native测试另行实际执行,4项Linux进程组检查在Mac未运行。最初Python3.14缺provider/测试导入路径,以及归档基线误继承父Git的失败原文保留;改用独立Git基线和受支持依赖后重新执行,不把环境错误归给PR或追认为历史通过。另一个私有fresh对照草稿漏设fresh参数,已经排除并以正确参数重验。
改动文件Ruff和compile、Bash语法、现有配置17文件mypy、双语/语义/diff通过;没有宣称对全部改动Python模块完成严格mypy。精确CQR为pass(cqr_1907f6bebcc0e490b70c);标准premerge的5项直接和19项选择自动检查通过,但aggregate仍为 manual_review_required:benchmark敏感变更须维护者明确评审后才可自合并。这项合并限制保留;本次不执行merge、安装或实验,也未查询远程CI。
语义与 CI 对齐
复用既有MODES/CONTEXTS;这是公开披露并验证的默认语义变更,不宣称行为保持。两种既有opt-in没有被会话默认值暗中启用;首会话、Goal/Agent范围、拒绝替换、后续恢复、原始任务与固定deadline条款保留。冻结研究和旧回执不得据此改名或重记。
我的整体评价
改动限基准 runtime、配置 CLI 与文档,未改变应用内页面;无需截图,未声明 installed App 或任何视觉资格。 4项Linux进程组用例在Mac未执行;真实Harbor作业、Docker隔离、付费模型采用、完整成本与长程因果收益仍未资格。
删除调用方的重复fresh默认,让shell也询问现有Execution;不增加兼容别名、第二套策略或新持久字段。
对长程维度的 improved 限定为已观察的会话历史连续性,user_experience 的 improved 是新试验少一项重复配置;模型是否真正采用这些历史、增加多少完整成本或提高多少结果,仍须后续匹配研究。本次 goal_achieved 只关闭这个源码默认值切片,结论 APPROVE。维护者合并限制、正式批准资格和安装/发布继续保留。
English verdict: APPROVE — head 301c1e2e6ab974215a8db7be14c642de66753e58 centralizes omitted heartbeat/Turn context as resume while preserving explicit fresh, other modes, opt-in transports and existing scoped-session refusal/recovery. Real native Codex base/head continuity and 701/690 source tests pass. Premerge automated checks pass; benchmark maintainer review remains a merge hold, and this COMMENTED conclusion grants no merge or experiment authority.
New heartbeat and Turn benchmark runs now reuse their native Codex conversation by default, so planning and later wakes retain the same Goal/Agent history without a separate
iteration_context: resumesetting. Explicitfreshremains available, other modes retain fresh sessions, and context delivery stays opt-in (context_delivery: "on"). Existing frozen trial settings are not rewritten.The shared Execution owner resolves the default for Harbor, direct workers and the LHTB renderer/preflight; the LHTB launch script and example follow it. EdgeBench/SForge already selected resume for heartbeat runs.
Validation: 454 focused Linux tests passed with the current native provider source (2 native-binary cases skipped there); both actual Mac Codex loopback session tests passed, including omitted configuration. After the final normalization assertion, 40 session/config tests passed (2 native cases conditional). Ruff, Bash syntax, diff hygiene and the 13-file public-boundary scan passed. Focused type checking passed with the repository's pre-existing unparameterized-dict diagnostics excluded. Earlier Linux failures came from missing Node on PATH and an outdated installed provider; the corrected pinned environment passed. No paid experiment is launched by this PR. Independent review and CI remain separate gates.