Repository navigation
Conversation
|
This pull request has merge conflicts with Choose the remote for the base repository, not an out-of-date fork. git fetch upstream
git rebase upstream/main
# Resolve each conflict, git add the resolved files, then git rebase --continue.
git push --force-with-lease origin HEADFor a same-repository clone whose Keep the DCO |
…servation loopx-project#7557 preserved capability candidate collections on the hot-path projection. The real-CLI budget fixture re-measures the same `quota should-run` workload at 24,925 -> 25,836 JSON characters for the multi-agent scenario (+911), which exceeds the unchanged 25,500 ceiling. Markdown is unchanged at 6,439. Small (23,265 against 25,000) and crowded (35,836 against 42,000) stay inside their ceilings, the per-Todo allowance (300 chars/unit) and the fixed semantic allowance (9,700) still cover crowded-minus-small growth, and every other surface ceiling is untouched. Raise only this ceiling to 27,000 and record the base/head measurement at the spec. The original overflow remains a recorded failure, not a reclassified pass. Runtime payload size and production transport limits are unchanged. Validation: tests/control_plane/test_cli_output_budget.py 38 passed, 1 skipped; ruff clean. Signed-off-by: xiaoan17 <anbc1224@gmail.com>
85a8372 to
722f2b9
Compare
|
This pull request has merge conflicts with Choose the remote for the base repository, not an out-of-date fork. git fetch upstream
git rebase upstream/main
# Resolve each conflict, git add the resolved files, then git rebase --continue.
git push --force-with-lease origin HEADFor a same-repository clone whose Keep the DCO |
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · reasoning_effort=max
未发现当前精确 head 722f2b9 的阻断问题,结论为 APPROVE。
动机 / Motivation
运行 CLI 输出回归检查、维护 Agent 决策上下文的开发者。
同一组三 Agent fixture 的完整 JSON 已有25,836字符,原25,500上限使基线回归检查失败;本 PR 只把该上限调到27,000,保留实际返回的全部内容。
实际基线失败已复现,当前原矩阵通过;六组真实console的字段、正文和命令保持,其他预算及负向拒绝仍有效。
本次不调整生产传输硬限制、模型执行预算、冻结性能标准,也不声称真实模型效果或长程收益已经通过。 这里恢复的是有意义的回归警戒线,不把测试变绿当作生产性能或模型效果的证据。
改动思路 / Approach
这是测试回归预算的单点校准;完整上下文已有消费者,删除候选正文来满足旧阈值会损失信息,沿既有预算owner调整一个数值即可。
本 PR 只交付多 Agent JSON回归预算和配对实测注释,未改变CLI运行时、fixture、行数或增长预算。
实际调用关系是完整console输出进入measure_cli_output,再由原baseline、行数、语义键和增长门判断;没有新owner、开关或执行效果。用文字足以说明这一处校准,不额外绘图。保留25500会持续报告已实测的同一失败,统一提高所有预算或删除正文都没有本次证据支持。
具体改动 / Changes
spec_ref: docs/development/testing-and-quality.md;spec_revision: e19d94c。对应既有 Budget Failure Decisions:用相同工作负载和计量口径归因,保留有用上下文及原失败,区别回归预算与硬/冻结边界。本项为implemented;不是改写实验通过线。
完整差异为一个既有模块+5/-1,仅一个常量和实测注释。生产AST在27000替回25500后逐项相同;fixture和原断言均未改。
关键代码讲解
- CLI_OUTPUT_BUDGET_SPECS 仅将multi_agent JSON阈值从25500改到27000,Markdown仍7000;行数650、每Todo增长300及固定语义增长9700均保持。
- measure_cli_output 按实际stdout记录字符、UTF-8字节和行数并解析JSON,本身未变,不先删候选或正文。本组合fixture当前均为ASCII,所以字符数与字节数恰好相同,不能因此混用一般口径。
- assert_cli_output_baseline 继续独立拒绝超字符、超行数和缺语义键;原矩阵仍检查JSON/Markdown、规模增长及完整bootstrap路由。
同物理fixture、同运行环境、三个不可变revision的实际console测量如下;这张表只表达测量,不证明模型收益:
| 场景/口径 | 候选保留前5a7cc8195 | 当前基线e19d94ca0 | PR head |
|---|---|---|---|
| small JSON字符 | 22618 | 23265 | 23265 |
| crowded JSON字符 | 34732 | 35836 | 35836 |
| multi_agent JSON字符 | 24925 | 25836 | 25836 |
| multi_agent Markdown字符 | 6439 | 6439 | 6439 |
历史到当前multi_agent增量911已复现,原25500超限失败也由原测试在基线复现。当前base与head的六对完整输出仅有逐字段列明的生成时钟、rollout event id和File导入版本摘要差异;选择、候选正文、完整诊断和命令及其他字段保持。历史版本的完整原文与当前投影元数据差异保留,不冒称历史到当前全部等价。
对主干的风险 / Risks
最强风险是用提上限掩盖无价值膨胀。当前实际headroom为1164字符,只调整这一场景;未减少输入规模或输出内容,所有其他AST与预算保持。真实输出上构造27001字符、652行和删除selected_todo三种变体,均由原helper拒绝。这些是预算helper的负例,不能冒充新的生产故障注入。
PASS:预算与actual-default portfolio模块88项、标准premerge六direct及11selected、改动文件Ruff、配置17文件mypy、完整TS typecheck、diff/DCO/双语门和严格精确CQR。原矩阵在基线明确FAIL为25836>25500,在head通过。原测试另有1个native temp namespace无法容纳depth-12布局的条件skip;其fixture、条件和断言未变,不把skip或一次本地绿变成原生Windows资格。未查询或等待CI。
私有验证驱动初次用了不存在的pytest节点,未执行测试;另一次误以为所有场景都有可选capability_gate而KeyError。修正读取实际结构后才完成上述六对证据;这些驱动失败保留,不算产品测试或PR回归。
语义与 CI 对齐
沿用既有测试预算owner,保持local_only,无新状态词汇、配置或authority。既有候选完整性测试和真实console对照承担语义证据;预算门通过本身不证明文本充分或模型采用。没有松动生产硬限制、执行预算或冻结实验标准。
我的整体评价 / Verdict
APPROVE,当前单点回归预算维护目标达到。long_horizon与user_experience均为preserved:原CLI接受、选择、诊断与恢复入口保持,无新增设置、输入或确认轮次。实际CLI上下文未缩水,尚未测量模型长期效果或总成本收益。
这是测试回归预算的单点校准;完整上下文已有消费者,删除候选正文来满足旧阈值会损失信息,沿既有预算owner调整一个数值即可。 本 PR 只交付多 Agent JSON回归预算和配对实测注释,未改变CLI运行时、fixture、行数或增长预算。
有界维护已应用,不需要新框架或人为后继。已检查 quota CLI、预算consumer及现有Markdown输出;没有页面、渲染组件或交互布局变化,视觉评审不适用。 原生Windows、安装态和真实模型采用/长程效果未在本轮验证。 合并、安装及其他公共发布权仍由各自owner决定。
English verdict: APPROVE — 722f2b9. Matched real-console measurements reproduce the 25,836-character baseline failure and justify the single 27,000-character regression ceiling. Other guards remain effective; current validation passes with one preserved environment-dependent skip. Model and long-horizon qualification remain separate.
Superseded by main — closingThe change this PR proposes is already on
This PR proposes 27,000 for the same fixture instead. Keeping both would leave The remaining CI failure on this PR was not caused by it: Closing rather than rebasing. Reopen if the 26,500 ceiling proves too tight for Superseded by main — closingThis PR's change is already on The one failing check here was a base failure, not this PR's: the delegation Reopen if 26,500 proves too tight; the measurement lives in |
Scope
One ceiling in the real-CLI output budget, plus the measurement that justifies it.
4b2b97b7b(#7557, "Preserve complete context in shared quota diagnostics")replaced candidate-collection compaction in
loopx/control_plane/quota/cli_projection.pywith preservation. The syntheticmodel-portfolio budget for that contract was already re-qualified upstream in
98ca82923(#7583), which raised 40,000 → 90,000 and added preflight checks thatcandidates survive. This PR covers the remaining surface that still measures the
old contract.
Current gap
On
mainate19d94ca0:The real
quota should-runpacket grew because the hot path now keeps thecapability candidate context. Nothing bounds that growth except this ceiling,
which was calibrated against the compaction contract #7557 removed.
Measurement (same fixture, base vs head)
Base is
4b2b97b7b^; head ise19d94ca0. Both arms run the existing budgetfixture.
Markdown is unchanged for crowded and multi-agent (8,148 / 6,439) and moves
7,479 → 7,481 for small, all inside their unchanged ceilings. The per-Todo
allowance (300 chars/unit) and fixed semantic allowance (9,700) are untouched
and still cover crowded-minus-small growth (12,571 measured).
Change
loopx/control_plane/testing/cli_output_budget.py— raise only the exceededmulti-agent JSON ceiling, 25,500 → 27,000, with the base/head measurement
recorded at the spec. No other surface, growth allowance or line budget changes.
No runtime behavior changes: the constant governs a qualification fixture, not
production transport.
Validation
The base/head arms were measured on the same fixture; the original 25,500
overflow is recorded here as a failure, not reclassified as a pass.