Skip to content

test: preserve recovery fixtures and pin CLI comparison ancestry - #7604

Merged
huangruiteng merged 5 commits into
mainfrom
codex/semantic-source-integration-20261029
Oct 29, 2026
Merged

huangruiteng merged 5 commits into
mainfrom
codex/semantic-source-integration-20261029

Conversation

@loopx-agent

@loopx-agent loopx-agent commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Goal And Delivered Outcome

CI can compare a candidate's CLI output against a newer, unrelated origin/main fetched after GitHub created the candidate merge commit. An unchanged candidate then appears to regress after upstream compacts its own output. The feedback and evidence journeys also need to retain their disposable runtime and original validation subject.

This proposal pins the default comparison to the candidate's common ancestry and an immutable commit before checkout. An explicit non-main historical base remains authoritative. It also keeps the feedback registry/server/Chat store on one temporary runtime, waits for accepted configuration, and restores the original Todo-bound validation before linked recovery. Related to #7526. Intended base: main.

Author Declaration

  • Written by: model_agent — OpenAI Codex, GPT-6 family.
  • Specification and revision: no written specification; the reproduced CI and fixture defects in this PR are the basis. Isolation, budget attribution and real-path validation follow AGENTS.md, docs/development/testing-and-quality.md and docs/development/good-smokes.md at 652f503842609e948e8ff832ae3977f23f79f28d.
Criterion Disposition Path Decisive validation
Compare one candidate against its actual development ancestry implemented cli-output-base-head-differential-smoke.py Disposable real Git: later upstream compaction no longer creates a false regression; moving the branch after resolution cannot change the pinned checkout
Preserve actual growth and explicit-base enforcement implemented test_cli_output_differential.py A real +1,000-character candidate regression still fails with the exact delta; explicit historical comparison still rejects +3,995; unresolved bases fail closed and candidate ceilings remain enforced
Use the disposable runtime throughout feedback implemented team-feedback-http-fixture.py Production HTTP/Chat/Inbox; accidental ambient default discovery fails before accessing a live runtime
Finish configuration before changing fixture state implemented team-feedback-recovery-smoke.mjs Await and assert the original configure response
Preserve the original subject through linked recovery implemented team-evidence-return-smoke.mjs Restore currentValidation; all original wrong-subject/version/loss/recovery assertions remain

Self-check: reviewed the complete five-file diff. Reused the existing measurement owner, semantic anchors, fixtures and original validation value. The small Python helper resolves Git I/O for the existing test harness; it adds no runtime decision owner or CLI option. Source-success interception was independently removed on current main and is no longer part of this PR's diff.

Scope And Continuation

Complete within this author test/harness repair scope. Product UI, authority, protocol, output ceilings, timeouts and frozen performance thresholds are unchanged by the PR diff. Remote Windows qualification, exact-head review and merge readiness remain separate requirements. The parent source-recovery/adoption program remains open: synthetic local journeys do not establish installed App adoption, actual model behavior, requester acceptance or external delivery. No new successor is needed for this bounded repair.

Validation

  • Tested revision: f3587909a1587a23593f8ffed462fa1278a84d5d, integrated with main 652f503842609e948e8ff832ae3977f23f79f28d.
  • Run state: finished for the listed local checks; remote checks for this revision are not yet qualified.
  • Input classes: synthetic, public_fixture.
Check kind Result Public-safe evidence / limitation
regression_parity passed Before the comparison fix, the real Git reproducer rejects unchanged output by +3,995 and misattributes actual +1,000 growth as +4,995. With the fix, unchanged ancestry passes and genuine growth still fails by +1,000. Explicit historical comparison remains +3,995 and fails
integration passed 184 CLI differential/probe-runner tests at the final head; immutable base selection, ref movement, cleanup, explicit history, unresolved refs and candidate baseline ceilings
real_entrypoint / real_backend passed Complete packaged team-feedback journey with production HTTP/Chat/Inbox on disposable SQLite/File state: lost acknowledgement after commit, exact retry, one ingress, scoped input, paused admission and mobile/keyboard reading
real_entrypoint passed Complete packaged team-evidence journey: wrong subject/version, linked downstream/core loss, revocation/restoration, navigation, comparison, pagination and keyboard/narrow reading. Evidence payloads are synthetic
real_entrypoint / real_backend passed Complete delegation-search journey through its real source endpoint: corruption/refusal/recovery, search/history/paging, stale responses, source isolation and read-only receipts
real_entrypoint passed Full CLI output budget regression smoke against the pinned main ancestry using the real File/SQLite fixture matrix, semantic anchors and unchanged budgets
static passed Fresh packaged Chat build/frontend checks; configured mypy (17 sources); control-plane TypeScript typecheck; CI Ruff scope plus changed Python; bilingual diff and five-file public boundary scan
integration passed Standard source-checkout premerge: 5 direct and 9 selected checks, zero failures, advisory failures or manual holds; current five-file change-quality receipt is valid. Validation grants no merge authority
integration not_run Remote Windows and six pytest shards for this new head are not yet qualified. Local reproduction demonstrates the baseline-selection defect; it does not establish that the old Windows failure had only this cause

Historical failures remain failures. The previous Windows run exceeded CLI differential allowances by +305 and +353 characters. Its GitHub-generated candidate merge used an earlier main parent, while checkout fetched main again later; the exact later fetched SHA was not logged. Current main includes independent CLI compaction, and the disposable real Git reproducer demonstrates the resulting comparison error. No raw trace or unverified Windows replay is claimed here. Earlier browser M2 measurements exceeded the frozen 1-second switch budget; their runner cause and sustained performance remain unqualified. The threshold and historical outcome were not changed. Earlier environment setup failures were corrected; they are not passing historical checks.

Frontend / Visual Evidence

UI impact: none. First viewport/preview: N/A because the diff changes test setup and measurement only. Existing packaged UI is exercised with synthetic content. No PostgreSQL authority-store change, live installation, external message or model dispatch is proposed.

Boundary Checklist

  • Public diff/body contain no private state, credentials, raw traces, internal links or local paths.
  • No duplicate benchmark work, new runtime owner or relaxed scoring/budget/performance criteria.
  • No paired RFC changed; bilingual diff check passed.
  • All contribution commits have DCO sign-off.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…mantic-source-integration-20261029

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
@loopx-agent loopx-agent changed the title test(chat): isolate feedback runtime and exercise real source recovery test(chat): preserve source and team recovery fixture boundaries Oct 28, 2026

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent; model=gpt-6.1-sol; provider=OpenAI; declaration_source=runtime_reported; reasoning_effort=max.

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

结论:APPROVE,未发现阻塞问题。 完整评审绑定 head f4fe19bc298c8b030a40bbea3cbb976e95bc71f2,不可变基线 0d4bf99a4aa6080f2774f3c4477fb4adc1b5ac46。三项原失败均已独立复现,修复后的完整路径通过;GitHub 实际 review state 使用 COMMENTED。

动机 / Motivation

维护团队证据、反馈和委派搜索测试的开发者,可能遇到测试误用默认 runtime、掩盖来源失败,或错误混合原请求与 Todo 的验收身份。
例如,真实来源已经不可用时,旧测试仍向浏览器返回成功;另一个恢复用例在检查 request-only 结果后继续沿用该身份,随后进入原 Todo-bound 证据链,UI 正确拒绝而测试失败。
本次让测试使用隔离来源、等待配置真正成功,并在返回原任务流程前恢复原验收 fixture。可验证结果是这些完整流程能检验既有拒绝与恢复语义。本 PR 不改变产品 UI、协议、权限或正式安装行为,也不完成原协作采用计划。

改动思路 / Approach

不修改会保留假成功、环境泄漏风险和不可用的集成回归。较小的修复是在原测试入口绑定同一个临时 runtime、删除成功 stub、等待既有 configure response,并复用已有 currentValidation。增加 mock 模式、重试或延长超时都不能解决错误来源和错误身份。
决定保留该实现的理由是:四处变化都修复测试的真实边界,生产 decision/effect owner 完全未改,现有负例和冻结超时保留。当前 PR 边界是这四个 fixture/流程修复,不扩展为运行时改造或前端设计。
正向流程仍是 packaged UI → 原 HTTP/Chat/Inbox owner → 临时存储 → 独立 inspect/readback。反馈的丢响应后同身份重试、paused 拒绝和零额外 model Turn 都保留;证据页面的 source/core/adoption 失效各由原读回逻辑处理。没有增加用户配置或确认轮次。

具体改动 / Changes

完整差异为四个测试文件、+17/-7,三个提交均有 DCO。对 loopx/** 和 dashboard src/** 的基线/head 比较为空,产品实现和 rendered presentation 没有变化。

关键代码讲解

  • team-feedback-http-fixture.py::main 将 registry 的 common_runtime_root、HTTP server override 和 Chat store 绑定到同一个临时目录。入口 tripwire 只拒绝默认 runtime discovery,未伪造 Inbox/Chat 成功。
  • team-feedback-recovery-smoke.mjs 在点击保存前注册 response wait,断言 configure 为 HTTP 200 后再修改合成 mode,避免配置未完成就推进后续状态。
  • team-evidence-return-smoke.mjs 从原 currentValidation 构造版本不匹配反例,恢复 Todo subject;不同输出版本不能显示为当前验收的断言保留。
  • delegation-search-browser-smoke.mjs 移除无条件 app-steward 成功拦截,让真实来源的 HTTP 409 能进入页面错误和恢复路径。

依据为变更前 good-smokes.md:spec_ref docs/development/good-smokes.md;spec_revision 0d4bf99a4aa6080f2774f3c4477fb4adc1b5ac46。用原章节名映射未编号要求:Start With The Contract → implemented,保留已交付公共流程;Prefer Semantic Pressure → implemented,真实拒绝和恢复不被成功 stub 盖住;Public-Safe Fixtures → implemented,同临时 runtime 且确定清理;Validate The Change → implemented,按真实入口与风险执行验证。它们是维护合同的映射,不是新增产品功能标准。

对主干的风险 / Risks

最强反例是测试能通过,但一次反馈实际访问了默认持久来源,或来源错误被路由 stub 掩盖。因此没有仅检查 fixture 文本:完整运行了团队证据、反馈和 delegation-search 三条 packaged journey,并用同一独立 tripwire 做基线/head 对照。tripwire 在任何默认来源 IO 前拒绝,既有 HTTP/Chat/Inbox 操作保持真实。

相同边界下的独立对照 基线 当前 head
完整 team-evidence-return,原超时和断言不变 FAIL:关联恢复无法取得独立验收提示,保留原失败 PASS:核心/采用失效、撤销、恢复、键盘、分页和零执行断言均通过
完整 feedback,加相同默认 discovery tripwire FAIL:实际命中默认 selector,IO 前拒绝,HTTP 409 PASS:默认 selector 新增调用 0;真实丢响应/同身份重试仅 1 ingress/1 message,paused 仍拒绝
完整 delegation-search,实际 source-unavailable FAIL:成功 stub 掩盖错误,页面等不到来源不可用状态 PASS:真实 409、恢复、历史/分页、stale response、来源隔离、中英文与窄屏均通过

新构建的 Chat bundle 使用当前源码。标准 canary premerge --from-git-diff --git-diff-base origin/main 为 PASS:5 direct、5 selected、0 manual holds。适用 Python 编译、diff hygiene、双语差分及公共边界检查通过。没有用产品 mock 冒充真实模型或跨会话正式采用;UI 证据 payload 仍是 synthetic,反馈的本地 HTTP/store/Inbox 是真实边界。

语义与 CI 对齐

这里复用原 runtime/source 和验收 subject vocabulary,无新 schema、权限、状态 owner 或产品默认。被删除的是成功 stub,不是 source failure 断言;被恢复的是原 Todo subject,不是放宽身份校验。产品 feature-off 路径由空产品 diff 保持。wait_for_ci=false,未查询或等待 CI。
作者记录的旧输出预算失败和 1,019.7 ms/1 秒历史 runner 失败未改写为通过;本次完整相关 journey 的通过也不证明持续 runner 性能。未改冻结阈值,历史性能原因仍保持未资格化,与本次四文件测试修复的验收分开。

我的整体评价 / Verdict

这四处变化共同修复同一测试维护问题,已达到该切片的 goal_achieved;没有新增 smoke、框架或平行生产逻辑。long_horizon 与 user_experience 判为 preserved:原后续恢复、来源拒绝与一次反馈义务继续受测,没有证明用户体验或模型效果的额外收益。
架构边界局部且相称,无需为已完成修复另建后继任务。visual_review 为 not_applicable:检查的四个路径只改变测试设置,产品 UI/样式完全未改;unverified_scope 为正式 App 安装采用、真实模型/receiver adoption 和持续性能。自动生成截图仅属于合成 packaged 流程观察,不是本 PR 的视觉改善证明。

English verdict: APPROVE — the original failures are reproduced and the complete repaired journeys pass without weakening rejection or recovery checks.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
@loopx-agent loopx-agent changed the title test(chat): preserve source and team recovery fixture boundaries test: preserve recovery fixtures and pin CLI comparison ancestry Oct 29, 2026

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent; model=gpt-6.1-sol; provider=OpenAI; declaration_source=runtime_reported; reasoning_effort=max.

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

结论:APPROVE,未发现阻塞问题。 本次重新检查完整 head f3587909a1587a23593f8ffed462fa1278a84d5d,基线 652f503842609e948e8ff832ae3977f23f79f28d;旧 head 的评审结论未继承。GitHub 实际评审使用 COMMENTED,独立审批、分支保护和受保护合并仍是另外的门。

动机 / Motivation

维护回归测试的开发者在 CI 已构建候选提交、随后主分支继续前进时,会遇到没有发生的输出增长告警。
例如,上游把自己的输出缩短后,旧比较器把候选的未改输出误报为增加 3,995 字符;修复后按候选共同祖先比较,未改输出通过,真实增加 1,000 字符仍失败。
可验证的改进是固定比较提交、保留真实增长拒绝,并让完整团队反馈和证据恢复流程继续检查原来源与验收身份。
本次不改产品界面、权限、输出预算、超时或冻结性能标准,也不宣称完成正式 App 采用、真实模型效果或跨会话送达。
原 Windows 失败的完整因果、该版本远端资格和受保护合并仍未完成;它们由原 PR 7604 的现有交付路径继续处理。
这会反复消耗定位时间,并诱使维护者错误压缩有效上下文或提高限制。保留旧比较方式不能可靠区分候选增长与无关上游变化;仅重新跑一次也无法约束比较基线。

改动思路 / Approach

保留该结构的理由是:Git 提交解析属于现有测试 harness 的 IO 边界,差分政策、语义锚点和预算继续复用原 owner;三处浏览器变化只纠正测试状态和来源,不增加生产决策。
当前 PR 边界是五个测试与测量文件的修复,交付可独立运行的真实比较及完整恢复流程。
正向路径为预算 smoke → 原基线选择规则 → 唯一共同祖先或显式历史提交 → 固定 SHA 的临时 checkout → 同一夹具的两臂测量 → 原差分与绝对预算断言。解析失败或出现多个共同祖先时停止并要求明确 base;显式非 main 历史比较仍保留,不能用共同祖先消除有意指定的差异。这里的 helper 不承载 Goal、quota 或调度规则。
反馈路径仍由 packaged UI 调用真实 HTTP/Chat/Inbox,再独立检查临时存储。保存设置先等原 configure 返回成功,丢确认后仍按同一身份重试且只产生一个 ingress。不存在新增用户准备或确认轮次;来源错误、暂停和失效证据仍由既有恢复流程处理。

具体改动 / Changes

完整基线到 head 差异是五个测试/夹具文件、+166/-6。相对旧评审 head 的增量是集成当前 main 并修复 Git 比较基线;source-success 拦截已被 main 独立移除,当前 PR 不再把那两行当作自己的交付。loopx/**、dashboard src/** 和 packages/** 的最终差分为空。

关键代码讲解

  • _pin_probe_base_ref 对选中的 main 取唯一共同祖先,对显式其它 ref 取其 commit;在创建临时 worktree 前固定 SHA。失败时说明如何传入明确的 --base-ref,比较失败也附上实际 base。
  • main 继续使用原 selector、probe、semantic anchors 和预算函数;base 为观察臂,candidate 保留强制预算,finally 负责删除临时 checkout。
  • team-feedback-http-fixture.py::main 将 registry、server override 与 Chat store 指到同一临时 runtime。入口 guard 只拒绝环境默认来源发现,不替代 HTTP/Inbox 的成功结果。
  • team-evidence-return-smoke.mjs 在返回 Todo-bound 恢复前复用原 currentValidation;版本不匹配仍清除时间和 witness。反馈的 configure response 等待位于 team-feedback-recovery-smoke.mjs:41,它没有引入额外重试或超时。

依据为变更前维护合同:spec_ref docs/development/good-smokes.md;spec_revision 652f503842609e948e8ff832ae3977f23f79f28d。逐项映射原章节:Start With The Contract → implemented,现有 CI/canary 调用该完整比较与恢复流程;Prefer Semantic Pressure → implemented,未改输出、真实增长、显式历史与来源/身份失效均走原拒绝语义;Public-Safe Fixtures → implemented,临时 Git 和 HTTP/runtime 夹具可清理且不访问活跃状态;Validate The Change → implemented,实际入口与负例验证如下。预算合同 docs/reference/contracts/interface-budget-contract.md 的同夹具两臂、semantic keys、增长检查及原失败保留要求均未放宽。

对主干的风险 / Risks

最强反例是“新的基线让所有候选都和自己比较,从而隐藏真实增长”。真实 Git 夹具没有 mock 共同祖先结果:上游独立推进后,无变化候选通过,但候选实际增加 1,000 仍按原门失败;显式非祖先比较仍报告 3,995 并拒绝。选定后再移动 ref,checkout 仍使用原 SHA,最后检查仅剩原 worktree、HEAD 和原文件未被改变。缺失 base 明确失败,不回退到成功结果。

同一边界的观察 旧实现 当前 head
上游后来压缩、候选输出未变 FAIL:误报 +3,995 PASS:对候选祖先测量
候选确实增长 +1,000 FAIL:误算 +4,995 正确拒绝 +1,000
显式其它历史提交 拒绝 +3,995 仍拒绝 +3,995
完整反馈/证据恢复 原来源、配置竞态或 subject 问题保留在历史证据中 三条 packaged journey 全部 PASS,原失效、暂停、丢响应与恢复断言保留

最终 head 的 184 个差分/probe 用例、真实 File/SQLite CLI 矩阵、团队反馈/证据及委派搜索完整 packaged 流程均为 PASS。新 Chat build、mypy、TypeScript typecheck、CI Ruff 范围与改变的 Python、双语差分和公开边界检查通过。标准 premerge 为 5 direct、9 selected、零失败/继承失败/人工 hold,精确范围 change-quality 回执有效。未运行与此测试修复无关的真实 PostgreSQL 或付费模型任务。

语义与 CI 对齐

复用既有预算、状态和验收 vocabulary;新 Git helper 保持局部。比较默认从可变 main 改为候选共同祖先,CLI help 和失败诊断公开该变化;没有压缩产品指令、缩小场景矩阵、删语义键或调高阈值。基线与本次 head 的 runtime instructions 不变。wait_for_ci=false,本次评审未查询、等待或轮询 CI;缺少远端结论不冒充本地失败,也不据此解除分支保护。旧 Windows 的 +305/+353 和历史 M2 超时保持未完全归因;本次真实 Git 反例只证明比较缺陷,不证明历史失败仅有这一个原因。

我的整体评价 / Verdict

这是 justified_increment:修复重复 CI 误归因和不可用的原恢复回归,原交付路径继续负责剩余合并与正式采用。long_horizon 判为 improved,限于后续重复验证能区分真实增长和无关 main 变化;未声称提升模型任务得分或完整执行成本。user_experience 判为 preserved,产品交互、用户授权和恢复路径保持原样,没有增加人工轮次。
保留该结构的理由是:Git 提交解析属于现有测试 harness 的 IO 边界,差分政策、语义锚点和预算继续复用原 owner;三处浏览器变化只纠正测试状态和来源,不增加生产决策。
当前 PR 边界是五个测试与测量文件的修复,交付可独立运行的真实比较及完整恢复流程。
visual_review 为 not_applicable:逐一检查的五个路径只修改测试与测量,产品 UI、样式和布局的最终差分为空。
unverified_scope 为正式安装采用、真实模型或接收方采用,以及持续 runner 性能;本次未把合成截图当成视觉改善证据。
本次没有新增兼容格式、状态机或并行 owner;复用原验证值、预算比较器和同夹具执行入口已是有证据支持的较小修复。未发现应在本 PR 继续扩大的重构。APPROVE 仅针对该精确代码范围,不授予 admin bypass,也不代表已合并。

English verdict: APPROVE — candidate ancestry is pinned, genuine growth still fails, and complete recovery journeys preserve the original negative checks.

@huangruiteng
huangruiteng merged commit 79c1e51 into main Oct 29, 2026
35 of 43 checks passed
@huangruiteng
huangruiteng deleted the codex/semantic-source-integration-20261029 branch October 29, 2026 07:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants