Skip to content

fix(status): fail closed on unreadable retry journals - #7693

Open
Duang777 wants to merge 2 commits into
loopx-project:mainfrom
Duang777:codex/core-bug-hunt-20261030
Open

Duang777 wants to merge 2 commits into
loopx-project:mainfrom
Duang777:codex/core-bug-hunt-20261030

Conversation

@Duang777

Copy link
Copy Markdown
Collaborator

Summary

  • fail closed when an existing post-writeback composition retry journal cannot be read
  • keep a missing journal equivalent to no recorded retries
  • cover both receipt readback and status projection readback with a fault-injection regression test

Problem

_iter_composition_retry_rows() caught every OSError and returned an empty row set. If a persisted retry journal became temporarily unreadable, status interpreted the read failure as "no pending retry receipt" and hid durable recovery work.

The fix handles FileNotFoundError as the existing empty-journal case and raises a path-neutral error for other I/O failures. The status entrypoint then returns its existing status_collection_failed response instead of publishing a healthy empty projection.

Validation

  • red/green regression: the new test failed before the fix with DID NOT RAISE RuntimeError, then passed after the exception handling change
  • uv run --extra test pytest -q tests/control_plane/test_post_writeback_composition_retry.py (21 passed)
  • uv run --extra test pytest -q tests/control_plane/test_status_receipt_retry_projection.py (14 passed)
  • real file-authority loopx status fault injection: exit 1, ok=false, status_collection_failed, path-neutral error
  • uv run --extra test ruff check loopx/control_plane/post_writeback_composition_retry.py tests/control_plane/test_post_writeback_composition_retry.py
  • uv run --extra test python scripts/generate_semantic_inventory.py --changed-from upstream/main (no new vocabulary carriers)
  • uv run --extra test loopx canary premerge --from-git-diff --git-diff-base upstream/main (10/10 selected catalog and risk-profile smokes passed)

Sibling sweep

The same fail-open shape was checked in worker_bridge.py and quota slot accounting. The former is optional diagnostic tracing; the latter preserves spend denial when settlement evidence is absent. Neither is the same recovery-projection defect, so this PR leaves them unchanged.

Composition retry readback treated every OSError as an empty journal, hiding persisted pending work from status. Preserve missing-file behavior but raise a path-neutral error for other read failures, with a regression test covering both receipt and projection reads.

Signed-off-by: Duang777 <duangjl007@gmail.com>
Signed-off-by: Duang777 <duangjl007@gmail.com>
@Duang777

Copy link
Copy Markdown
Collaborator Author

Exact-head validation refreshed for 4eb3f713eee62dcaded04022801b4ff28750f370 after merging main at 7c6fadd6fe20429468c979eec77ccb1445304dae.

  • PR diff remains limited to the composition retry reader and its regression test.
  • test_post_writeback_composition_retry.py: 21 passed.
  • test_status_receipt_retry_projection.py: 14 passed.
  • Real file-authority loopx status fault injection returned exit 1, ok=false, and status_collection_failed with a path-neutral error.
  • Ruff, diff hygiene, and the diff-scoped semantic advisory passed.
  • Standard premerge passed all 10 selected catalog and risk-profile smokes.
  • Failures: none. Skips: none. Manual holds: none.

This validation does not grant merge authority; maintainer review remains required.

@Duang777

Copy link
Copy Markdown
Collaborator Author

CI triage for exact head 4eb3f713eee62dcaded04022801b4ff28750f370:

windows-powershell failed four Python distribution activation cases in tests/test_self_update_runtime_activation.py (python_pip and python_pipx). The job passed 415 other tests, and the failure does not involve either file changed by this PR. The test-fixture correction is already owned by #7675.

I am not mixing that unrelated self-update test change into this recovery-journal PR. The remaining checks continue to run; after the owning fix reaches main, this branch can sync that exact base and rerun CI.

Failed job: https://github.com/loopx-project/loopx/actions/runs/40918492419/job/123609945350

@Duang777

Copy link
Copy Markdown
Collaborator Author

Additional CI triage for 4eb3f713eee62dcaded04022801b4ff28750f370:

dashboard-browser (2) timed out in examples/personal-workspace-browser/research-display-language.mjs while waiting for a page condition. This PR does not change the dashboard, browser scenarios, or presentation code. The same browser job is also failing on #7675 on a separate timing assertion, so there is no recovery-journal change to make for this result.

Failed job: https://github.com/loopx-project/loopx/actions/runs/40918492419/job/123609944854

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; reasoning_effort=max

[P2] 目录权限错误仍绕过新的 fail-closed 分支。 当前 head 4eb3f713eee62dcaded04022801b4ff28750f370 的 collect_pending_composition_retry_projection 仍先执行 journal_path.is_file()。在 Python 3.14 中,已有 journal 的父目录不可访问会使这个检查返回 false,随后直接跳过 reader。真实 File/SQLite 的 status 均返回 exit 0、ok=true 并省略 pending receipt;恢复目录权限后,同一条记录重新出现。移除 collect_pending_composition_retry_projection 的 is_file 预检查,或显式传播非缺失的 stat 错误;让现有 reader 统一区分缺失与不可读,并补真实 status 权限恢复回归。

动机 / Motivation

维护者或 Agent 通过 status 查找已提交写回后的待重试工作。

日志文件失去读取权限时,旧版将失败当作空记录;本版返回明确错误,但父目录失去访问权限的情况仍会隐藏记录。

独立 File/SQLite 检查确认文件读取拒绝已显式失败,目录访问拒绝仍返回 ok=true,恢复权限后同一条记录重新出现。

范围为重试记录读取与状态反馈;写回、扣额、hook 和持久格式保持原合同。跨平台和持续性能未验。

改动思路 / Approach

读取失败由现有 journal IO owner 处理,CLI 复用既有错误投影。is_file 预检查重复了存在性决策,却会提前吞掉权限错误,应收敛到同一次读取。 当前 PR 只需补齐同一条重试发现路径,无需新增协议、配置或通用恢复器。

原写回 owner 生成有稳定身份的 append-only receipt;reader 折叠每条 receipt 的最新状态,状态命令负责公开呈现待恢复工作。修复应让“没有文件”和“没有读到文件”保持区别。保留一个额外的存在性判断没有独立合同收益,反而把本 PR 要修复的假健康路径留下。私有进程只删除该预检查后,两个真实后端的目录拒绝都进入既有错误分支;这是一项最小修复验证,没有改写当前提案。

选定 Goal 的实际状态 不可变基线 当前 head
可读 pending journal 成功,1 条 pending 同样成功,1 条 pending
文件确实缺失 成功,无 pending 保留原行为
文件读取权限拒绝 错误地成功、无 pending 正确返回不可读错误、exit 1
父目录访问权限拒绝 错误地成功、无 pending 缺口仍在
恢复访问权限 原 pending 重新出现 原 pending 重新出现

具体改动 / Changes

实际不可变基线 7c6fadd6fe20429468c979eec77ccb1445304dae。当前完整差异两文件、+43/-1:生产 +5/-1,原测试模块 +38。没有新增配置、schema、存储 writer 或 actor;错误分类使用异常类型,既有 retryable/settled 状态和幂等身份继续由原 owner 维护。

关键代码讲解

  • _iter_composition_retry_rows 现在仅把 FileNotFoundError 视为缺失,其他 OSError 包装为不带路径的 RuntimeError。真实文件读取拒绝已验证这项改进,原全量折叠和校验逻辑保留。
  • collect_pending_composition_retry_projection 负责 Goal/Agent 过滤与展示。第 463–464 行的前置 is_file() 使目录拒绝无法到达新分支,是当前最小修复位置。
  • handle_status_command 使用已有 status_collection_failed 和 exit 1 监督收集错误;无须另加恢复协议。

规格检索范围是当前 docs、公开故障与上述 reader/status 调用链,未找到独立规定该 journal 读取错误的书面规格,采用 no_spec。预期来自可独立复现的不变量:读取失败不能证明没有耐久恢复记录;新增测试不是它自己的规格。旧接口的调用者、compaction、原身份 replay 与 settle 路径一并检查,没有理由把可选 worker tracing 或 quota ledger 并为同一个 reader。

对主干的风险 / Risks

新增测试只故障注入 Path.read_text,所以前面的 is_file() 仍成功;该测试通过时,真实目录权限反例仍能失败。这解释了为什么已有 35 项 retry/status 测试全部通过,仍不足以批准完整修复。此遗漏在基线也存在,但它属于本 PR 要完成的同一发现路径,不能归为无关历史红灯。

本轮在同一合成 canonical File/SQLite 状态上执行 20 次不可变基线/当前 CLI,另有两次私有最小修复对照。保留全部命令、完整包、fixture 和观察指纹;8 组正常、缺失、目录拒绝和恢复包除经校验的 observation clock/派生 source age 外一致。目录拒绝的相等恰好证明缺口保留,不代表行为正确。全部原 fixture 内容保持;没有清空 journal、重放业务或扣减额度。

PASS:35 focused tests、Ruff、py_compile、配置内17文件 mypy、两变更文件的公开边界检查。FAIL:当前 standard premerge 的 semantic-vocabulary-drift-smoke 在原120秒上限超时;6项 direct 通过、10项 selected 中9项通过。该轮与私有权限探针/文档构建有重叠;未完成同不可变基线归因,不能猜测原因、称作已修复或标为无关。不为获取绿色结果重复 gate,也未放宽限制。未获取、轮询或等待远端 CI,作者评论中的 CI 状态仍只是作者来源。

语义与 CI 对齐

差异 advisory 没有发现受支持的新 vocabulary carrier;新增文字是中性的读取错误,既有 schema、actor authority 和 replay 说明未改。该 advisory 不证明全部语义覆盖,更不抵消真实目录拒绝和本地 gate 失败。没有新增 default-off 能力或激活声明。原收集失败是确定性错误,恢复说明不授予重试、业务或合并权。

我的整体评价 / Verdict

REQUEST_CHANGES。 局部文件读取修复有价值,体量合理;当前完整目标的 long_horizon 和 user_experience 均为 not_yet_proven,因为后续调用仍可看见错误的成功空视图。读取失败由现有 journal IO owner 处理,CLI 复用既有错误投影。is_file 预检查重复了存在性决策,却会提前吞掉权限错误,应收敛到同一次读取。 当前 PR 只需补齐同一条重试发现路径,无需新增协议、配置或通用恢复器。

已检查 status CLI 和既有投影消费者;此差异没有首屏、布局或可操作页面变更,视觉审查不适用。 视觉未验证范围:未执行浏览器或原生 Windows 资格;没有受影响的视觉布局。全 Goal 聚合 CLI、原生 Windows、PostgreSQL 和持续性能未在本轮独验;没有 storage/schema 变更,不借其它矩阵冒充覆盖。

请先补齐上述前置 stat 分支及真实权限恢复回归,再重验受影响状态路径,并解决或独立归因当前所需的本地超时。无需为此补丁完成整个父 roadmap,也无需新框架。

English verdict: REQUEST_CHANGES — 4eb3f71. Real File/SQLite status still reports a healthy empty view when the existing retry journal's parent directory is inaccessible. Route that failure through the existing reader and cover permission recovery. The file-read-denial fix works; 35 focused tests pass, while the current standard semantic smoke times out at its unchanged 120-second limit.

except OSError:
except FileNotFoundError:
return []
except OSError as error:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 让目录访问失败到达这个 fail-closed 分支。调用方 collect_pending_composition_retry_projection 仍先调用 journal_path.is_file();Python 3.14 在父目录无访问权限时返回 false,随后跳过 reader。真实 File/SQLite 的 status 都继续 exit 0、ok=true 且无 pending receipt,恢复权限后原记录重现。请去掉该预检查或显式传播非缺失的 stat 错误,让同一 reader 区分缺失与不可读;增加真实 status 的目录拒绝与权限恢复回归。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants