Skip to content

perf: batch remote filesystem walks - #4543

Open
7Sageer wants to merge 1 commit into
MoonshotAI:mainfrom
7Sageer:perf/remote-fs-walk-v2
Open

7Sageer wants to merge 1 commit into
MoonshotAI:mainfrom
7Sageer:perf/remote-fs-walk-v2

Conversation

@7Sageer

@7Sageer 7Sageer commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator

Requirement or Bug

在 #4062 的远程执行环境上,文件搜索、补全和无 ripgrep 时的 grep 不应按目录数量放大 RPC 往返时间。本 PR 新增独立 fs/walk 原语,将同一 workspace root 的目录枚举合并为一次 RPC,不改动大文件 read/write 或已有 ripgrep 快路径。

Bug Reproduction Steps

N/A

Root Cause

N/A

Code Changes

核心思路是把“递归枚举目录”下沉为文件系统能力,而不是让 Workspace 在 RPC 客户端上逐目录递归。Workspace 对批量返回的条目按 parent 分组后恢复 DFS 访问,因此原有匹配、去重和展示逻辑继续复用;旧 executor 不支持新方法时自动回退到逐目录读取。

flowchart LR
    A[Workspace search / suggest / grep] --> B[HostFileSystem.walk]
    B --> C[Local / ACP bounded walker]
    B --> D[RemoteFileSystem]
    D -->|fs/walk| E[Executor-side bounded walker]
    D -->|MethodNotFound or oversized ignore| F[Bounded per-directory fallback]
Loading

本地与远程共用同一个 walker 的限制和错误语义;远程只替换目录读取传输,所以不需要为忽略规则、symlink 和 DFS 语义维护第二套实现。

决策 选择 没选 理由
原语边界 独立 fs/walk,只替代目录枚举 改 read/write 或把搜索参数传到 executor 消除目录数 × RTT,同时保持 Workspace 搜索行为集中
时间预算 调用方传绝对 deadline;executor 接收剩余 timeoutMs 每个 root 重置超时,或用 Promise.race 模拟取消 多 root 共享 10 秒 suggest / 30 秒 grep 总预算,已发现的 partial inventory 始终可用
超时结果 返回 { entries, truncated: true } 丢弃已遍历结果或抛出统一超时 补全和搜索可以消费部分结果;无候选且无扫描进度的 grep 仍报告超时
卡死目录读取 依赖现有 RPC transport 硬超时 声称底层 readdir 可取消 预算检查不能中断已进入内核的 readdir,保留真实的能力边界
旧 executor 首次 MethodNotFound 后缓存 unsupported,使用同一有界 walker 回退 要求同步升级两端 保持新旧版本兼容,也避免每次操作重复探测
 WorkspaceFsService.search / suggestWithNode / grepWithNode
-  readdir(root) -> readdir(child) -> ...    每个目录一次 RPC
+  hostFs.walk(root, ignoreRules, deadline)  新 executor 每个 root 一次 RPC
+  group entries by parent -> DFS visit      保持访问、匹配与去重顺序
+  consume partial entries + truncated       超预算后保留结果并停止后续 root
  • walk 保留根 .gitignore、始终排除 .git、隐藏文件、glob、symlink 和去重语义;目录 symlink 不递归,远程默认使用 POSIX 路径规则。
  • 限制深度 64、目录 10,000、条目 100,000、单目录 50,000、响应约 8 MiB;普通 walk 默认 55 秒,executor 参数上限 60 秒,远程调用默认上限 55 秒,低于 transport deadline。
  • suggest 的 10 秒 deadline 从 rg 尝试前开始;grep 的 30 秒 deadline 从操作开始计算。超时后 grep 至少扫描一个已发现文件,避免已有候选时直接报零进度超时。
  • 目录枚举期间的普通读取错误继续跳过;超大目录仍映射为 FS_TOO_MANY_RESULTS。
  • 协议 README 同步 fs/walk 与兼容回退;没有新的用户配置或文档操作流程。

阅读顺序:hostFsWalk.ts 看统一遍历与预算语义,remoteFileSystem.ts 和 fsHandler.ts 看 RPC 契约与兼容回退,workspaceFs/fsService.ts 看 partial inventory、DFS 恢复和整体 deadline;其余是接口接线、测试与协议文档。

Behavior Changes and Affected Users

Behavior Before After Who relies on the old behavior Escape hatch
远程 Node 文件枚举 每个目录一次 fs/readDirectory 新 executor 每个 workspace root 一次 fs/walk 使用 SSH / Docker / command 环境的 TUI、print、desktop / web、VS Code、ACP 与 SDK 用户 无开关;旧 executor 自动兼容
旧 executor 原有逐目录枚举 首次 walk 探测失败后缓存能力并逐目录回退,保留原行为 尚未升级远端 kimi exec-server 的远程会话 不需要升级;升级后启用批量 walk
大 gitignore 本地 matcher 逐目录筛选 不超过 1 MiB 的规则在 executor 侧剪枝;超限完整保留规则并回退逐目录 使用大 .gitignore 的远程 workspace 自动回退,无配置
遍历范围与结果完整性 深度限制 64;无全局目录 / 条目 / 响应预算 新增有界预算;达到目录、条目、响应或时间预算时 truncated=true 并保留部分结果 极大 workspace 中的搜索、补全及无 ripgrep 的 grep 缩小路径 / glob / 查询范围;ripgrep 路径不变
多 root 超时 每个递归路径独立受 signal 控制 suggest / grep 共用整个操作的绝对 deadline,超时后不再访问后续 root 多 workspace root 且高 RTT 的远程用户 缩小查询范围;没有恢复旧计时的开关
本地与 ACP 目录访问 本机逐目录递归 共用同一有界 walk 后恢复 DFS 顺序 默认本地会话及使用 ACP 文件能力的客户端 无需配置
文件读写与 rg 快路径 既有实现 不改变 所有文件读写用户;已安装 ripgrep 的搜索路径 N/A

旧路径功能对应:根 .gitignore 读取与缓存保留在 ignoreFor,.git/ 在无 gitignore 模式也作为默认剪枝规则;隐藏文件 / glob 继续在 Workspace visitor 筛选;symlink 仍只作为条目访问、不递归;real-root 去重及多 root 展示路径不变;DFS 访问顺序由 parent 分组恢复;普通 readdir 失败继续跳过,超大目录错误仍抛出。

测试覆盖远程单 RPC、ignore 剪枝、能力缓存、大规则回退、partial inventory、共享 deadline、DFS 顺序、symlink、深度与响应预算;ACP 文件能力也完成回归。没有验证真实 SSH / Docker 高 RTT 环境,性能结论来自 RPC 调用数从目录数降为 root 数。

验证结果
  • 干净 Linux 隔离环境全量测试:18,232 passed,20 skipped。
  • 远程 / Workspace / 本地 walk 相关回归:165/165;新增 deadline 回归后相关三组:137/137。
  • ACP:151/151。
  • agent-core-v2、kap-server、acp-server typecheck 通过。
  • type-aware lint:0 errors;comment-free 检查通过。
  • import-boundary 仅剩基线已有的 loop.ts、loopService.ts、origin.ts 3 条;本 diff 无新增违规。

Checklist

  • I have read the CONTRIBUTING document.
  • I have linked a related issue(延续 feat: remote environments for SSH and Docker execution targets (phase 1: creation-time binding) #4062 的远程环境实现)。
  • I have added tests that prove my feature works.
  • The behavior-change table above is complete, and every removed behavior or flipped default is named in the changeset and either has an escape hatch or was explicitly approved by a maintainer in this PR.
  • Ran gen-changesets skill:patch,已写入。
  • Ran gen-docs skill:性能优化没有新的用户操作或配置,无需更新用户文档;远程协议 README 已更新。

Copilot AI balanced review requested due to automatic review settings October 30, 2026 04:30
@changeset-bot

changeset-bot Bot commented Oct 30, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 3f497c5

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@moonshot-ai/kimi-code Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@7Hanrui

7Hanrui commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-30T04:38:06.538334Z 3f497c5 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@pkg-pr-new

pkg-pr-new Bot commented Oct 30, 2026

Copy link
Copy Markdown
pnpm dlx https://pkg.pr.new/@moonshot-ai/kimi-code@3f497c5
npx https://pkg.pr.new/@moonshot-ai/kimi-code@3f497c5

commit: 3f497c5

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3f497c5099

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

for (const file of filePaths) {
if (signal.aborted) {
const outOfTime = signal.aborted || Date.now() >= deadline;
if (outOfTime && (filesScanned > 0 || filePaths.length === 0)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Handle zero-file inventory timeouts before the scan loop

When ripgrep is unavailable and the batched walk reaches the 30-second deadline before yielding any file (for example, when files exist only beyond the partial inventory), filePaths is empty, so this loop and its only FS_GREP_TIMEOUT check never run. When the array is nonempty, this condition can only be entered after filesScanned > 0, making the nested zero-progress throw unreachable there as well; the Node fallback therefore returns a successful empty/truncated response instead of the timeout error for a zero-progress scan. Check the zero-progress timeout before entering the loop.

Useful? React with 👍 / 👎.

"@moonshot-ai/kimi-code": patch
---

Speed up file completion in remote environments without ripgrep.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Disclose the new fallback walk limits

The new WorkspaceFsService.walk applies 10,000-directory, 100,000-entry, 8 MiB, and per-directory 50,000-entry limits to local/ACP searches and Node-fallback suggest/grep, while the previous local recursive path had only the depth boundary. Large local workspaces can therefore now omit hits or fail with a too-many-results error, but this release note mentions only faster remote completion and does not name the behavior users lose; update it to disclose the truncation/error behavior (and any retained escape hatch).

AGENTS.md reference: AGENTS.md:L117-L117

Useful? React with 👍 / 👎.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants