Interrupted bulk-load recovery can be repeatedly watchdog-killed while rebuilding indexes #1887

Activity

1. danusha2345

   added a commit that references this issue on Sep 26, 2026

   Merge codex/integration-all-2026-09-26 (colbymchenry#1959, colbymchen…

   ...

   eda130a
2. WQMYH commented on Oct 5, 2026

   

   WQMYH

   on Oct 5, 2026 – with ChatGPT Codex Connector

   More actions

   Additional Windows field evidence: ~14–16 GiB MCP-process memory spikes with an incomplete large index

   Adding measured field evidence that appears consistent with this issue, including the memory-pressure impact. This is a source/state correlation, not a captured runtime stack proving which CREATE INDEX was executing.

   Environment

   • CodeGraph 1.6.0, official npm Windows x64 distribution (@colbymchenry/codegraph and @colbymchenry/codegraph-win32-x64).
   • Windows 11 25H2, build 26200.9457, x64, approximately 32 GiB RAM.
   • MCP subprocess launched by the Codex desktop/app-server, with codegraph serve --mcp and no explicit --path.
   • Measurements collected on 2026-10-05. No termination, configuration change, upgrade or index rebuild was performed during this investigation.

   The same live process repeatedly changed size

   Windows process counters showed the following for one continuously running CodeGraph node.exe process; its PID and creation time stayed unchanged:

   Relative sample time	Private committed memory	Working set
   0:00	14.46 GiB	14.07 GiB
   1:52	1.41 GiB	1.30 GiB
   3:05	5.92 GiB	5.68 GiB
   12:06	13.80 GiB	13.45 GiB
   17:18	15.61 GiB	15.24 GiB

   At the last sample, system committed memory was 43,093,766,144 / 43,892,195,328 bytes (~98.2%), with about 3,000 MiB available physical memory. The affected process had been alive for over seven hours.

   This is user-mode CodeGraph memory, not the separate Windows NtFC kernel-pool problem: NtFC nonpaged bytes stayed around 58,672 bytes, and total nonpaged pool was about 1 GiB. The substantial drop and subsequent regrowth also mean these observations alone do not establish an unreclaimable memory leak.

   Read-only database and process inspection

   A read-only process-directory probe identified the large workspace as this process's working directory. Duplicated file-handle inspection confirmed that it actually held the large workspace's codegraph.db, WAL and SHM open. The same process also held a second, small healthy project's database open.

   The large default workspace includes repository snapshots and third-party source corpora:

   • Main database: 14,978,273,280 bytes (~13.95 GiB).
   • files: 174,369 rows.
   • nodes: 2,823,016 rows.
   • unresolved_refs: 11,200,451 rows.
   • project_metadata.index_state: indexing, last updated 2026-09-29.
   • All three FTS maintenance triggers were present.
   • The following nine required secondary indexes were absent in two inspections several minutes apart:

   idx_unresolved_from_node
   idx_unresolved_name
   idx_unresolved_file_path
   idx_unresolved_from_name
   idx_unresolved_failed_tail
   idx_edges_kind
   idx_edges_source_kind
   idx_edges_target_kind
   idx_edges_provenance
   

   For comparison, the second open database was only 16,846,848 bytes, had 133 file rows, state complete, and all these indexes present.

   Database observations used SQLite URI mode=ro, PRAGMA query_only=ON and bounded query execution; they did not open the database through CodeGraph's healing path.

   Recovery / watchdog evidence

   The large workspace's existing daemon log contains repeated daemon starts followed by:

   Main thread unresponsive for ~60s — killing the wedged process so a fresh one can start (#850).
   

   This pattern recurred across multiple days. In the latest inspected pair, the daemon started and was watchdog-terminated about 60 seconds later.

   Inspection of the installed 1.6.0 implementation, rather than assuming current main matches it, found:

   • DatabaseConnection.open() synchronously invokes healBulkSecondaryIndexes().
   • That method loops over the required indexes and executes each missing CREATE INDEX via synchronous db.exec, without yielding.
   • Connection configuration sets temp_store = MEMORY; its configured page cache is only about 64 MB and mmap is 256 MB.
   • The MCP proxy can fall back to a local in-process engine and replay in-flight calls when the shared daemon disappears. That is a possible amplification path; the exact fallback event for the high-memory process was not separately captured.

   These conditions strongly match interrupted-index recovery, with a very large corpus amplifying its memory and I/O cost. However, no live CPU/heap stack or SQLite statement trace was collected, so attribution of each observed memory peak to a particular rebuild/sort remains a hypothesis.

   Could recovery validation include memory peaks on a large incomplete graph, in addition to watchdog liveness, and whether daemon-loss fallback can cause multiple clients to independently repeat recovery? PR #1888 appears relevant; this report does not claim that a released version already fixes this case.

   Only this sanitized summary is shared: no usernames, absolute local paths, project names, raw logs, screenshots, database files or dumps are attached.
3. AxiomOperator

   added a commit that references this issue on Oct 5, 2026

   Phase 2: reliability and crash safety

   ...

   a90852a