Large CSV auto-indexing can exhaust memory and trigger repeated worker recovery (Windows, v0.11.0) #2908

Summary

Opening a workspace in VS Code caused CBM to start indexing automatically, without clicking the extension's Index command. Large CSV data exports entered the parallel extraction pipeline. Existing worker logs show memory-budget failures, allocator failures, and repeated crash-recovery attempts. The machine became unresponsive; uninstalling the navigation extension stopped the startup trigger.

Please consider skipping large CSV data exports by default, or handling them with a strictly bounded metadata/header-only path. These files are data exports rather than source code for navigation.

Relationship to #1084

I also participated in #1084. I saw that it was closed after #2022 adjusted background-indexing CPU concurrency. This report adds a specific workload and memory/recovery evidence: large CSV exports in the initial automatic index, with no existing index database.

The evidence below is from v0.11.0. I have not retested this workload on v0.12.1 or current main, so this is not a claim that the #2022 fix regressed or failed.

Source review of #2022

Reviewing merged commit 29aa74867bd8ac80a5482156dc3e555904b7a382, the background-policy gate requires both the internal background flag and an already servable, committed project index. The helper above it deliberately leaves a project's first automatic index at full worker width.

The new tests mcp_auto_index_in_process_fresh_project_uses_full_width and mcp_background_flag_fresh_project_uses_full_width explicitly preserve that first-index behavior.

The incident here logged pipeline.route path=full reason=no_existing_db, so its initial-index worker policy is outside that CPU-headroom gate. Reviewing all 13 changed files, #2022 adjusts concurrency policy and LSP surface-build worker accounting; it does not introduce a large-CSV exclusion or change memory-failure recovery policy.

This source review narrows the question to safe handling of bulk CSV exports during the initial automatic index. It does not establish runtime behavior after other changes in newer builds.

Environment and workload

• Version recorded in the worker-start log: 0.11.0.
• Platform: Windows x64; approximately 16 GB RAM (total_ram_mb=16110).
• Client: VS Code with CBM Navigator, which opens a stdio MCP session at startup. The installed extension version was not recorded.
• Install channel and binary variant: not independently verified.
• Configuration read during investigation: auto_index=true, auto_index_limit=50000, auto_watch=true, watcher_enabled=true.
• A mixed workspace containing code/documents and bulk CSV exports; discovery found 538 files.
• CSV extraction-start records included files of 199922, 146819, 136782, 100040, and 92899 KiB (roughly 91-195 MiB). These are individual file sizes, not total workspace size.
• The worker selected full indexing because there was no existing database, and used 16 parallel extraction workers.

Observed workflow

1. Open the workspace in VS Code while automatic indexing is enabled.
2. The extension initializes its CBM MCP connection; no manual Index command is issued.
3. CBM automatically starts the initial full index, including large CSV data files.
4. Memory use exceeds the budget. Worker crashes and recovery/quarantine attempts follow.

The existing logs contain 24 worker-start records for this same workspace across approximately 316 seconds, based on the worker-log modification-time window. Recovery logs show repeated failures and CSV quarantine attempts. This is an observed finite window; subsequent recovery behavior was not observed.

A Task Manager screenshot during the reported problem showed approximately 90% CPU and 4 GB memory for the CBM process group. That snapshot and the log excerpts below are separate observations, not a synchronized CPU/RSS trajectory.

Sanitized diagnostic excerpts

From one worker attempt:

level=info msg=mem.init budget_mb=4027 total_ram_mb=16110 source=ram_fraction
level=info msg=pipeline.discover files=538 elapsed_ms=220
level=info msg=pipeline.route path=full reason=no_existing_db
level=info msg=pipeline.mode mode=parallel workers=16 files=538
level=info msg=parallel.extract.start files=538 workers=16
level=info msg=parallel.extract.retention retain_sources=true total_mb=503 per_file_mb=32
level=info msg=parallel.mem.budget total_mb=4027 per_worker_mb=251
level=info msg=mem.spill.sweep parked=1 charged_mb=5989
level=error msg=mem.budget.exceeded rss_mb=5861 budget_mb=4027 phase=parallel_extract action=fail_whole

Supervisor/recovery records from other attempts in the same incident (paths replaced with placeholders):

level=warn msg=index.supervisor.worker_memory job_limit_bytes=6335018496 peak_job_memory_bytes=6364422144
level=warn msg=index.supervisor.worker_failed outcome=crash exit_code=-1073741819 log=<redacted>
level=warn msg=daemon.index.file_quarantined path=<workspace>/data/export.csv outcome=crash attempt=5

Another attempt also logged mimalloc allocation failures and Windows error 1455 (0x5AF, cannot commit OS memory). The CSV files were entering the extraction phase while these resource failures occurred; the precise per-file allocation source has not been profiled.

Expected behavior / requested handling

• Preserve useful small CSV support, but skip bulk CSV exports above a reasonable size by default, or inspect only a bounded prefix/header without parsing every row. An explicit opt-in could cover users who intentionally want large CSV content indexed.
• Apply per-file/source-byte safeguards before expensive extraction and parallel allocation. A file-count threshold alone does not protect a workspace with only 538 files and several very large data exports.
• Treat memory-budget exhaustion as a resource failure with a clear terminal result or backoff. Recovery should avoid rapidly repeating expensive attempts across many large CSV files.

Reproduction and privacy limits

This report is based on an actual local incident and retained diagnostics, not a validated public minimal reproducer. No proprietary source, CSV rows, original filenames, workspace names, user paths, machine identifiers, or unredacted log attachments are included. The paths above are placeholders.

A CBM_DIAGNOSTICS trajectory was not captured at the time. I have not deliberately reproduced the OS lockup or rerun the workload on a newer release. The report is intended to provide the existing evidence safely, consistent with the guidance in #1084.