Two index-currency signals report "current" when the index is not: sync ignores isIndexStale(), and status.pendingChanges only hash-checks what git nominates #1

Activity

1. inth3shadows commented on Sep 10, 2026

   

   inth3shadows

   on Sep 10, 2026

   OwnerAuthor

   More actions

   Both mechanisms verified against source — one confirmed as written, one needs a correction

   Read against v1.6.0-em.1 (c18d6fa), whose src/ is upstream 1.6.0 plus one Python resolver patch that touches neither of these paths.

   A — confirmed, and the codebase already documents the invariant it violates

   EXTRACTION_VERSION has exactly three functional consumers in the whole tree:

   src/index.ts:694          write it to metadata at index time
   src/index.ts:1233         isIndexStale() compares it
   src/bin/codegraph.ts:1020 report it in status --json
   

   isIndexStale() in turn has two callers — src/bin/codegraph.ts:985 (the status command) and src/ui-server/api/stats.ts:113. Neither is sync, and its own docstring scopes it deliberately:

   This is the signal behind codegraph status's re-index hint and codegraph upgrade's reminder.

   The stronger finding is at src/index.ts:686, where the stamp is written:

   // Stamp the index with the engine that built it, so `codegraph status`
   // and `codegraph upgrade` can recommend a re-index when the running
   // engine produces richer extraction than the one on disk. Only on a
   // real full index — a sync touches a subset, so it must NOT advance the
   // extraction stamp (the bulk would still be stale). See extraction-version.ts.

   So the invariant is already understood and written down: a sync leaves the bulk stale. The code deliberately withholds the stamp for exactly that reason, computes isIndexStale() from it, and surfaces reindexRecommended. Sync then does nothing with any of it. The gap is enforcement, not knowledge — which makes the fix smaller than the original report implied.

   extraction-version.ts also names this precise case as a reason to bump:

   BUMP THIS when a release changes extraction output enough that existing indexes should be rebuilt to benefit — e.g. a new language/framework extractor, a new dynamic-dispatch synthesizer, a new node/edge kind, or a resolver fix that materially changes which edges exist.

   Which is what happened: the resolver fix bumped 25 → 26, the index sat at 25, reindexRecommended was true, and sync re-extracted only edited files.

   B — correction: pendingChanges is not the git diff, but its candidate set is

   My original description said the field is computed from the working-tree diff. That is wrong in a way worth fixing before this goes upstream. getChangedFiles() (src/extraction/index.ts:2935) does hash-compare against the DB and the comparison is correct. The defect is one level up — where the candidates come from:

   const gitChanges = getGitChangedFiles(this.rootDir);
   if (gitChanges) {
     // === Git fast path ===
     ...
     for (const filePath of [...gitChanges.modified, ...gitChanges.added]) {
       const contentHash = hashContent(content);
       const tracked = this.queries.getFileByPath(filePath);
       if (!tracked) added.push(filePath);
       else if (tracked.contentHash !== contentHash) modified.push(filePath);
     }

   and getGitChangedFiles (:1107) builds that list from

   git status --porcelain --no-renames -uall
   

   A committed file has no porcelain entry, so it never enters the loop and is never hash-compared — regardless of what the DB holds. Committing is what removes a file from the candidate set. The hash check is sound; it is simply never reached.

   Two details that matter for the fix:

   • The non-git fallback below (:2979, "full scan") walks scanDirectory and compares against getAllFiles(). That path is correct. So this is specific to the git fast path, and a correct implementation already exists in the same function.
   • There is a comment at :2952 recording that untracked (??) files had to be hash-compared rather than always counted as added, "otherwise status reports them as pending forever (see issue codegraph status still reports untracked files as pending after sync colbymchenry/codegraph#206)". That is the same class of bug — git state standing in for index state — patched for one case without generalizing.

   Revised fix shape

   • A: have sync consult isIndexStale() and either force a full re-extract or refuse with a pointer to index. The signal, the threshold and the intent are all already in the tree.
   • B: in the git fast path, union the git candidate set with the DB's own file list (getAllFiles()), so a file the index tracks is hash-checked whether or not git currently reports it — and a file the index does NOT track is caught by a scan rather than by git's opinion. The fallback path already demonstrates the comparison; the fast path just needs to stop treating "git says nothing changed" as "nothing changed".

   Both repros in the issue body still stand unchanged; only the explanation of B's cause has been corrected.
2. inth3shadows

   changed the title [-]Two index-currency signals report "current" when the index is not: sync ignores extraction-version drift, and status.pendingChanges reads the git working tree[/-] [+]Two index-currency signals report "current" when the index is not: sync ignores isIndexStale(), and status.pendingChanges only hash-checks what git nominates[/+] on Sep 10, 2026
3. inth3shadows commented on Sep 10, 2026

   

   inth3shadows

   on Sep 10, 2026

   OwnerAuthor

   More actions

   Filed upstream and fixed — with one correction to the revised fix shape above

   Upstream: half A as a reproduced case on colbymchenry#1798 (that issue already proposed background rebuilds off the same foundation, so this went as a comment rather than a near-duplicate); half B as colbymchenry#1829.

   Re-verified against current upstream/main @ 3ed73bc before filing: getChangedFiles, isIndexStale and collectGitStatus are byte-identical to the v1.6.0-em.1 versions read above (hashed at both revs), so none of the analysis had drifted. Both halves were also reproduced live twice — on the published 1.6.0 bundle and on a source build of 3ed73bc — including a sharper demonstration of A: with the stamp at 25 and the engine at 26, codegraph sync prints Already up to date about an index the same tool reports as reindexRecommended: true.

   Correction: unioning with getAllFiles() is not sufficient for B

   The revised fix shape above says to union the git candidate set with the DB's file list. That catches a committed modification to a file the index already tracks — but not a committed new file, which is the first case in this issue's own repro. The DB has no row for pkg/two.py, so it is not in getAllFiles() either, and it stays invisible.

   What was implemented instead: the index stamps the commit it was built at, and the fast path adds git diff --name-status <stamp> HEAD to its candidates. That covers added, modified and deleted in one cheap call, and it is the only cheap way to catch a committed modification — the alternative is hashing every tracked file on every status.

   • The stamp is written by a full index and by every whole-tree sync (a --paths sync absorbs only part of the diff, so it must not stamp), captured before change detection runs so a commit landing mid-sync is never claimed as absorbed.
   • Committed lines run through the same filter working-tree lines do — an excluded dir stays excluded (Incremental sync (git-status fast path) bypasses the ignore matcher — tracked files in excluded dirs leak back into the index colbymchenry/codegraph#766, codegraph index wedges indefinitely at "Resolving refs" 94% on a repo with a large *tracked* JS/TS asset dir and there's no way to exclude it colbymchenry/codegraph#999).
   • A stamp git cannot resolve (rebase, gc, shallow clone) falls back to the full scan, as does an index built before stamping existed; the next sync stamps it and the fast path returns.

   For A, the enforcement point is where you said it was: codegraph sync now re-indexes in full when the extractor moved and says why, --no-upgrade keeps incremental with a warning, and -q (git hooks) stays incremental so a commit is never held up by a rebuild. SyncResult.staleEngine reports it to library callers rather than forcing a surprise rebuild on the watcher.

   Branch: claude-stop-reporting-index-current-1829 (d14fda7) — 9 new tests, 5 of which fail against pre-fix source. Full suite clean: the 13 remaining failures reproduce on pristine upstream/main @ 3ed73bc in a throwaway worktree against this machine's own index.
4. inth3shadows commented on Sep 26, 2026

   

   inth3shadows

   on Sep 26, 2026

   OwnerAuthor

   More actions

   Both halves now ship on this fork's main (22a3f5e):

   • B (committed-but-unindexed changes reported as current): upstream main now stamps the commit an index was built at (INDEXED_AT_COMMIT_KEY in src/extraction/index.ts), the stamp-and-diff approach described above. The fork inherits it from upstream ba3c21e.
   • A (sync ignores extraction staleness): carried from upstream PR fix(cli): reject sync for stale extraction indexes colbymchenry/codegraph#1842 (@bompus, merged here as c66dcd2). sync now refuses a stale index and points at codegraph index instead of printing "Already up to date". One fork-only change: that PR's quiet-mode test now launches the CLI with the runtime flags production uses (22a3f5e), because with fix(directory): a schema-less codegraph.db does not make a project initialized (#1895) colbymchenry/codegraph#1913 also carried it picked up node:sqlite's ExperimentalWarning.

   Reopen if either reproduces on the fork's main.
5. inth3shadows

   closed this as completedon Sep 26, 2026