Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
- In Dart, an annotation now links to the constant or constructor it names, read the way it is written: `@riverpod` reaches Riverpod's `const riverpod = Riverpod();`, `@Riverpod(keepAlive: true)` the `Riverpod` class, an annotation written through a class, like `@SomeClass.value` or `@SomeClass.named(…)`, that class's own constant, enum value or constructor, and `@meta.immutable` only what the `meta` import brings in. Before, an annotation was matched like a function decorator, so a method, getter or function that only shared its name won: every `@riverpod` provider in Riverpod's own repository was linked to an unrelated getter called `riverpod`, and `callers` and impact for the real annotation missed them. Re-index Dart projects after upgrading.
- In Dart, an annotation is no longer lost when a comment sits between it and the declaration, like `@override` followed by `// ignore: must_call_super`, a `///` comment written below the annotations, or `@riverpod` followed by an `// expect_lint: …` line. Before, the comment hid the annotation, so a provider written that way was missing from the annotation's `callers` and impact. Re-index Dart projects after upgrading.
- In Dart, a call written through an import prefix, like `http.get(…)` after `import 'package:http/http.dart' as http;`, or `utils.Listener()` and `kit.Box.create()` through a prefix of your own, now links to what the library imported with that prefix exports, following its `export` chains, `show` and `hide`, and to nothing when that library is a package outside your project. Before, the prefix was taken for a variable, so `http.get(…)` could link to an unrelated `Http` class's `get` method, and calls made through a prefix into your own libraries were missing from `callers` and impact. Re-index Dart projects after upgrading.
- `codegraph sync` and the file watcher now link an import to the file it names when that file is added after the importing file was indexed, or comes back after being deleted: `import 'package:app/b.dart';` in Dart, `import './side'` in JavaScript or TypeScript, `#include "net/socket.h"` in C and C++, `require 'inc/db.php'` in PHP, `import pkg.mod` in Python, a Lua `require`, or a C# `using` of a namespace declared later. Before, the import stayed unlinked until the importing file itself changed or the project was re-indexed, so `codegraph affected` and the files listed as using the new file missed it. Re-index after upgrading.

## [1.6.2] - 2026-10-03

Expand Down
425 changes: 425 additions & 0 deletions __tests__/sync-import-retry.test.ts

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions docs/viewer-launch-changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,8 @@ These describe `codegraph ui` and its screens. They were taken out of `## [Unrel

- **`codegraph ui` on Windows moves on from a port another program holds.** Windows refuses a port that another program keeps for itself, or one inside a range it reserves, with a different error than a port that is simply in use, so the viewer gave up at the first such port instead of trying the next one, and blamed ports below 1024 needing elevated privileges — a rule Windows doesn't have. It now moves on to the next free port, and when none is left, or the port you pinned with `--port` is refused, it says what Windows did and how to list the reserved ranges. Thanks @ijbranch for the report and @sx4im. (#2299)

- **A file's outside imports are listed in more languages.** The file screen lists the imports that name nothing in your project, like a third-party package or a system header. In C and C++, Objective-C, JavaScript and TypeScript, Python, C#, Lua, R, Solidity, Svelte and Vue, an import that named no file had been linked to its own import statement, so it was listed nowhere. Re-index after upgrading.

## Entries whose graph and `codegraph_explore` half already shipped

The release notes carry a reworded version of each of these. Keep only the viewer half at launch.
Expand Down
24 changes: 23 additions & 1 deletion src/db/migrations.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,12 @@
*/

import { SqliteDatabase } from './sqlite-adapter';
import { referenceNameTail } from './reference-tail';

/**
* Current schema version
*/
export const CURRENT_SCHEMA_VERSION = 11;
export const CURRENT_SCHEMA_VERSION = 12;

/**
* Migration definition
Expand Down Expand Up @@ -216,6 +217,27 @@ const migrations: Migration[] = [
`);
},
},
{
version: 12,
description: 'Retry a failed import when sync adds the file it names: path tails and failed-import indexes',
up: (db) => {
// Partial over failed imports, so both stay small. Keep the definitions
// in lockstep with schema.sql.
db.exec(`
CREATE INDEX IF NOT EXISTS idx_unresolved_failed_import_tail ON unresolved_refs(reference_kind, name_tail) WHERE status = 'failed' AND reference_kind = 'imports';
CREATE INDEX IF NOT EXISTS idx_unresolved_failed_import_name ON unresolved_refs(reference_kind, reference_name) WHERE status = 'failed' AND reference_kind = 'imports';
`);
// A path import parked before this version carries its dotted tail —
// the extension or a path fragment — which no file's keys match.
// Rewrite it to the path tail a failed import is parked under now.
// Idempotent: a rewritten tail rewrites to itself.
const update = db.prepare('UPDATE unresolved_refs SET name_tail = ? WHERE id = ?');
const rows = db
.prepare("SELECT id, reference_name FROM unresolved_refs WHERE status = 'failed' AND reference_kind = 'imports' AND reference_name LIKE '%/%'")
.all() as Array<{ id: number; reference_name: string }>;
for (const row of rows) update.run(referenceNameTail(row.reference_name, 'imports'), row.id);
},
},
];

/**
Expand Down
90 changes: 69 additions & 21 deletions src/db/queries.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import { kindBonus, nameMatchBonus, scorePathRelevance } from '../search/query-u
import { parseQuery, boundedEditDistance } from '../search/query-parser';
import { isGeneratedFile } from '../extraction/generated-detection';
import { splitIdentifierSegments } from '../search/identifier-segments';
import { referenceNameTail } from './reference-tail';

/**
* Files that should not be candidates for "dominant file" detection: test/spec
Expand Down Expand Up @@ -164,21 +165,6 @@ interface UnresolvedRefRow {
name_tail: string;
}

/**
* Last segment of a (possibly dotted/qualified) reference name — the part a
* new symbol's plain node name could match: 'util.greet' → 'greet',
* 'mod::fn' → 'fn', 'greet' → 'greet'. Written to unresolved_refs.name_tail
* when a ref is marked failed, so the #1240 retry lookup can match dotted
* refs against newly-added node names.
*/
function referenceNameTail(referenceName: string): string {
// Erlang refs carry a written arity (`f/1`, `mod::fn/2` — #1610); the tail a
// new symbol's plain name could match is the arity-less function name.
const base = referenceName.replace(/\/\d{1,3}$/, '') || referenceName;
const idx = Math.max(base.lastIndexOf('.'), base.lastIndexOf(':'));
return idx >= 0 ? base.slice(idx + 1) : base;
}

/**
* Convert database row to Node object
*/
Expand Down Expand Up @@ -3739,7 +3725,7 @@ export class QueryBuilder {
let changed = 0;
const markMany = this.db.transaction((items: typeof refs) => {
for (const ref of items) {
changed += stmt.run(referenceNameTail(ref.referenceName), ref.fromNodeId, ref.referenceName, ref.referenceKind).changes;
changed += stmt.run(referenceNameTail(ref.referenceName, ref.referenceKind), ref.fromNodeId, ref.referenceName, ref.referenceKind).changes;
}
});
markMany(refs);
Expand All @@ -3754,15 +3740,15 @@ export class QueryBuilder {
* can differ per call site (receiver-type inference reads the ref's line),
* so a sibling must not inherit this row's failure.
*/
markReferencesFailedByRowIds(refs: Array<{ rowId: number; referenceName: string }>): number {
markReferencesFailedByRowIds(refs: Array<{ rowId: number; referenceName: string; referenceKind: string }>): number {
if (refs.length === 0) return 0;
const stmt = this.db.prepare(
"UPDATE unresolved_refs SET status = 'failed', name_tail = ? WHERE id = ?"
);
let changed = 0;
const markMany = this.db.transaction((items: typeof refs) => {
for (const ref of items) {
changed += stmt.run(referenceNameTail(ref.referenceName), ref.rowId).changes;
changed += stmt.run(referenceNameTail(ref.referenceName, ref.referenceKind), ref.rowId).changes;
}
});
markMany(refs);
Expand Down Expand Up @@ -3824,6 +3810,66 @@ export class QueryBuilder {
}));
}

/**
* Failed `imports` refs a sync should retry once files appear. An import
* names a file, a folder or a namespace, which the symbol lookup above
* cannot match, so it is found two ways:
* - by `pathKeys` (`importPathKeys` of each added file), as its tail or its
* whole name: `package:app/b.dart` waits for a file named `b.dart`, a
* bare `#include "b.h"` or `require 'db.php'` is the file's whole name;
* - by `names` (the changed files' namespaces and modules), as its whole
* name when its tail is something else: a C# `using Foo.Bar` waits for
* the namespace node `Foo.Bar`. An import whose tail is the name is the
* symbol lookup's, under that lookup's ceiling.
* Same per-name ceiling as {@link getRetryableFailedReferences}, counted
* per key.
*/
getRetryableFailedImports(pathKeys: string[], names: string[] = [], perNameCeiling: number = 500): UnresolvedReference[] {
const failedImports = "status = 'failed' AND reference_kind = 'imports'";
const lookups = [
{ keys: pathKeys, column: 'name_tail', where: failedImports },
{ keys: pathKeys, column: 'reference_name', where: failedImports },
{ keys: names, column: 'reference_name', where: `${failedImports} AND name_tail != reference_name` },
];
const rows = new Map<number, UnresolvedRefRow>();
for (const { keys, column, where } of lookups) {
const unique = [...new Set(keys)].filter((key) => key.length > 0);
// Pass 1: per-key counts, chunked under the SQLite parameter limit.
const retryKeys: string[] = [];
for (let i = 0; i < unique.length; i += SQLITE_PARAM_CHUNK_SIZE) {
const chunk = unique.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
const placeholders = chunk.map(() => '?').join(',');
const counts = this.db
.prepare(`SELECT ${column} AS key, COUNT(*) AS count FROM unresolved_refs WHERE ${where} AND ${column} IN (${placeholders}) GROUP BY ${column}`)
.all(...chunk) as Array<{ key: string; count: number }>;
for (const row of counts) {
if (row.count <= perNameCeiling) retryKeys.push(row.key);
}
}
// Pass 2: load the surviving rows; a row two lookups find is kept once.
for (let i = 0; i < retryKeys.length; i += SQLITE_PARAM_CHUNK_SIZE) {
const chunk = retryKeys.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
const placeholders = chunk.map(() => '?').join(',');
const chunkRows = this.db
.prepare(`SELECT * FROM unresolved_refs WHERE ${where} AND ${column} IN (${placeholders})`)
.all(...chunk) as UnresolvedRefRow[];
for (const row of chunkRows) rows.set(row.id, row);
}
}

return [...rows.values()].map((row) => ({
fromNodeId: row.from_node_id,
referenceName: row.reference_name,
referenceKind: row.reference_kind as EdgeKind,
line: row.line,
column: row.col,
candidates: row.candidates ? safeJsonParse(row.candidates, undefined) : undefined,
filePath: row.file_path,
language: row.language as Language,
rowId: row.id,
}));
}

/**
* Resolution edges whose TARGET symbol is named one of `names` — the edges a
* sync must re-resolve after `names` gained or lost a definition (CG-33).
Expand Down Expand Up @@ -3935,16 +3981,18 @@ export class QueryBuilder {
/**
* Distinct node names present in the given files — the symbol names a sync
* pass uses to look up retryable failed refs after those files changed.
* `kinds` narrows them to nodes of those kinds.
*/
getNodeNamesByFiles(filePaths: string[]): string[] {
getNodeNamesByFiles(filePaths: string[], kinds?: readonly NodeKind[]): string[] {
if (filePaths.length === 0) return [];
const kindFilter = kinds ? ` AND kind IN (${kinds.map(() => '?').join(',')})` : '';
const names = new Set<string>();
for (let i = 0; i < filePaths.length; i += SQLITE_PARAM_CHUNK_SIZE) {
const chunk = filePaths.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
const placeholders = chunk.map(() => '?').join(',');
const rows = this.db
.prepare(`SELECT DISTINCT name FROM nodes WHERE file_path IN (${placeholders})`)
.all(...chunk) as Array<{ name: string }>;
.prepare(`SELECT DISTINCT name FROM nodes WHERE file_path IN (${placeholders})${kindFilter}`)
.all(...chunk, ...(kinds ?? [])) as Array<{ name: string }>;
for (const row of rows) names.add(row.name);
}
return [...names];
Expand Down
57 changes: 57 additions & 0 deletions src/db/reference-tail.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
/**
* The keys sync's failed-ref retry (#1240) matches on: the tail a failed ref
* is parked under, and the names a newly added file can be imported as.
* Shared by the query layer, which writes and reads them, and the migration
* that rewrote the tails parked before path imports had one of their own.
*/

/**
* Last segment of a (possibly dotted/qualified) reference name — the part a
* new symbol's plain node name could match: 'util.greet' → 'greet',
* 'mod::fn' → 'fn', 'greet' → 'greet'. Written to unresolved_refs.name_tail
* when a ref is marked failed, so the #1240 retry lookup can match dotted
* refs against newly-added node names.
*
* An import written as a path names a FILE, so its tail is the path's last
* segment cut at its first dot — `package:app/b.dart` → 'b', `./foo` → 'foo',
* `inc/db.php` → 'db' — the stem {@link importPathKeys} takes from a file
* that could satisfy it. The dotted tail was the extension ('dart', 'php') or
* a path fragment ('/foo'), which no file ever matched.
*/
export function referenceNameTail(referenceName: string, referenceKind?: string): string {
if (referenceKind === 'imports') {
const path = referenceName.replace(/\/+$/, '');
const slash = path.lastIndexOf('/');
const stem = slash >= 0 ? fileNameStem(path.slice(slash + 1)) : '';
// `..` and `.` name a folder only through the importing file's location.
if (!/^\.*$/.test(stem)) return stem;
}
// Erlang refs carry a written arity (`f/1`, `mod::fn/2` — #1610); the tail a
// new symbol's plain name could match is the arity-less function name.
const base = referenceName.replace(/\/\d{1,3}$/, '') || referenceName;
const idx = Math.max(base.lastIndexOf('.'), base.lastIndexOf(':'));
return idx >= 0 ? base.slice(idx + 1) : base;
}

/** A file or folder name up to its first dot: 'b.dart' → 'b', 'types.d.ts' → 'types', '.eslintrc.js' → '.eslintrc'. */
function fileNameStem(name: string): string {
const dot = name.indexOf('.', 1);
return dot > 0 ? name.slice(0, dot) : name;
}

/**
* What an import of the file at `filePath` can be written as, for the failed
* imports a sync that adds the file retries: its name (`b.h`, a bare include)
* and stem (`b`, the tail of `a/b.h`, `./b`, `pkg.b`), and the same two for
* its folder, which an import names for its index file (`./ui` →
* `ui/index.ts`, `pkg` → `pkg/__init__.py`) or as a Go package.
*/
export function importPathKeys(filePath: string): string[] {
const parts = filePath.split('/');
const keys = new Set<string>();
for (const name of parts.slice(-2)) {
keys.add(name);
keys.add(fileNameStem(name));
}
return [...keys];
}
6 changes: 6 additions & 0 deletions src/db/schema.sql
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,12 @@ CREATE INDEX IF NOT EXISTS idx_unresolved_file_path ON unresolved_refs(file_path
CREATE INDEX IF NOT EXISTS idx_unresolved_from_name ON unresolved_refs(from_node_id, reference_name);
CREATE INDEX IF NOT EXISTS idx_unresolved_status ON unresolved_refs(status);
CREATE INDEX IF NOT EXISTS idx_unresolved_failed_tail ON unresolved_refs(name_tail) WHERE status = 'failed';
-- Sync's failed-import retry looks failed imports up by tail and by whole
-- name. The leading reference_kind is what makes the planner pick these over
-- idx_unresolved_failed_tail / idx_unresolved_name, whose per-key averages
-- hide how many failed calls a common tail (`index`, `types`) has.
CREATE INDEX IF NOT EXISTS idx_unresolved_failed_import_tail ON unresolved_refs(reference_kind, name_tail) WHERE status = 'failed' AND reference_kind = 'imports';
CREATE INDEX IF NOT EXISTS idx_unresolved_failed_import_name ON unresolved_refs(reference_kind, reference_name) WHERE status = 'failed' AND reference_kind = 'imports';
CREATE INDEX IF NOT EXISTS idx_edges_provenance ON edges(provenance);
-- Sync's third-file wiring lookup must not scan every synthesized edge.
-- CASE short-circuits malformed metadata; keep these expressions identical
Expand Down
5 changes: 5 additions & 0 deletions src/extraction/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,8 @@ export interface SyncResult {
*/
durationMs: number;
changedFilePaths?: string[];
/** The part of `changedFilePaths` that was not indexed before this sync. */
addedFilePaths?: string[];
/** Paths not absorbed because reading or extraction failed; retain for status/retry. */
failedFilePaths?: string[];
/**
Expand Down Expand Up @@ -3302,6 +3304,7 @@ export class ExtractionOrchestrator {
let filesRemoved = 0;
let nodesUpdated = 0;
const changedFilePaths: string[] = [];
const addedFilePaths: string[] = [];
// `file\0name` definition pairs for the files this sync touches, sampled
// BEFORE their nodes are replaced/deleted. Compared against the post-store
// pairs below to derive `definitionDelta` (CG-33).
Expand Down Expand Up @@ -3476,6 +3479,7 @@ export class ExtractionOrchestrator {
onFileChange?.(filePath, content);
filesToIndex.push(filePath);
changedFilePaths.push(filePath);
addedFilePaths.push(filePath);
filesAdded++;
} else if (tracked.contentHash !== contentHash || neverParsed) {
onFileChange?.(filePath, content);
Expand Down Expand Up @@ -3545,6 +3549,7 @@ export class ExtractionOrchestrator {
nodesUpdated,
durationMs: Date.now() - startTime,
changedFilePaths: changedFilePaths.length > 0 ? changedFilePaths : undefined,
...(addedFilePaths.length > 0 ? { addedFilePaths } : {}),
...(failedFilePaths.length > 0 ? { failedFilePaths } : {}),
definitionDelta: definitionDelta.length > 0 ? definitionDelta : undefined,
};
Expand Down
15 changes: 15 additions & 0 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ import {
import { DatabaseConnection, getDatabasePath, removeDatabaseFiles } from './db';
import { WalCheckpointValve, resolveWalValveMb } from './db/wal-valve';
import { QueryBuilder } from './db/queries';
import { importPathKeys } from './db/reference-tail';
import {
isInitialized,
createDirectory,
Expand Down Expand Up @@ -989,6 +990,20 @@ export class CodeGraph {
const retryable = this.queries.getRetryableFailedReferences(
this.queries.getNodeNamesByFiles(result.changedFilePaths)
);
// A failed import waits for a file, a folder or a namespace, not a
// symbol: `package:app/b.dart` for a file named `b.dart`, `./ui`
// for `ui/index.ts`, `using Foo.Bar` for that namespace's node.
// Look those up by what the ADDED files can be imported as (a
// modified file was already there for any import of it), and by
// the namespaces and modules the changed files declare.
const retryRows = new Set(retryable.map((ref) => ref.rowId));
const importRetry = this.queries.getRetryableFailedImports(
(result.addedFilePaths ?? []).flatMap(importPathKeys),
this.queries.getNodeNamesByFiles(result.changedFilePaths, ['namespace', 'module'])
);
for (const ref of importRetry) {
if (!retryRows.has(ref.rowId)) retryable.push(ref);
}
if (retryable.length > 0) {
options.onProgress?.({
phase: 'resolving',
Expand Down
Loading