diff --git a/CHANGELOG.md b/CHANGELOG.md index ae8b7b2c7..04df20b0f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -68,6 +68,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - JSON files in a `templates` or `sections` folder are now read as Shopify Liquid only inside a Shopify theme, where the folder that contains `templates` or `sections` also has `layout/theme.liquid` or `config/settings_schema.json`. Before, all such JSON was, so a .NET project template's `template.json`, a schematic or a CMS's content showed up in the index as an empty Liquid file, and one that lists `sections` could be linked to an unrelated Liquid file of the same name. Shopify themes, including one kept in a subfolder of a repository, are indexed as before, and the next sync removes JSON that an earlier version indexed this way. - In a repository that holds more than one Shopify theme, a theme's `{% render %}`, `{% include %}` and `{% section %}` tags and the sections its JSON templates list now link only to that theme's own snippets and sections, which is where Shopify looks for them. Before, a section or snippet the theme doesn't have was linked to another theme's file of the same name, and a theme kept in a subfolder could link to the root theme's copy instead of its own. A reference the theme can't satisfy now links to nothing. Projects with a single theme are indexed as before. Re-index projects with several Shopify themes after upgrading. - `codegraph sync` and the file watcher now link a Shopify section or snippet to the files that name it when it is added after them, or comes back after being deleted, whether they name it in a `{% render %}`, `{% include %}` or `{% section %}` tag or as a section `type` in a JSON template. Before, the link appeared only once a file naming it changed or the project was re-indexed, even in a repository with a single theme, so the new section or snippet looked unused and `codegraph affected` missed the files that use it. Re-index Shopify themes after upgrading to pick up links an earlier version missed. +- `codegraph sync` and the file watcher now link a route to the page or layout it loads lazily when that file is added after the router was indexed, comes back after being deleted, or gets the component the route renders in a later edit, whether or not the import names the file's extension: React Router's `lazy: () => import('./pages/Team')`, Vue Router's `component: () => import('@/views/Login')` and Angular's `loadComponent: () => import('./home/home.component')`. Before, the route stayed unlinked until the router file changed or the project was re-indexed, so the page looked unused and `codegraph affected` missed the tests that reach it through the router. Re-index after upgrading to pick up links an earlier version missed. - In C, C++ and Objective-C, a struct or class member declared as a pointer, reference, array or function, like `SharedState* shared;`, `jv elements[];` or `virtual Status Put(…) = 0;`, no longer makes the struct look like it inherits from the member's type. Before, `codegraph_explore` and impact listed those types as base classes, every struct holding a pointer to a type as one of its subclasses, and a method of that struct as an override of the type's method with the same name. Real base classes like `class Derived : public Base`, Objective-C superclasses and Go's embedded fields are unchanged. Re-index C, C++ and Objective-C projects after upgrading. - In C++, a call on a variable, parameter or member declared through a `typedef` or `using` alias now links to the method of the type the alias names, with the alias looked up the way C++ does it: in the calling function, its class and the classes that class inherits from, then the namespaces around it. In google/leveldb, `table_.Insert(buf)` on a `Table table_;` declared next to `typedef SkipList Table;` now reaches `SkipList::Insert` instead of the unrelated `HandleTable::Insert`, and `Table::Iterator iter(&table_);` reaches `SkipList::Iterator`'s constructor. A class that only shares the alias's name is no longer taken for it, and when the aliased type has no such method, as with an alias of `std::vector` or of a template parameter, no method is guessed from the receiver's name. Re-index C++ projects after upgrading. - In C++, a base class like `Message` in `class DynamicMessage final : public Message` now links to the class C++ itself would pick: the one declared in the namespaces around the class, or brought in by the file's `using` declarations and namespace aliases, and the class a `typedef` or `using` alias names. Before, a base class was matched by its name alone, so it could land on a class of the same name in another namespace or another language: in protocolbuffers/protobuf, `DynamicMessage` derived from the JSON parser's internal `Message` class, and in google/leveldb and RocksDB, iterators derived from an unrelated nested `Iterator` class such as the skip list's. The wrong base showed up in `codegraph_explore`'s type hierarchy and in impact, and tied the base's methods to the wrong overrides. A template parameter used as a base no longer links to a class that shares its name, and a base written from the global scope, like `public ::testing::Test`, now links to its class. Re-index C++ projects after upgrading. diff --git a/__tests__/sync-path-reference-retry.test.ts b/__tests__/sync-path-reference-retry.test.ts index a6c50a097..f6953cb0e 100644 --- a/__tests__/sync-path-reference-retry.test.ts +++ b/__tests__/sync-path-reference-retry.test.ts @@ -44,13 +44,14 @@ describe('referenceNameTail', () => { expect(referenceNameTail('snippets/price.liquid', 'references')).toBe('price.liquid'); expect(referenceNameTail('sections/404.liquid', 'references')).toBe('404.liquid'); expect(referenceNameTail('snippets/icon.logo.liquid', 'references')).toBe('icon.logo.liquid'); - expect(referenceNameTail('lazy-import:./routes/about.tsx', 'references')).toBe('about.tsx'); }); it('keeps the symbol tail of a reference that names no file', () => { - // No path, a path with no file name at its end, or an Erlang arity. + // No path, a path with no file name at its end, or an Erlang arity. (A + // route's module, `lazy-import:./routes/about.tsx` or + // `import:./home.component#HomeComponent`, is sync-route-module-retry's.) expect(referenceNameTail('Foo.Bar', 'references')).toBe('Bar'); - expect(referenceNameTail('import:./home.component#HomeComponent', 'references')).toBe('component#HomeComponent'); + expect(referenceNameTail('snippets/price', 'references')).toBe('snippets/price'); expect(referenceNameTail('lists::map/2', 'references')).toBe('map'); // A call's slashes sit in its arguments, a comment or a division. expect(referenceNameTail('assert.logfile("logs/error.log").has.line', 'calls')).toBe('line'); diff --git a/__tests__/sync-route-module-retry.test.ts b/__tests__/sync-route-module-retry.test.ts new file mode 100644 index 000000000..2e138329f --- /dev/null +++ b/__tests__/sync-route-module-retry.test.ts @@ -0,0 +1,454 @@ +/** + * A sync links a route to the module it lazily loads once the module appears. + * + * A router names a lazily loaded page or layout by the module's path: React + * Router's `lazy: () => import('./pages/Team')` is the reference + * `lazy-import:./pages/Team`, Vue Router's `component: () => + * import('@/views/Login')` is `import:@/views/Login#Login`, and Angular's + * `loadComponent: () => import('./home/home.component')` is + * `import:./home/home.component#default`. Sync retries a parked failed ref by + * its tail (#1240), and these were parked under a fragment of the path — + * `/pages/Team`, `@/views/Login#Login`, `component#default` — that no file's + * keys match. So a route whose module was added after the router was indexed + * stayed unlinked until the router file changed or the project was indexed + * again, and so did one whose module gained its component in a later edit. + * #2392 fixed the same gap for imports, #2403 for Liquid's path references. + */ +import { describe, it, expect, afterEach, vi } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; +import { DatabaseConnection, getDatabasePath } from '../src/db'; +import { CURRENT_SCHEMA_VERSION, getCurrentVersion, runMigrations } from '../src/db/migrations'; +import { QueryBuilder } from '../src/db/queries'; +import { moduleReferenceKeys, referenceNameTail } from '../src/db/reference-tail'; + +type Files = Record; +/** Each route's links: `[ layout] ::`. */ +type Links = Record; + +interface Scenario { + name: string; + initial: Files; + added: Files; + expected: Links; +} + +const pkg = (dependencies: Record) => JSON.stringify({ name: 'app', dependencies }); +const angularComponent = (name: string, exported = 'export') => + `import { Component } from '@angular/core';\n\n@Component({ selector: 'app-x', template: '' })\n${exported} class ${name} {}\n`; +const vueView = (text: string) => `\n\n`; + +const SCENARIOS: Scenario[] = [ + { + name: 'React Router', + initial: { + 'package.json': pkg({ react: '^18', 'react-router-dom': '^6' }), + 'tsconfig.json': JSON.stringify({ compilerOptions: { baseUrl: '.', paths: { '@/*': ['src/*'] } } }), + 'src/router.tsx': [ + "import { createBrowserRouter } from 'react-router-dom';", + 'export const router = createBrowserRouter([', + " { path: '/team', lazy: () => import('./pages/Team') },", + " { path: '/about', lazy: () => import('./pages/About.tsx') },", + " { path: '/docs', lazy: () => import('./pages/Docs') },", + " { path: '/settings', lazy: () => import('./pages/Settings') },", + " { path: '/billing', lazy: () => import('@/pages/Billing') },", + " { path: '/app', lazy: () => import('./layouts/Shell'), children: [", + " { path: 'inbox', lazy: () => import('./pages/Inbox') },", + ' ] },', + ']);', + '', + ].join('\n'), + }, + added: { + 'src/pages/Team.tsx': 'export default function Team() { return null; }\n', + 'src/pages/About.tsx': 'export default function About() { return null; }\n', + // A default export named unlike its file. + 'src/pages/Docs.tsx': 'export default function DocsPage() { return null; }\n', + // A folder's index file. + 'src/pages/Settings/index.tsx': 'export default function Settings() { return null; }\n', + // Through a tsconfig alias, rendering the module's `Component` export. + 'src/pages/Billing.tsx': 'export function Component() { return null; }\n', + 'src/layouts/Shell.tsx': 'export default function Shell() { return null; }\n', + 'src/pages/Inbox.tsx': 'export default function Inbox() { return null; }\n', + }, + expected: { + '/team': ['references src/pages/Team.tsx::Team'], + '/about': ['references src/pages/About.tsx::About'], + '/docs': ['references src/pages/Docs.tsx::DocsPage'], + '/settings': ['references src/pages/Settings/index.tsx::Settings'], + '/billing': ['references src/pages/Billing.tsx::Component'], + '/app': ['references src/layouts/Shell.tsx::Shell'], + '/app/inbox': ['references layout src/layouts/Shell.tsx::Shell', 'references src/pages/Inbox.tsx::Inbox'], + }, + }, + { + name: 'Vue Router', + initial: { + 'package.json': pkg({ vue: '^3', 'vue-router': '^4' }), + 'src/router/index.js': [ + "import { createRouter, createWebHistory } from 'vue-router'", + 'export default createRouter({', + ' history: createWebHistory(),', + ' routes: [', + " { path: '/', component: () => import('../views/Home.vue') },", + " { path: '/login', component: () => import('@/views/Login') },", + " { path: '/admin', component: () => import('@/layout/index.vue'), children: [", + " { path: 'dashboard', component: () => import('@/views/dashboard/index') },", + ' ] },', + ' ]', + '})', + '', + ].join('\n'), + }, + added: { + 'src/views/Home.vue': vueView('home'), + 'src/views/Login.vue': vueView('login'), + 'src/layout/index.vue': vueView(''), + 'src/views/dashboard/index.vue': vueView('dashboard'), + }, + expected: { + '/': ['calls src/views/Home.vue::Home'], + '/login': ['calls src/views/Login.vue::Login'], + '/admin': ['calls src/layout/index.vue::index'], + '/admin/dashboard': ['calls src/views/dashboard/index.vue::index', 'references layout src/layout/index.vue::index'], + }, + }, + { + name: 'Angular', + initial: { + 'package.json': pkg({ '@angular/core': '^17', '@angular/router': '^17' }), + 'src/app/app.routes.ts': [ + "import { Routes } from '@angular/router';", + 'export const routes: Routes = [', + " { path: '', loadComponent: () => import('./home/home.component').then((m) => m.HomeComponent) },", + " { path: 'about', loadComponent: () => import('./about/about.component') },", + " { path: 'admin', loadComponent: () => import('./admin/shell.component').then((m) => m.ShellComponent), children: [", + " { path: 'users', loadComponent: () => import('./admin/users.component').then((m) => m.UsersComponent) },", + ' ] },', + '];', + '', + ].join('\n'), + }, + added: { + 'src/app/home/home.component.ts': angularComponent('HomeComponent'), + 'src/app/about/about.component.ts': angularComponent('AboutComponent', 'export default'), + 'src/app/admin/shell.component.ts': angularComponent('ShellComponent'), + 'src/app/admin/users.component.ts': angularComponent('UsersComponent'), + }, + expected: { + '/': ['references src/app/home/home.component.ts::HomeComponent'], + '/about': ['references src/app/about/about.component.ts::AboutComponent'], + '/admin': ['references src/app/admin/shell.component.ts::ShellComponent'], + '/admin/users': ['references layout src/app/admin/shell.component.ts::ShellComponent', 'references src/app/admin/users.component.ts::UsersComponent'], + }, + }, + { + // Ghostfolio's markets page: the child loads the class its layout + // renders, so the child's route and layout references make one edge, and + // the reference resolved first names it — in a full index, the one + // written first. + name: 'Angular, a child loading its layout\'s class', + initial: { + 'package.json': pkg({ '@angular/core': '^17', '@angular/router': '^17' }), + 'src/app/app.routes.ts': [ + "import { Routes } from '@angular/router';", + "import { MarketsComponent } from './markets/markets.component';", + 'export const routes: Routes = [', + " { path: 'markets', component: MarketsComponent, children: [", + " { path: '', loadComponent: () => import('./markets/markets.component').then((m) => m.MarketsComponent) },", + ' ] },', + '];', + '', + ].join('\n'), + }, + added: { 'src/app/markets/markets.component.ts': angularComponent('MarketsComponent') }, + expected: { '/markets': ['references src/app/markets/markets.component.ts::MarketsComponent'] }, + }, +]; + +/** + * The module is there from the start, but only a later edit gives it the + * component the route renders: React's default export, a component a Vue + * route loads from a script, Angular's class. + */ +const EDITED: Scenario[] = [ + { + name: 'React Router', + initial: { + 'package.json': pkg({ react: '^18', 'react-router-dom': '^6' }), + 'src/router.tsx': "import { createBrowserRouter } from 'react-router-dom';\nexport const router = createBrowserRouter([{ path: '/team', lazy: () => import('./pages/Team') }]);\n", + 'src/pages/Team.tsx': 'export const placeholder = 1;\n', + }, + added: { 'src/pages/Team.tsx': 'export default function Team() { return null; }\n' }, + expected: { '/team': ['references src/pages/Team.tsx::Team'] }, + }, + { + name: 'Vue Router', + initial: { + 'package.json': pkg({ vue: '^3', 'vue-router': '^4' }), + 'src/router/index.js': "import { createRouter, createWebHistory } from 'vue-router'\nexport default createRouter({ history: createWebHistory(), routes: [{ path: '/', component: () => import('../views/Home.js') }] })\n", + 'src/views/Home.js': 'export const placeholder = 1\n', + }, + added: { 'src/views/Home.js': 'export default function Home() { return null }\n' }, + expected: { '/': ['calls src/views/Home.js::Home'] }, + }, + { + name: 'Angular', + initial: { + 'package.json': pkg({ '@angular/core': '^17', '@angular/router': '^17' }), + 'src/app/app.routes.ts': "import { Routes } from '@angular/router';\nexport const routes: Routes = [{ path: '', loadComponent: () => import('./home/home.component').then((m) => m.HomeComponent) }];\n", + 'src/app/home/home.component.ts': 'export const placeholder = 1;\n', + }, + added: { 'src/app/home/home.component.ts': angularComponent('HomeComponent') }, + expected: { '/': ['references src/app/home/home.component.ts::HomeComponent'] }, + }, +]; + +let root: string | undefined; +let cg: CodeGraph | undefined; +let db: DatabaseConnection | undefined; + +afterEach(() => { + cg?.close(); + cg = undefined; + db?.close(); + db = undefined; + if (root) fs.rmSync(root, { recursive: true, force: true }); + root = undefined; +}); + +function write(dir: string, files: Files): void { + for (const [file, text] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(dir, file)), { recursive: true }); + fs.writeFileSync(path.join(dir, file), text); + } +} + +function routeLinks(graph: CodeGraph): Links { + const links: Links = {}; + for (const route of graph.getNodesByKind('route')) { + links[route.name] = graph + .getOutgoingEdges(route.id) + .filter((edge) => edge.kind !== 'contains') + .map((edge) => { + const target = graph.getNode(edge.target); + return `${edge.kind}${edge.metadata?.layout ? ' layout' : ''} ${target?.filePath}::${target?.name}`; + }) + .sort(); + } + return links; +} + +const unlinked = (expected: Links): Links => Object.fromEntries(Object.keys(expected).map((route) => [route, []])); + +/** The route links of a fresh index of `files`, in a folder of its own: indexing again over an index skips its unchanged files. */ +async function freshRouteLinks(files: Files): Promise { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-route-module-fresh-')); + try { + write(dir, files); + const fresh = await CodeGraph.init(dir, { index: true }); + try { + return routeLinks(fresh); + } finally { + fresh.close(); + } + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +} + +describe('sync links a route to the module it lazily loads', () => { + it.each(SCENARIOS)('when the module is added after the router: $name', async (s) => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-route-module-')); + write(root, s.initial); + cg = await CodeGraph.init(root, { index: true }); + expect(routeLinks(cg)).toEqual(unlinked(s.expected)); + + write(root, s.added); + expect((await cg.sync()).filesAdded).toBe(Object.keys(s.added).length); + // The router did not change: only the retry of its parked references links it. + expect(routeLinks(cg)).toEqual(s.expected); + expect(cg.getPendingReferenceCount()).toBe(0); + expect(await freshRouteLinks({ ...s.initial, ...s.added })).toEqual(s.expected); + }, 60_000); + + // The component a route renders is the module's, so an edit that gives the + // module one is what the route waited for. + it.each(EDITED)('when an edit gives the module its component: $name', async (s) => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-route-module-')); + write(root, s.initial); + cg = await CodeGraph.init(root, { index: true }); + expect(routeLinks(cg)).toEqual(unlinked(s.expected)); + + write(root, s.added); + expect((await cg.sync()).filesModified).toBe(1); + expect(routeLinks(cg)).toEqual(s.expected); + expect(cg.getPendingReferenceCount()).toBe(0); + expect(await freshRouteLinks({ ...s.initial, ...s.added })).toEqual(s.expected); + }, 60_000); + + it('when a deleted module comes back', async () => { + const [react] = SCENARIOS; + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-route-module-')); + write(root, { ...react!.initial, ...react!.added }); + cg = await CodeGraph.init(root, { index: true }); + expect(routeLinks(cg)['/team']).toEqual(['references src/pages/Team.tsx::Team']); + + fs.rmSync(path.join(root, 'src/pages/Team.tsx')); + expect((await cg.sync()).filesRemoved).toBe(1); + expect(routeLinks(cg)['/team']).toEqual([]); + expect(cg.getPendingReferenceCount()).toBe(0); + + write(root, { 'src/pages/Team.tsx': react!.added['src/pages/Team.tsx']! }); + expect((await cg.sync()).filesAdded).toBe(1); + expect(routeLinks(cg)).toEqual(react!.expected); + }, 60_000); + + it('in an index whose references an older version parked', async () => { + const [react] = SCENARIOS; + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-route-module-')); + write(root, react!.initial); + cg = await CodeGraph.init(root, { index: true }); + cg.close(); + cg = undefined; + + // What a version before 14 left: the tail cut at the path's last dot or colon. + db = DatabaseConnection.open(getDatabasePath(root)); + const refs = db.getDb().prepare("SELECT id, reference_name AS name FROM unresolved_refs WHERE reference_name LIKE '%lazy-import:%'").all() as Array<{ id: number; name: string }>; + expect(refs).toHaveLength(8); + for (const ref of refs) { + const legacy = ref.name.slice(Math.max(ref.name.lastIndexOf('.'), ref.name.lastIndexOf(':')) + 1); + db.getDb().prepare('UPDATE unresolved_refs SET name_tail = ? WHERE id = ?').run(legacy, ref.id); + } + db.getDb().exec(`DELETE FROM schema_versions WHERE version >= 14; + INSERT OR IGNORE INTO schema_versions(version, applied_at, description) VALUES (13, 0, 'legacy fixture');`); + db.close(); + db = undefined; + + cg = await CodeGraph.open(root); + write(root, react!.added); + await cg.sync(); + expect(routeLinks(cg)).toEqual(react!.expected); + expect(cg.getPendingReferenceCount()).toBe(0); + }, 60_000); +}); + +describe('referenceNameTail', () => { + it('parks a route\'s reference to a module under the stem of its path, behind module:', () => { + // React Router: a lazy route, and the lazy layout route around another. + expect(referenceNameTail('lazy-import:./pages/Team', 'references')).toBe('module:Team'); + expect(referenceNameTail('lazy-import:./routes/about.tsx', 'references')).toBe('module:about'); + expect(referenceNameTail('lazy-import:@/pages/Billing', 'references')).toBe('module:Billing'); + expect(referenceNameTail('lazy-import:./pages/Settings/', 'references')).toBe('module:Settings'); + expect(referenceNameTail('layout:lazy-import:./layouts/Shell', 'references')).toBe('module:Shell'); + // Vue Router: the component a route renders is a call, its layout a reference; + // vben's `#/` alias puts a `#` in the path. + expect(referenceNameTail('import:@/views/Login#Login', 'calls')).toBe('module:Login'); + expect(referenceNameTail('import:../views/Home.vue#Home', 'calls')).toBe('module:Home'); + expect(referenceNameTail('import:#/views/dashboard/analytics/index.vue#index', 'calls')).toBe('module:index'); + expect(referenceNameTail('layout:import:@/layout/index.vue#index', 'references')).toBe('module:index'); + // Angular: a member the import names, or the module's default. + expect(referenceNameTail('import:./home/home.component#HomeComponent', 'references')).toBe('module:home'); + expect(referenceNameTail('import:./about/about.component#default', 'references')).toBe('module:about'); + expect(referenceNameTail('layout:import:./admin/shell.component#ShellComponent', 'references')).toBe('module:shell'); + }); + + it('keeps the tail of a reference that names no module', () => { + // A layout named by its component, a module path of dots only, an import + // name with no member, and an import itself. + expect(referenceNameTail('layout:MainLayout', 'references')).toBe('MainLayout'); + expect(referenceNameTail('lazy-import:..', 'references')).not.toMatch(/^module:/); + expect(referenceNameTail('import:./x', 'calls')).toBe('/x'); + expect(referenceNameTail('./pages/Team', 'imports')).toBe('Team'); + // The same names as another kind of reference are not a route's. + expect(referenceNameTail('import:./home.component#HomeComponent', 'imports')).toBe('home'); + expect(referenceNameTail('lazy-import:./pages/Team', 'type_of')).toBe('/pages/Team'); + }); +}); + +describe('moduleReferenceKeys', () => { + it('gives a file the keys a module reference it could satisfy waits under', () => { + expect(moduleReferenceKeys('src/pages/Team.tsx')).toEqual(['module:pages', 'module:Team.tsx', 'module:Team']); + expect(moduleReferenceKeys('src/pages/Settings/index.tsx')).toEqual(['module:Settings', 'module:index.tsx', 'module:index']); + expect(moduleReferenceKeys('src/app/home/home.component.ts')).toEqual(['module:home', 'module:home.component.ts']); + }); +}); + +describe('schema v14', () => { + function fixture(): QueryBuilder { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-route-module-')); + db = DatabaseConnection.initialize(path.join(root, 'test.db')); + const queries = new QueryBuilder(db.getDb()); + queries.insertNode({ id: 'r', kind: 'route', name: '/team', qualifiedName: 'src/router.tsx::route:/team', filePath: 'src/router.tsx', + language: 'tsx', startLine: 1, endLine: 1, startColumn: 0, endColumn: 0, updatedAt: 0 }); + for (const [referenceName, referenceKind, status, tail] of [ + ['lazy-import:./pages/Team', 'references', 'failed', '/pages/Team'], + ['layout:lazy-import:./layouts/Shell', 'references', 'failed', '/layouts/Shell'], + ['lazy-import:./pages/About.tsx', 'references', 'failed', 'About.tsx'], + ['import:@/views/Login#Login', 'calls', 'failed', '@/views/Login#Login'], + ['layout:import:@/layout/index.vue#index', 'references', 'failed', 'vue#index'], + ['import:./home/home.component#HomeComponent', 'references', 'failed', 'component#HomeComponent'], + // Only a failed row is looked up by its tail. + ['lazy-import:./pages/Docs', 'references', 'pending', ''], + ['layout:MainLayout', 'references', 'failed', 'MainLayout'], + ['snippets/price.liquid', 'references', 'failed', 'price.liquid'], + ['./pages/Team', 'imports', 'failed', 'Team'], + ['Team', 'calls', 'failed', 'Team'], + ] as const) { + queries.insertUnresolvedRef({ fromNodeId: 'r', referenceName, referenceKind, line: 1, column: 0, filePath: 'src/router.tsx', language: 'tsx' }); + db.getDb().prepare('UPDATE unresolved_refs SET status = ?, name_tail = ? WHERE reference_name = ?').run(status, tail, referenceName); + } + return queries; + } + + const tails = () => db!.getDb().prepare('SELECT reference_name AS name, name_tail AS tail FROM unresolved_refs ORDER BY id').all(); + + it('rewrites the tail of a module reference parked by an older version, and replays cleanly', () => { + fixture(); + db!.getDb().exec(`DELETE FROM schema_versions WHERE version >= 14; + INSERT OR IGNORE INTO schema_versions(version, applied_at, description) VALUES (13, 0, 'legacy fixture');`); + db!.close(); + db = DatabaseConnection.open(path.join(root!, 'test.db')); + expect(getCurrentVersion(db.getDb())).toBe(CURRENT_SCHEMA_VERSION); + const migrated = tails(); + expect(migrated).toEqual([ + { name: 'lazy-import:./pages/Team', tail: 'module:Team' }, + { name: 'layout:lazy-import:./layouts/Shell', tail: 'module:Shell' }, + { name: 'lazy-import:./pages/About.tsx', tail: 'module:About' }, + { name: 'import:@/views/Login#Login', tail: 'module:Login' }, + { name: 'layout:import:@/layout/index.vue#index', tail: 'module:index' }, + { name: 'import:./home/home.component#HomeComponent', tail: 'module:home' }, + { name: 'lazy-import:./pages/Docs', tail: '' }, + { name: 'layout:MainLayout', tail: 'MainLayout' }, + { name: 'snippets/price.liquid', tail: 'price.liquid' }, + { name: './pages/Team', tail: 'Team' }, + { name: 'Team', tail: 'Team' }, + ]); + + // A module reference waits for a file that could be the module, not for + // a symbol: `Team` is the tail of the import and the call, not of the route. + const queries = new QueryBuilder(db.getDb()); + const retried = (names: string[]) => queries.getRetryableFailedReferences(names).map((ref) => ref.referenceName).sort(); + expect(retried(moduleReferenceKeys('src/pages/Team.tsx'))).toEqual(['lazy-import:./pages/Team']); + expect(retried(moduleReferenceKeys('src/app/home/home.component.ts'))).toEqual(['import:./home/home.component#HomeComponent']); + expect(retried(['Team', 'home', 'index'])).toEqual(['./pages/Team', 'Team']); + + db.getDb().exec('DELETE FROM schema_versions WHERE version >= 14'); + runMigrations(db.getDb(), 13); + expect(tails()).toEqual(migrated); + }); + + it('reads only the references its prefixes name, not every failed one', () => { + fixture(); + const prepare = vi.spyOn(db!.getDb(), 'prepare'); + db!.getDb().exec('DELETE FROM schema_versions WHERE version >= 14'); + runMigrations(db!.getDb(), 13); + const select = prepare.mock.calls.map(([sql]) => sql as string).find((sql) => sql.includes("GLOB 'lazy-import:*'")); + prepare.mockRestore(); + const plan = db!.getDb().prepare(`EXPLAIN QUERY PLAN ${select}`).all().map((row) => (row as { detail: string }).detail).join('; '); + expect(plan).toMatch(/USING (COVERING )?INDEX idx_unresolved_name/); + expect(plan).not.toMatch(/idx_unresolved_status|SCAN unresolved_refs/); + }); +}); diff --git a/src/db/migrations.ts b/src/db/migrations.ts index 7e5d1ccdc..7c07a99e3 100644 --- a/src/db/migrations.ts +++ b/src/db/migrations.ts @@ -10,7 +10,7 @@ import { referenceNameTail } from './reference-tail'; /** * Current schema version */ -export const CURRENT_SCHEMA_VERSION = 13; +export const CURRENT_SCHEMA_VERSION = 14; /** * Migration definition @@ -256,6 +256,34 @@ const migrations: Migration[] = [ } }, }, + { + version: 14, + description: 'Retry a failed route module reference when sync adds or changes its file: module tails', + up: (db) => { + // A route's reference to the module it lazily loads, parked before this + // version — React Router's `lazy-import:./pages/Team`, Vue Router's and + // Angular's `import:./home/home.component#HomeComponent`, each also + // behind `layout:` — carries a fragment of its path as its tail + // ('/pages/Team', 'component#HomeComponent'), which no file's keys + // match. Rewrite it to the module tail it is parked under now. + // Idempotent: a rewritten tail rewrites to itself. + // + // Each prefix is a range of idx_unresolved_name. The `+` keeps the + // planner off idx_unresolved_status, which it otherwise picks although + // 'failed' is nearly every row: on vscode's index that read 830K rows, + // about a second, to select none. + const update = db.prepare('UPDATE unresolved_refs SET name_tail = ? WHERE id = ?'); + const rows = db + .prepare(`SELECT id, reference_name, reference_kind, name_tail FROM unresolved_refs + WHERE (reference_name GLOB 'lazy-import:*' OR reference_name GLOB 'import:*' OR reference_name GLOB 'layout:*') + AND +status = 'failed' AND +reference_kind IN ('references', 'calls')`) + .all() as Array<{ id: number; reference_name: string; reference_kind: string; name_tail: string }>; + for (const row of rows) { + const tail = referenceNameTail(row.reference_name, row.reference_kind); + if (tail !== row.name_tail) update.run(tail, row.id); + } + }, + }, ]; /** diff --git a/src/db/queries.ts b/src/db/queries.ts index 93da4774a..2e18bace9 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -3829,6 +3829,9 @@ export class QueryBuilder { * (`get`, `map`, …) that one new definition won't resolve — the same * rationale as resolution's AMBIGUOUS_NAME_CEILING (#999) — and retrying an * arbitrary subset would be both wasted work and incoherent coverage. + * + * A name can also be one of a changed file's `moduleReferenceKeys`, which a + * route's reference to the module it lazily loads is parked under. */ getRetryableFailedReferences(names: string[], perNameCeiling: number = 500): UnresolvedReference[] { if (names.length === 0) return []; diff --git a/src/db/reference-tail.ts b/src/db/reference-tail.ts index af4ea5a7c..61c16e4a1 100644 --- a/src/db/reference-tail.ts +++ b/src/db/reference-tail.ts @@ -2,8 +2,8 @@ * The keys sync's failed-ref retry (#1240) matches on: the tail a failed ref * is parked under, and the names a newly added file can be imported as. * Shared by the query layer, which writes and reads them, and the migrations - * that rewrote the tails parked before path imports and path references had - * ones of their own. + * that rewrote the tails parked before path imports, path references and + * module references had ones of their own. */ /** @@ -19,6 +19,17 @@ * that could satisfy it. The dotted tail was the extension ('dart', 'php') or * a path fragment ('/foo'), which no file ever matched. * + * A route's reference to the module it lazily loads names a file the same + * way, by the module's path with or without its extension: React Router's + * `lazy-import:./pages/Team`, Vue Router's and Angular's + * `import:./home/home.component#HomeComponent`. It is parked under that + * path's stem behind 'module:' — 'module:Team', 'module:home' — the key + * {@link moduleReferenceKeys} gives a file that could be the module. A bare + * stem would be any symbol's name, which a module reference does not wait + * for: 'index' and 'types' are the tails of thousands of failed calls. Its + * dotted tail was a path fragment ('/pages/Team') or what follows the + * module's own dot ('component#HomeComponent'), which no file ever matched. + * * Any other reference written as a path to a file names it by the file's own * name: Liquid's `{% render 'price' %}` is `snippets/price.liquid`, parked as * 'price.liquid' — the name path matching looks the file up by, and the name @@ -27,12 +38,15 @@ * division, so only a `references` ref is read as a path. */ export function referenceNameTail(referenceName: string, referenceKind?: string): string { - if (referenceKind === 'imports') { - const path = referenceName.replace(/\/+$/, ''); - const slash = path.lastIndexOf('/'); - const stem = slash >= 0 ? fileNameStem(path.slice(slash + 1)) : ''; - // `..` and `.` name a folder only through the importing file's location. - if (!/^\.*$/.test(stem)) return stem; + if (referenceKind === 'imports' && referenceName.replace(/\/+$/, '').includes('/')) { + const stem = pathStem(referenceName); + if (stem) return stem; + } + // Vue Router's route names the component it renders as a call. + if (referenceKind === 'references' || referenceKind === 'calls') { + const route = MODULE_REFERENCE.exec(referenceName); + const stem = route ? pathStem(route[1] ?? route[2]!) : ''; + if (stem) return MODULE_KEY + stem; } if (referenceKind === 'references') { const fileName = referenceName.slice(referenceName.lastIndexOf('/') + 1); @@ -48,6 +62,25 @@ export function referenceNameTail(referenceName: string, referenceKind?: string) /** A file name with an extension: 'price.liquid', 'icon.logo.liquid', 'about.tsx' — not 'x.component#Name'. */ const FILE_NAME = /^[\w$@+~.-]+\.[A-Za-z][A-Za-z0-9]*$/; +/** + * A route's reference to the module it lazily loads, capturing the module's + * path: React Router's `lazy-import:`, Vue Router's and Angular's + * `import:#` — a path may hold a `#` itself, as vben's `#/views/…` + * alias does — each also behind `layout:`, for a layout route's module. + */ +const MODULE_REFERENCE = /^(?:layout:)?(?:lazy-import:(.+)|import:(.+)#[^#]+)$/; + +/** What a module reference's tail starts with. No symbol's name does, so a lookup by symbol names never finds one. */ +const MODULE_KEY = 'module:'; + +/** A path's last segment up to its first dot — `./pages/Team` → 'Team', `a/b.dart` → 'b' — or '' when that is only dots. */ +function pathStem(path: string): string { + const trimmed = path.replace(/\/+$/, ''); + const stem = fileNameStem(trimmed.slice(trimmed.lastIndexOf('/') + 1)); + // `..` and `.` name a folder only through the referring file's location. + return /^\.*$/.test(stem) ? '' : stem; +} + /** A file or folder name up to its first dot: 'b.dart' → 'b', 'types.d.ts' → 'types', '.eslintrc.js' → '.eslintrc'. */ function fileNameStem(name: string): string { const dot = name.indexOf('.', 1); @@ -70,3 +103,15 @@ export function importPathKeys(filePath: string): string[] { } return [...keys]; } + +/** + * The keys a module reference the file at `filePath` could satisfy is parked + * under: its {@link importPathKeys} behind 'module:' — `src/pages/Team.tsx` + * is the module of `lazy-import:./pages/Team`, `pages/Team/index.tsx` of the + * same path through its folder. A sync looks them up for the files it adds + * AND the ones it changes: a route renders the component its module exports, + * so an edit that gives the module one is what the route waited for. + */ +export function moduleReferenceKeys(filePath: string): string[] { + return importPathKeys(filePath).map((key) => MODULE_KEY + key); +} diff --git a/src/index.ts b/src/index.ts index 862add024..a789730ee 100644 --- a/src/index.ts +++ b/src/index.ts @@ -30,7 +30,7 @@ import { import { DatabaseConnection, getDatabasePath, removeDatabaseFiles } from './db'; import { WalCheckpointValve, resolveWalValveMb } from './db/wal-valve'; import { QueryBuilder } from './db/queries'; -import { importPathKeys } from './db/reference-tail'; +import { importPathKeys, moduleReferenceKeys } from './db/reference-tail'; import { isInitialized, createDirectory, @@ -984,12 +984,17 @@ export class CodeGraph { // Look them up by the symbol names the changed files now carry // and re-resolve just that set. The names include each file's // own, which a reference written as a path - // (`snippets/price.liquid`) waits under. On a sync where no failed - // ref matches, this is one indexed lookup. + // (`snippets/price.liquid`) waits under, and the keys a route's + // lazily loaded module waits under (`module:Team` for + // `lazy-import:./pages/Team`): a route renders the component its + // module exports, so an edit can satisfy it as well as an added + // file. On a sync where no failed ref matches, this is one + // indexed lookup. const tRetry = Date.now(); - const retryable = this.queries.getRetryableFailedReferences( - this.queries.getNodeNamesByFiles(result.changedFilePaths) - ); + const retryable = this.queries.getRetryableFailedReferences([...new Set([ + ...this.queries.getNodeNamesByFiles(result.changedFilePaths), + ...result.changedFilePaths.flatMap(moduleReferenceKeys), + ])]); // A failed import waits for a file, a folder or a namespace, not a // symbol: `package:app/b.dart` for a file named `b.dart`, `./ui` // for `ui/index.ts`, `using Foo.Bar` for that namespace's node. @@ -1004,6 +1009,11 @@ export class CodeGraph { for (const ref of importRetry) { if (!retryRows.has(ref.rowId)) retryable.push(ref); } + // In row order, as a full index resolves them: when two refs make + // the same edge, such as a route's lazily loaded class that its + // layout also renders, the one written first names it, as it + // does in a fresh index. + retryable.sort((a, b) => (a.rowId ?? 0) - (b.rowId ?? 0)); if (retryable.length > 0) { options.onProgress?.({ phase: 'resolving',