test: add the toolkit join parity harness - #1113
Conversation
Build the oracle S5 and S6 are judged by: does joining the extracted layers (enrichment + curation) against a catalog snapshot reproduce the toolkit JSON we ship today? - capture-catalog-snapshot.ts pulls a raw /v1/tool_metadata snapshot (paginated at 1000), records total_count, and refuses to write a truncated fetch. The snapshot is gitignored (~10 MB, never committed). - verify-toolkit-join.ts joins a snapshot with the enrichment and curation layers through the real merger (no LLM, committed toolkit as previousToolkit) and reports the first structural difference per file by JSON path. Comparison is structural, not raw-byte: generatedAt is volatile and summary's key position varies across generator versions. Until S3/S5 land the enrichment/ and curation/ directories, the layers are read straight out of the committed toolkit JSON (the default). Proven offline: projecting every committed tool back to its pre-merge shape and re-joining reproduces all 117 toolkits with zero differences; an injected one-character enrichment change and a missing catalog item each fail with a named JSON path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
These scripts live in scripts/, which nothing type-checks today — #1106 widens the generator project to cover it. Under that project's exactOptionalPropertyTypes and noUncheckedIndexedAccess they do not compile, so #1106 landing would break this PR. - IMetadataSource was imported from src/sources/interfaces, which does not export it; it lives in src/sources/internal, where all eight other consumers import it from. It is an 'import type', so it erased at runtime and no test could catch it. - Bind the indexed argv reads before use: a truthiness check on argv[i + 1] does not narrow a later, separate read of the same index. - Declare the two optional properties that legitimately receive an explicit undefined as '| undefined' rather than spreading conditionally at each call site. No behaviour change; the harness still reproduces all 117 toolkits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e9a38de. Configure here.
| total_count, | ||
| }); | ||
| return groupToolsByToolkit(items); | ||
| }; |
There was a problem hiding this comment.
Verifier skips snapshot truncation check
Medium Severity
loadCatalogTools never checks that snapshot items.length matches the recorded totalCount/total_count. Capture comments say truncated fetches are detectable by the verifier, but a partial snapshot still parses and proceeds, so parity failures look like missing tools instead of a bad snapshot.
Reviewed by Cursor Bugbot for commit e9a38de. Configure here.
| } | ||
| items.push(...page.items); | ||
| process.stdout.write(`\r fetched ${items.length}/${totalCount} tools`); | ||
| } |
There was a problem hiding this comment.
Snapshot pagination skips short pages
Medium Severity
The capture loop advances offset by fixed PAGE_SIZE instead of page.items.length, unlike EngineApiSource which this script claims to mirror. A short non-final page skips tools and trips the truncation guard, so a catalog the generator can fetch may be impossible to snapshot.
Reviewed by Cursor Bugbot for commit e9a38de. Configure here.
1 participant
Why
The generator is being split into upstream data, enrichment, and curation layers. This adds a deterministic parity gate that answers one question: does joining those layers still reproduce the toolkit JSON we ship today?
It turns output preservation from a manual review task into a mechanical check, protecting the pipeline as its sources are extracted and reorganized.
How it fits
The harness sits alongside generation rather than replacing it: it uses the production merger, reports the first structural difference by JSON path, and makes future pipeline slices independently verifiable.
Changes
total_countand rejects truncated fetches.Verification
generatedAtare handled structurally to avoid false positives.Independent of #1112 at merge time, but intended to validate it and the subsequent enrichment/source-extraction slices.
Note
Low Risk
New scripts and tests only; no changes to production merge or publish paths. Harness reads local JSON and optional Engine credentials for snapshot capture.
Overview
Adds a mechanical parity gate for the upcoming catalog / enrichment / curation split: join those inputs through the real
mergeToolkitpath and assert the result matches committed toolkit JSON underdata/toolkits.capture-catalog-snapshot.tspaginates Engine/v1/tool_metadata(same URL shape as production), recordstotalCount, and refuses to write if the fetch is truncated. Snapshots are gitignored (~10 MB).verify-toolkit-join.tsloads a snapshot viaparseToolMetadataResponse+groupToolsByToolkit, runsjoinToolkit(merger + frozen enrichment overlay forcodeExample/secretsInfo/summary), and compares structurally withfirstDifference—ignoringgeneratedAtand treating optional null/absent keys as equivalent. Until separate layer dirs exist, enrichment and curation default to the reference toolkit files. Non-zero exit on any JSON-path mismatch or reference tool missing from the catalog.Vitest covers the diff helper, round-trip parity across all committed toolkits (offline catalog projected from each file), drift detection, and API reshape fixtures.
Reviewed by Cursor Bugbot for commit e9a38de. Bugbot is set up for automated code reviews on this repo. Configure here.