From a0266a1030108b9e9df0015a46c4aa7db1c96c87 Mon Sep 17 00:00:00 2001 From: PatrickSys Date: Tue, 6 Oct 2026 10:38:57 +0200 Subject: [PATCH 1/2] docs: publish benchmark evidence and released setup guidance --- README.md | 278 ++-- docs/benchmark-prior-public-report.md | 182 +++ docs/benchmark.md | 190 +-- docs/capabilities.md | 48 +- docs/cli.md | 57 +- docs/client-setup.md | 375 +++-- docs/comparison-table.md | 49 +- docs/demo.md | 29 +- results/benchmark-presentation-evidence.json | 1251 +++++++++++++++++ ...contextbench-engineering-note-summary.json | 213 +++ 10 files changed, 2149 insertions(+), 523 deletions(-) create mode 100644 docs/benchmark-prior-public-report.md create mode 100644 results/benchmark-presentation-evidence.json create mode 100644 results/contextbench-engineering-note-summary.json diff --git a/README.md b/README.md index ed333e2..dbc2d3f 100644 --- a/README.md +++ b/README.md @@ -1,251 +1,163 @@ -# codebase-context - -## Map your team's conventions before your AI agent starts searching. +# Codebase Context [![npm version](https://img.shields.io/npm/v/codebase-context)](https://www.npmjs.com/package/codebase-context) [![license](https://img.shields.io/npm/l/codebase-context)](./LICENSE) [![node](https://img.shields.io/node/v/codebase-context)](https://github.com/PatrickSys/codebase-context/blob/master/package.json) -You're tired of AI agents writing code that "just works" but still misses how your team actually builds things. They search too broadly, pick generic examples, and spend tokens exploring before they understand the shape of the repo. - -`codebase-context` changes the first step. Start with a bounded conventions map that shows the architecture, dominant patterns, and strongest local examples. Then search for the exact file, symbol, or workflow you need. - -Here's what codebase-context does: - -**Starts with a bounded conventions map** - The first call shows architecture layers, active patterns, golden files, and next calls without dumping vendored repos, fixtures, generated output, or oversized entrypoint lists into the default surface. - -**Finds the right local example** - Search does not just return code. Each result comes back with pattern signals, file relationships, and quality indicators so the agent can move from the map to the most relevant local example instead of wandering through raw hits. - -**Knows what is current** - Conventions are detected from your code and git history, not only from rules you wrote. The map distinguishes what is common from what is rising or declining, and points at the files that best represent the current direction. - -**Adds support signals when you need them** - Team memory and edit-readiness checks stay available, but as supporting context after the map and search have already narrowed the work. - -Map first, search second, local-first throughout. Your code never leaves your machine by default. - -See the [current discovery benchmark](https://github.com/PatrickSys/codebase-context/blob/master/docs/benchmark.md) for the checked-in discovery-only proof. The gate is still `pending_evidence`, and `claimAllowed` remains `false`. +## Your coding agent doesn't understand your codebase. -### What it looks like +Coding agents can read files, but they still have to discover how your repository is organized, which patterns your team follows, and which examples are worth copying. -Real CLI output against `angular-spotify`, the repo used for the launch screenshots. +Codebase Context gives an agent a local view of that information through code search, team patterns, strong examples, and project memory. It runs as an MCP server - a local tool that your editor or command-line agent can call while it works - and keeps the index on your machine by default. -**Lead signal: pattern drift and golden files** +## Set up your AI client -![codebase-context patterns screenshot](https://raw.githubusercontent.com/PatrickSys/codebase-context/master/docs/assets/patterns.png) - -This is the part most tools miss: what the team is doing now, what it is moving away from, and which files are the strongest examples to follow. - -**Before editing: preflight and impact** - -![codebase-context search preflight screenshot](https://raw.githubusercontent.com/PatrickSys/codebase-context/master/docs/assets/search-query.png) - -When the agent searches with edit intent, it gets a compact decision card: confidence, whether it's safe to proceed, which patterns apply, the best example, and which files are likely to be affected. - -More CLI examples in [`docs/cli.md`](./docs/cli.md). Full walkthrough: [demo.md on GitHub](https://github.com/PatrickSys/codebase-context/blob/master/docs/demo.md). - -## Quick Start +Choose your coding tool and run its command once. Use Node.js 22 or newer. These commands use published npm `2.2.0` and do not require a project folder in your configuration: ```bash -claude mcp add codebase-context -- npx -y codebase-context -``` - -The server runs in two modes. Use stdio unless you need multiple clients connected at once: +# Claude Code +claude mcp add --scope user --transport stdio codebase-context -- npx -y codebase-context@2.2.0 -| Mode | How it runs | When to use | -| ------------------- | ------------------------------------------------ | ------------------------------------------ | -| **stdio** (default) | Process spawned by the client | One AI client talking to one or more repos | -| **HTTP** | Long-lived server at `http://127.0.0.1:3100/mcp` | Multiple clients sharing one server | +# Codex CLI +codex mcp add codebase-context -- npx -y codebase-context@2.2.0 -Client support at a glance: - -| Client | stdio | HTTP | -| ----------------- | ----- | -------------------------------------------- | -| Claude Code | Yes | No (stdio only) | -| Claude Desktop | Yes | No | -| Cursor | Yes | Yes — `.cursor/mcp.json` with `type: "http"` | -| Windsurf | Yes | Not yet | -| Codex | Yes | Yes — `--mcp-config` flag | -| VS Code (Copilot) | Yes | No | -| OpenCode | Yes | Not documented yet | - -Copy-pasteable templates: [`templates/mcp/stdio/.mcp.json`](./templates/mcp/stdio/.mcp.json) and [`templates/mcp/http/.mcp.json`](./templates/mcp/http/.mcp.json). +# OpenCode 1.x (keep the quoted separator on Windows) +opencode mcp add codebase-context '--' npx -y codebase-context@2.2.0 +``` -Full per-client setup, HTTP server instructions, and local build testing: [`docs/client-setup.md`](./docs/client-setup.md). +Start a new agent session in your project, then ask: -## First Use +> Use Codebase Context to find [feature] in this repository. Pass this repository's absolute path as project when checking get_indexing_status and searching. Wait for indexing if needed, read codebase://context, then search_codebase and open a returned source file. Show me the relevant files. -Get a conventions map of your codebase before exploring or editing: +Replace `[feature]` with something you want to find. The agent supplies the repository path in its tool calls, so the registration can serve different projects. Initial indexing may need a local model download. The October 6 isolated checks proved these client registrations and published-package project selection/search; they did not establish a full native agent investigation. -```bash -# See your codebase conventions — architecture layers, patterns, golden files -npx -y codebase-context map +For Codex Desktop, create or merge `.codex/config.toml` in the project you want to search: -# Then search for what you need -npx -y codebase-context search --query "auth middleware" +```toml +[mcp_servers.codebase-context] +command = "npx" +args = ["-y", "codebase-context@2.2.0"] +startup_timeout_sec = 120 ``` -Your AI agent uses the same map via the `codebase://context` MCP resource on first call. +Trust the project if asked, restart Codex, and start a new task there. Use the prompt above. Config placement alone does not establish successful first use; this no-folder recipe has not been accepted in a fresh native Desktop task. -## Common First Commands +Other clients use their own setup commands: -Three commands to understand a repo before you edit it: +| Client | Shortest current setup | +| --------------------------- | ---------------------------------------------------------------------------- | +| Gemini CLI | `gemini mcp add --scope user codebase-context npx -y codebase-context@2.2.0` | +| Cursor | Add `.cursor/mcp.json` | +| VS Code with GitHub Copilot | Add `.vscode/mcp.json` | +| GitHub Copilot CLI | `copilot mcp add codebase-context -- npx -y codebase-context@2.2.0` | +| Windsurf | Add `~/.codeium/windsurf/mcp_config.json` | -```bash -# What are the main conventions and best examples? -npx -y codebase-context map - -# Then search for the local example you need -npx -y codebase-context search --query "auth middleware" +Check an existing same-name entry before replacing it. To give the server a default folder, append that folder's absolute path to the `npx` arguments. The [client setup guide](./docs/client-setup.md) covers scopes, optional fixed-folder configuration, verification limits and the unreleased installer. Published `2.2.0`'s interactive `init` has registration bugs; use the commands above. -# What patterns is the team actually using right now? -npx -y codebase-context patterns -``` +The [client setup guide](./docs/client-setup.md) has the exact commands and config for every client, plus what was checked locally and what still relies on official instructions. -This is also what your AI agent consumes automatically via MCP tools; the CLI is the human-readable version of the same map-plus-search flow. +The default connection is `stdio` (standard input/output): your client starts the server when it needs it. HTTP is an advanced, client-dependent option; verify the setup guide and client support before relying on it. -## What it does +## What your agent gets -### The Search Tool (`search_codebase`) +### Relevant code -One call returns ranked results with `file`, `summary`, `score`, compact type (`componentType:layer`), pattern trend signals, relationship hints, related team memories, a search quality assessment, and a preflight decision card when `intent="edit"`. The decision card shows `ready` (boolean), `nextAction` when not ready, `patterns` (do/avoid), `bestExample`, impact coverage (`"3/5 callers in results"`), and `whatWouldHelp`. +`search_codebase` ranks files and symbols for the task instead of returning an unstructured dump. The agent can ask for a compact result first, then read the code it needs. -Default output is lean — if the agent wants code, it calls `read_file`. Add `includeSnippets: true` for inline code with scope headers (e.g. `// AuthService.getToken()`). +### Team patterns and examples -See [`docs/capabilities.md`](./docs/capabilities.md) for the full field reference. +`get_team_patterns` shows the approaches used in the repository and points to representative files. Published `2.2.0` includes dedicated analyzers for Angular, React and Next.js, with a generic analyzer for other stacks. NestJS support belongs to the newer source candidate. -### Patterns & Conventions (`get_team_patterns`) +### Project memory -Detects what your team actually does by analyzing the codebase: adoption percentages for DI, state management, testing, and library patterns; trend direction (Rising / Stable / Declining) from git recency; golden files ranked by modern pattern density; conflicts when two approaches both exceed 20%. +`remember` stores a convention, decision, gotcha, or past failure for the project. `get_memory` retrieves relevant entries in later sessions, including when the agent or editor changes. -### Team Memory (`remember` + `get_memory`) +## How it works -Record a decision once. It surfaces automatically in search results and preflight cards from then on. Conventional commits (`refactor:`, `migrate:`, `fix:`, `revert:`) from the last 90 days auto-extract into memory during indexing — no setup required. +1. **Index locally.** Codebase Context scans the project, builds a keyword index, and creates local semantic embeddings - numeric representations used to match code by meaning as well as exact words. +2. **Understand the repository.** The agent can request a compact codebase map with structure, patterns, and representative files. +3. **Find the code for the task.** Search returns ranked files and symbols; the agent reads the selected files before editing. -Memory types: `convention`, `decision`, `gotcha`, `failure`. Confidence decay: conventions never decay, decisions 180-day half-life, gotchas/failures 90-day. Stale memories get flagged instead of blindly trusted. +The same information is available from the terminal. Run these commands from your project root. The first index can take a while because it scans the project and creates local embeddings; once it is ready, inspect the map and search for the code you need: -## Tools +```bash +# Build or refresh the local index +npx -y codebase-context@2.2.0 reindex -| Tool | What it does | -| ------------------------------ | ----------------------------------------------------- | -| `search_codebase` | Hybrid search + decision card when `intent="edit"` | -| `get_team_patterns` | Pattern frequencies, golden files, conflict detection | -| `get_symbol_references` | Concrete references to a symbol (count + snippets) | -| `remember` | Record a convention, decision, gotcha, or failure | -| `get_memory` | Query team memory with confidence decay scoring | -| `get_codebase_metadata` | Project structure, frameworks, dependencies | -| `get_style_guide` | Style guide rules for the current project | -| `detect_circular_dependencies` | Import cycles between files | -| `refresh_index` | Full or incremental re-index + git memory extraction | -| `get_indexing_status` | Progress and stats for the current index | +# Repository structure, patterns, and representative files +npx -y codebase-context@2.2.0 map -## Multi-project +# Ranked code search +npx -y codebase-context@2.2.0 search --query "auth middleware" -One server, multiple repos. Three cases: +# Current team patterns +npx -y codebase-context@2.2.0 patterns +``` -| Case | What happens | -| --------------------------------------------- | --------------------------------------------------- | -| One project | Routing is automatic | -| Multiple projects, active project already set | Routes to the active project | -| Multiple projects, ambiguous | Returns `selection_required` — retry with `project` | +One stdio server can route across several repositories. Supply `project` in tool calls to select the intended repository; a successful selection becomes the default for later calls in that process. Some clients also announce workspace roots: one root can auto-select, while an ambiguous selection asks for a project instead of guessing. MCP deprecated Roots in its July 2026 revision, so explicit project selection is the documented default rather than a dependency on client discovery. -`project` accepts a project root path, file path, `file://` URI, or relative subproject path (e.g. `apps/dashboard`). +## See it -```json -{ - "name": "search_codebase", - "arguments": { - "query": "auth interceptor", - "project": "apps/dashboard" - } -} -``` +These are real CLI results from the open-source `angular-spotify` repository. -If you get `selection_required`, retry with one of the paths from `availableProjects`. Full routing details and response shapes in [`docs/capabilities.md`](./docs/capabilities.md#project-routing). +**Patterns and representative files** -## Language Support +![Codebase Context showing repository patterns and representative files](https://raw.githubusercontent.com/PatrickSys/codebase-context/master/docs/assets/patterns.png) -10 languages with full symbol extraction via Tree-sitter: TypeScript, JavaScript, Python, Java, Kotlin, C, C++, C#, Go, Rust. 30+ languages with indexing and retrieval coverage, including PHP, Ruby, Swift, Scala, Shell, and config formats. Angular, React, Next.js, and NestJS have dedicated analyzers; everything else uses the Generic analyzer with AST-aligned chunking when a grammar is available. +The map shows the patterns found in the project, how common they are, and files that demonstrate them. -## Configuration +**Search before an edit** -| Variable | Default | Description | -| ------------------------------ | --------------------------------- | ------------------------------------------------------------------- | -| `EMBEDDING_PROVIDER` | `transformers` | `openai` (fast, cloud) or `transformers` (local, private) | -| `OPENAI_API_KEY` | — | Required only if using `openai` provider | -| `CODEBASE_ROOT` | — | Bootstrap root for CLI and single-project MCP clients | -| `CODEBASE_CONTEXT_DEBUG` | — | Set to `1` for verbose logging | -| `EMBEDDING_MODEL` | `Xenova/bge-small-en-v1.5` | Local embedding model override | -| `CODEBASE_CONTEXT_HTTP` | — | Set to `1` to start in HTTP mode (same as `--http` flag) | -| `CODEBASE_CONTEXT_PORT` | `3100` | HTTP server port override (same as `--port`; ignored in stdio mode) | -| `CODEBASE_CONTEXT_CONFIG_PATH` | `~/.codebase-context/config.json` | Override the server config file path | +![Codebase Context showing a ranked search and edit preflight](https://raw.githubusercontent.com/PatrickSys/codebase-context/master/docs/assets/search-query.png) -Large projects can override the default 5,000 searchable-chunk safety limit per project: +The search result shows ranked files, relevant project patterns, and what the agent should read before it changes code. -```json -{ - "projects": [ - { - "root": "/path/to/large-project", - "parsing": { "maxChunks": 25000 } - } - ] -} -``` +More examples are available in the [CLI gallery](./docs/cli.md) and [walkthrough](https://github.com/PatrickSys/codebase-context/blob/master/docs/demo.md). -## Performance +## Privacy -- **First indexing**: 2-5 minutes for ~30k files (embedding computation). -- **Subsequent queries**: milliseconds from cache. -- **Incremental updates**: `refresh_index` with `incrementalOnly: true` processes only changed files (SHA-256 manifest diffing). +Code and indexes stay on the machine with the default local embedding provider. Docker, a GPU, and an API key are not required. -## File Structure +Cloud embeddings are optional. If you select a cloud provider, code chunks are sent to that provider to create the search index. The provider, model, project root, and local HTTP port can be changed through environment variables or the project config; see the [capabilities reference](./docs/capabilities.md). -``` -.codebase-context/ - memory.json # Team knowledge (should be persisted in git) - index-meta.json # Index metadata and version (generated) - intelligence.json # Pattern analysis (generated) - relationships.json # File/symbol relationships (generated) - index.json # Keyword index (generated) - index/ # Vector database (generated) -``` +This is the privacy boundary of Codebase Context itself. Your AI client may send search results or file contents to the model provider configured in that client. Codebase Context does not control that connection. If you commit and push `.codebase-context/memory.json`, the recorded project memory also travels with the repository. -**Recommended `.gitignore`:** +Generated indexes belong in `.gitignore`. Project memory can be kept in version control when the team wants to share it: ```gitignore -# Codebase Context - ignore generated files, keep memory .codebase-context/* !.codebase-context/memory.json ``` -## What to add to your CLAUDE.md / AGENTS.md - -Paste this into `.cursorrules`, `CLAUDE.md`, `AGENTS.md`, or wherever your AI reads project instructions: +## Evidence -```markdown -## Codebase Context (MCP) +Codebase Context runs locally through the Model Context Protocol (MCP), with indexed code kept on your machine by default. -**Start of every task:** Call `get_memory` to load team conventions before writing any code. +Its ranked code search combines **Match words**, **Search by meaning**, and **Rank results** to give the agent ranked files, a best example, project patterns, relationships, and relevant memory before they edit. -**Before editing existing code:** Call `search_codebase` with `intent: "edit"`. If the preflight card says `ready: false`, read the listed files before touching anything. +The [benchmark](./docs/benchmark.md) reports a corrected 100-attempt retrieval family across five local code-context tools: 99 completed attempts and 1 failed attempt. Codebase Context recovered 25.7% of expected gold files with 11.5% file precision in that fixed adapter run; jCodeMunch recovered 27.1% with 11.0% file precision. Raw-native is a deterministic lexical adapter here, not a full normal coding-agent baseline. These are retrieval observations, not a winner or a coding-quality claim. The [sanitized evidence extract](./results/benchmark-presentation-evidence.json) records the exact values and source hashes. -**Before writing new code:** Call `get_team_patterns` to check how the team handles DI, state, testing, and library wrappers — don't introduce a new pattern if one already exists. +The same report retains a separate 300-attempt repeatability history, a 30-run two-task full-agent pilot, and a metered replay whose tool-use validity remains unresolved. A historical Codebase Context indexing observation with embeddings enabled took about 12 minutes 31 seconds, but it does not establish warm-use speed, query latency, install cost, or a cross-tool index-speed ranking. The report does not combine these families into a pooled score and does not establish patch correctness or end-to-end coding quality. Earlier public reports are preserved separately in the [benchmark archive](./docs/benchmark-prior-public-report.md). -**When asked to "remember" or "record" something:** Call `remember` immediately, before doing anything else. - -**When adding imports that cross module boundaries:** Call `detect_circular_dependencies` with the relevant scope after adding the import. -``` +The method and failures are documented so the measurements can be inspected with their limits. -These are the behaviors that make the most difference day-to-day. Copy, trim what doesn't apply to your stack, and add it once. +## Limits -## Links +- Local semantic indexing does more work than a plain text index, especially on a fresh checkout and CPU-only machine. +- Retrieval measurements describe expected-file coverage, file precision, and reported `peakPrivateGb`, not patch correctness or end-to-end coding quality. +- The paired token observation covers two frozen investigation tasks and records observed agent behavior; it is not a universal token or time guarantee. +- Setup checks differ by client. The detailed guide distinguishes a written config, a config recognized by the client, a local connection, and instructions checked only against official docs. +- Published `2.2.0` has dedicated Angular, React and Next.js analyzers; NestJS support is in the newer source candidate. Other projects use the generic analyzer and the language parsers available for that stack. +- The default searchable-chunk limit is 5,000 per project. Larger repositories can raise it in `.codebase-context/config.json`. +- The agent must identify its repository in tool calls when no default or unambiguous client root is available. Concurrent HTTP client isolation is not established by the stdio routing checks. -- [Benchmark](https://github.com/PatrickSys/codebase-context/blob/master/docs/benchmark.md) — current discovery suite results and gate truth -- [Demo](https://github.com/PatrickSys/codebase-context/blob/master/docs/demo.md) — real CLI walkthrough -- [Client Setup](./docs/client-setup.md) — per-client config, HTTP setup, local build testing -- [Capabilities Reference](./docs/capabilities.md) — tool API, retrieval pipeline, decision card schema -- [CLI Gallery](./docs/cli.md) — formatted command output examples -- [Motivation](https://github.com/PatrickSys/codebase-context/blob/master/MOTIVATION.md) — research and design rationale -- [Contributing](https://github.com/PatrickSys/codebase-context/blob/master/CONTRIBUTING.md) — dev setup and eval harness -- [Changelog](https://github.com/PatrickSys/codebase-context/blob/master/CHANGELOG.md) +## Reference -## License +- [Client setup](./docs/client-setup.md#client-setup) - commands, config, proof level, and client limits +- [Capabilities](./docs/capabilities.md) - tools, response fields, routing, and configuration +- [CLI](./docs/cli.md) - terminal commands and example output +- [Benchmark](./docs/benchmark.md) - method, measurements, and failures +- [Demo](https://github.com/PatrickSys/codebase-context/blob/master/docs/demo.md) - a complete repository walkthrough +- [Motivation](https://github.com/PatrickSys/codebase-context/blob/master/MOTIVATION.md) - the design problem and research background +- [Contributing](https://github.com/PatrickSys/codebase-context/blob/master/CONTRIBUTING.md) - local development and evaluation commands +- [Changelog](https://github.com/PatrickSys/codebase-context/blob/master/CHANGELOG.md) - release history -Elastic-2.0 +Elastic-2.0. See [LICENSE](./LICENSE). diff --git a/docs/benchmark-prior-public-report.md b/docs/benchmark-prior-public-report.md new file mode 100644 index 0000000..506360f --- /dev/null +++ b/docs/benchmark-prior-public-report.md @@ -0,0 +1,182 @@ +# Earlier public benchmark report (snapshot) + +Preserved from `docs/benchmark.md` at public master commit `cc54fb5aee50ec177402e459b23be0fe8da7836e`. The content below is historical; its gates and dated observations are not new measurements. In the one-task section, ContextBench scores measure code-context overlap, not patch correctness. The original external artifact availability was not reverified for this presentation. + +[Return to the current engineering note](./benchmark.md). + +--- + +# Benchmarks + +This page tracks two separate benchmark surfaces: + +- The ContextBench implementation pilot, which uses the official ContextBench evaluator on one frozen task and five scoreable lanes. +- The older discovery benchmark, which measures local discovery usefulness and payload cost only. + +Neither section currently supports a broad benchmark-win claim. + +## ContextBench Implementation Pilot + +This is the current implementation-quality pilot for ContextBench. It is real scoreable evidence, but it is still a pilot because it covers one frozen task rather than the full frozen 20-task slice. + +### Scope + +- Protocol: `tests/fixtures/contextbench-benchmark-protocol.json` +- Task manifest: `tests/fixtures/contextbench-task-manifest.json` +- Selection file: `scripts/contextbench-five-lane-selections.json` +- Workflow: `.github/workflows/contextbench-five-lane-score.yml` +- Required lanes: `raw-native`, `codebase-context`, `codebase-memory-mcp`, `grepai`, `ripgrep-lexical` +- Model used for selection: `gpt-5.4-mini-high` +- Target task: `SWE-Bench-Pro__go__maintenance__bugfix__4df06349` +- Repository under test: `navidrome/navidrome` +- Base commit: `537e2fc033b71a4a69190b74f755ebc352bb4196` + +`CodeGraphContext` is not counted in this five-lane pilot because its supported CLI path indexed successfully but returned zero task-relevant candidates during readiness. That remains a readiness blocker, not a quality result. + +### Current Audited Run + +- Run: `25663469903` +- Job: `75329796667` +- Commit: `bbd3a8348aaec15809fd09dd8fc729e64df6d878` +- Artifact: `6915576867` +- Artifact digest: `sha256:718fd32049a2d98ed62fb0c15189d7dc9f1b027c202f286923de91d9f8985def` +- Artifact size: `88.9 KB` +- Uploaded files: `42` +- Status: `success` + +The artifact contains `summary.json`, `publishable-summary.json`, `publishable-validation.json`, `humanized-summary.md`, logs, lane selections, lane predictions, and official evaluator score files. It intentionally excludes full cloned repos and evaluator caches so the evidence package is small enough to inspect. + +### Quality Results + +Only rows scored by the official ContextBench evaluator are included here. Setup failures, tool errors, empty predictions, and judge failures are reliability outcomes, not quality rows. + +| Lane | File cov | File prec | Span cov | Span prec | Line cov | Line prec | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| `raw-native` | 0.222 | 0.667 | 0.370 | 0.391 | 0.365 | 0.365 | +| `codebase-context` | 0.889 | 1.000 | 0.899 | 0.356 | 0.887 | 0.323 | +| `codebase-memory-mcp` | 0.222 | 0.667 | 0.346 | 0.380 | 0.315 | 0.337 | +| `grepai` | 0.333 | 0.500 | 0.048 | 0.042 | 0.059 | 0.061 | +| `ripgrep-lexical` | 0.222 | 0.667 | 0.401 | 0.341 | 0.419 | 0.302 | + +### Cost And Telemetry + +The report separates setup, indexing, query, selector, evaluator, and row-wall timing from quality. It also reports candidate counts, candidate token estimates when available, prediction token estimates, and selector token telemetry fields. + +`n/a` means the measurement was explicitly unavailable, not zero and not a hidden failure. Current gaps are: + +- Selector wall-clock and provider token telemetry were not captured in this proof artifact. +- `raw-native`, `codebase-context`, and `codebase-memory-mcp` emitted candidate counts but not candidate-pack bytes. +- `codebase-context` readiness did not emit index/query duration in the source artifact. + +### Bias Controls + +The generated `publishable-validation.json` must pass these checks before the report is treated as evidence: + +- Quality rows come only from the official ContextBench evaluator. +- Failed or unscoreable rows stay out of the quality table. +- All required lanes are scoreable. +- Setup, index, and query costs are separate from quality. +- Timing and token fields exist or carry explicit unavailable reasons. +- The protocol is frozen and `claimAllowed` is `false`. +- The task manifest attests that lane outputs were not observed during task selection. + +### What This Supports + +- It supports saying that the benchmark harness can produce real official ContextBench scores across five lanes. +- It supports saying that setup/index/query cost and context/token cost are now tracked separately from quality. +- It supports saying that the one-task pilot found a strong `codebase-context` result on this specific task. + +### What This Does Not Support + +- It does not support claiming that `codebase-context` beats competitors overall. +- It does not support claiming patch correctness or productivity improvements. +- It does not replace the full frozen 20-task, repeated-run benchmark required for claim-bearing results. + +## Discovery Benchmark + +This section documents the current public discovery proof from the checked-in result artifacts on `master`. +It is a discovery benchmark, not an implementation-quality benchmark. + +## Discovery Scope + +- Frozen fixtures: + - `tests/fixtures/discovery-angular-spotify.json` + - `tests/fixtures/discovery-excalidraw.json` + - `tests/fixtures/discovery-benchmark-protocol.json` +- Frozen repos used in the current proof run: + - `repos/angular-spotify` + - `repos/excalidraw` +- Current gate artifact: + - `results/gate-evaluation.json` +- Comparator evidence: + - `results/comparator-evidence.json` + +## Discovery Reproduction + +Run the repo-local proof artifacts from the current `master` checkout: + +```bash +node scripts/run-eval.mjs repos/angular-spotify --mode=discovery --fixture-a=tests/fixtures/discovery-angular-spotify.json --skip-reindex --output=results/codebase-context-angular-spotify.json +node scripts/run-eval.mjs repos/excalidraw --mode=discovery --fixture-a=tests/fixtures/discovery-excalidraw.json --skip-reindex --output=results/codebase-context-excalidraw.json +node scripts/benchmark-comparators.mjs --repos repos/angular-spotify,repos/excalidraw --output results/comparator-evidence.json +node scripts/run-eval.mjs repos/angular-spotify repos/excalidraw --mode=discovery --fixture-a=tests/fixtures/discovery-angular-spotify.json --fixture-b=tests/fixtures/discovery-excalidraw.json --competitor-results=results/comparator-evidence.json --skip-reindex --output=results/gate-evaluation.json +``` + +## Discovery Current Result + +From `results/gate-evaluation.json`: + +- `status`: `pending_evidence` +- `suiteStatus`: `complete` +- `claimAllowed`: `false` +- `totalTasks`: `24` +- `averageUsefulness`: `0.75` +- `averageEstimatedTokens`: `1827.0833` +- `bestExampleUsefulnessRate`: `0.125` + +Repo-level outputs from the same rerun: + +| Repo | Tasks | Avg usefulness | Avg estimated tokens | Best-example usefulness | +| --- | ---: | ---: | ---: | ---: | +| `angular-spotify` | 12 | 0.8333 | 2138.4167 | 0.25 | +| `excalidraw` | 12 | 0.6667 | 1506.0833 | 0 | + +## Discovery Gate Truth + +The gate is intentionally still blocked. + +- The combined suite covers both public repos. +- `claimAllowed` remains `false` because comparator evidence still does not support a benchmark-win claim. +- Two comparator artifacts now return `status: "ok"`, but that does not yet close the gate: + - `raw Claude Code` still leaves the baseline `pending_evidence` because `averageFirstRelevantHit` is `null` + - `codebase-memory-mcp` now has real current metrics, but the gate still marks it `failed` on the frozen tolerance rule +- Three comparator lanes still fail setup entirely: `GrepAI`, `jCodeMunch`, and `CodeGraphContext`. + +## Discovery Comparator Reality + +The current comparator artifact records incomplete comparator evidence, not benchmark wins. + +| Comparator | Status | Current reason | +| --- | --- | --- | +| `codebase-memory-mcp` | comparator artifact: `ok`; gate: `failed` | Runs through the repaired graph-backed path and now records real metrics (`averageUsefulness: 0.1875`, `averageFirstRelevantHit: 1.2857`, `bestExampleUsefulnessRate: 0.5`), but the frozen gate still fails it on the required usefulness comparisons | +| `jCodeMunch` | `setup_failed` | `MCP error -32000: Connection closed` | +| `GrepAI` | `setup_failed` | Local Go binary and Ollama model path not present | +| `CodeGraphContext` | `setup_failed` | `MCP error -32000: Connection closed` | +| `raw Claude Code` | comparator artifact: `ok`; gate: `pending_evidence` | The explicit Haiku CLI runner now returns current metrics (`averageUsefulness: 0.0278`, `averageEstimatedTokens: 32.1667`), but the baseline still lacks `averageFirstRelevantHit`, so the gate keeps this lane as missing evidence | + +`CodeGraphContext` remains part of the frozen comparison frame. It is not omitted from the public story just because the lane still fails to start. + +## Discovery Important Limitations + +- This benchmark measures discovery usefulness and payload cost only. +- It does not measure implementation correctness, patch quality, or end-to-end task completion. +- Comparator setup remains environment-sensitive, and the checked-in comparator outputs still do not satisfy the frozen claim gate. +- The reranker cache is currently corrupted on this machine. During the proof rerun, search fell back to original ordering after `Protobuf parsing failed` while still completing the harness. +- `averageFirstRelevantHit` remains `null` in the current gate output, which is enough to keep the raw-Claude baseline in `pending_evidence`. + +## Discovery Claims Supported + +- It can support claims about the shipped discovery surfaces and their current measured outputs on the frozen public tasks. +- It can support claims that the proof gate is still blocked by comparator evidence. +- It cannot support claims that `codebase-context` beats the named comparators today. +- It cannot support claims about edit success, code quality, or implementation speed. diff --git a/docs/benchmark.md b/docs/benchmark.md index bf71928..73d073b 100644 --- a/docs/benchmark.md +++ b/docs/benchmark.md @@ -1,174 +1,80 @@ -# Benchmarks +# What our local code-context experiments show -This page tracks two separate benchmark surfaces: +This report describes local evidence from five ways of giving an AI coding agent context: Raw-native, Codebase Context, jCodeMunch, Repowise, and context-mode. The evidence is split into separate families because retrieval, repeated runs, full-agent observations, and a metered replay do not answer the same question. The families are not pooled into one score. -- The ContextBench implementation pilot, which uses the official ContextBench evaluator on one frozen task and five scoreable lanes. -- The older discovery benchmark, which measures local discovery usefulness and payload cost only. +The clearest current finding is narrow: in this fixed retrieval setup, Codebase Context recovered 25.7% of expected gold files at 11.5% file precision, while fresh indexing had material local cost. These measurements do not show better completed coding tasks. -Neither section currently supports a broad benchmark-win claim. +## Evidence families -## ContextBench Implementation Pilot +| Evidence family | Attempts | Completed | Failed or otherwise excluded | What it answers | +| --- | ---: | ---: | ---: | --- | +| Corrected retrieval | 100 | 99 of 100 | 1 failed | Expected-file coverage, file precision, and reported `peakPrivateGb` | +| Repeated retrieval | 300 | 288 of 300 | 12 failed | Repeatability, setup, and runtime history | +| Full-agent pilot | 30 | 30 of 30 | 0 | Observed token and task-time results across two frozen tasks | +| Metered replay (excluded) | 100 | 100 counter records in historical summary | Tool-use validity unresolved | Why those replay records are not scored | -This is the current implementation-quality pilot for ContextBench. It is real scoreable evidence, but it is still a pilot because it covers one frozen task rather than the full frozen 20-task slice. +The first two families contain 400 retrieval attempts. The full-agent pilot and the metered replay remain separate from those retrieval measurements. -### Scope +## Retrieval results -- Protocol: `tests/fixtures/contextbench-benchmark-protocol.json` -- Task manifest: `tests/fixtures/contextbench-task-manifest.json` -- Selection file: `scripts/contextbench-five-lane-selections.json` -- Workflow: `.github/workflows/contextbench-five-lane-score.yml` -- Required lanes: `raw-native`, `codebase-context`, `codebase-memory-mcp`, `grepai`, `ripgrep-lexical` -- Model used for selection: `gpt-5.4-mini-high` -- Target task: `SWE-Bench-Pro__go__maintenance__bugfix__4df06349` -- Repository under test: `navidrome/navidrome` -- Base commit: `537e2fc033b71a4a69190b74f755ebc352bb4196` +Expected-file coverage means expected gold files recovered divided by expected gold files, averaged within the corrected retrieval-only family. File precision means expected gold files recovered divided by all files returned. The table reports both alongside completion status and the reported `peakPrivateGb` telemetry field. The underlying unit of that field was not independently verified, so these values are not labelled as GiB or treated as install size. -`CodeGraphContext` is not counted in this five-lane pilot because its supported CLI path indexed successfully but returned zero task-relevant candidates during readiness. That remains a readiness blocker, not a quality result. +| Tool | Completed | Failed | Expected-file coverage | File precision | Reported `peakPrivateGb` | +| --- | ---: | ---: | ---: | ---: | ---: | +| Raw search | 20/20 | 0 | 4.3% | 4.0% | 0.06 | +| Codebase Context | 20/20 | 0 | 25.7% | 11.5% | 2.90 | +| jCodeMunch | 20/20 | 0 | 27.1% | 11.0% | 0.33 | +| Repowise | 19/20 | 1 | 20.1% (failed run = 0) | 12.0% | 4.82 | +| context-mode | 20/20 | 0 | 16.8% | 11.7% | 0.15 | -### Current Audited Run +The corrected retrieval family therefore includes 99 completed attempts and 1 failed attempt. Repowise is the only tool with a failed attempt in this table. Its 20.1% is failure-inclusive: the failed attempt is counted as zero. Raw search is a deterministic lexical adapter in this protocol, not a full normal coding-agent baseline. These values describe file retrieval and returned-file precision, not a coding score. The [sanitized evidence extract](../results/benchmark-presentation-evidence.json) records the exact values and source hashes. -- Run: `25663469903` -- Job: `75329796667` -- Commit: `bbd3a8348aaec15809fd09dd8fc729e64df6d878` -- Artifact: `6915576867` -- Artifact digest: `sha256:718fd32049a2d98ed62fb0c15189d7dc9f1b027c202f286923de91d9f8985def` -- Artifact size: `88.9 KB` -- Uploaded files: `42` -- Status: `success` +## Paired token and time observation -The artifact contains `summary.json`, `publishable-summary.json`, `publishable-validation.json`, `humanized-summary.md`, logs, lane selections, lane predictions, and official evaluator score files. It intentionally excludes full cloned repos and evaluator caches so the evidence package is small enough to inspect. +The full-agent pilot used `gpt-5.4-mini-high`, with three runs per tool and task. It covers one frozen investigation task for each of two open-source codebases. The two tasks are PonyC and fmt. The paired tables compare the raw-native agent lane with Codebase Context; all five lanes remain available in the evidence extract. -### Quality Results +### PonyC -Only rows scored by the official ContextBench evaluator are included here. Setup failures, tool errors, empty predictions, and judge failures are reliability outcomes, not quality rows. +| Tool | Input tokens | Cached input tokens | Output tokens | Reasoning output tokens | Task time | +| --- | ---: | ---: | ---: | ---: | ---: | +| Raw search | 653,348 | 610,304 | 9,011 | 4,494 | 185,061 ms | +| Codebase Context | 223,558 | 190,976 | 5,564 | 3,024 | 114,664 ms | -| Lane | File cov | File prec | Span cov | Span prec | Line cov | Line prec | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | -| `raw-native` | 0.222 | 0.667 | 0.370 | 0.391 | 0.365 | 0.365 | -| `codebase-context` | 0.889 | 1.000 | 0.899 | 0.356 | 0.887 | 0.323 | -| `codebase-memory-mcp` | 0.222 | 0.667 | 0.346 | 0.380 | 0.315 | 0.337 | -| `grepai` | 0.333 | 0.500 | 0.048 | 0.042 | 0.059 | 0.061 | -| `ripgrep-lexical` | 0.222 | 0.667 | 0.401 | 0.341 | 0.419 | 0.302 | +### fmt -### Cost And Telemetry +| Tool | Input tokens | Cached input tokens | Output tokens | Reasoning output tokens | Task time | +| --- | ---: | ---: | ---: | ---: | ---: | +| Raw search | 290,186 | 272,384 | 7,567 | 4,024 | 142,002 ms | +| Codebase Context | 247,868 | 206,848 | 7,930 | 4,784 | 143,643 ms | -The report separates setup, indexing, query, selector, evaluator, and row-wall timing from quality. It also reports candidate counts, candidate token estimates when available, prediction token estimates, and selector token telemetry fields. +All values above are medians across three `gpt-5.4-mini-high` runs for the named task and tool. Cached input is included in input and must not be added to input again. Each column is aggregated independently. Codebase Context used fewer input tokens in both examples, but its median task time was slightly longer on fmt. This does not establish better answers or patches. -`n/a` means the measurement was explicitly unavailable, not zero and not a hidden failure. Current gaps are: +The prompt requested at most 12 shell or tool calls. The CLI did not hard-enforce that request. The result describes observed agent behavior, not universal or hard-call-controlled savings. -- Selector wall-clock and provider token telemetry were not captured in this proof artifact. -- `raw-native`, `codebase-context`, and `codebase-memory-mcp` emitted candidate counts but not candidate-pack bytes. -- `codebase-context` readiness did not emit index/query duration in the source artifact. +## What can be reproduced from the public repository -### Bias Controls - -The generated `publishable-validation.json` must pass these checks before the report is treated as evidence: - -- Quality rows come only from the official ContextBench evaluator. -- Failed or unscoreable rows stay out of the quality table. -- All required lanes are scoreable. -- Setup, index, and query costs are separate from quality. -- Timing and token fields exist or carry explicit unavailable reasons. -- The protocol is frozen and `claimAllowed` is `false`. -- The task manifest attests that lane outputs were not observed during task selection. - -### What This Supports - -- It supports saying that the benchmark harness can produce real official ContextBench scores across five lanes. -- It supports saying that setup/index/query cost and context/token cost are now tracked separately from quality. -- It supports saying that the one-task pilot found a strong `codebase-context` result on this specific task. - -### What This Does Not Support - -- It does not support claiming that `codebase-context` beats competitors overall. -- It does not support claiming patch correctness or productivity improvements. -- It does not replace the full frozen 20-task, repeated-run benchmark required for claim-bearing results. - -## Discovery Benchmark - -This section documents the current public discovery proof from the checked-in result artifacts on `master`. -It is a discovery benchmark, not an implementation-quality benchmark. - -## Discovery Scope - -- Frozen fixtures: - - `tests/fixtures/discovery-angular-spotify.json` - - `tests/fixtures/discovery-excalidraw.json` - - `tests/fixtures/discovery-benchmark-protocol.json` -- Frozen repos used in the current proof run: - - `repos/angular-spotify` - - `repos/excalidraw` -- Current gate artifact: - - `results/gate-evaluation.json` -- Comparator evidence: - - `results/comparator-evidence.json` - -## Discovery Reproduction - -Run the repo-local proof artifacts from the current `master` checkout: +The public repository snapshot at commit `cc54fb5` contains the [benchmark script](https://github.com/PatrickSys/codebase-context/blob/cc54fb5aee50ec177402e459b23be0fe8da7836e/scripts/contextbench-runner.mjs), [benchmark rules](https://github.com/PatrickSys/codebase-context/blob/cc54fb5aee50ec177402e459b23be0fe8da7836e/tests/fixtures/contextbench-benchmark-protocol.json), and [20-task manifest](https://github.com/PatrickSys/codebase-context/blob/cc54fb5aee50ec177402e459b23be0fe8da7836e/tests/fixtures/contextbench-task-manifest.json). Those links were reachable on October 5, 2026; earlier local-commit links were not. You can validate the checked-in fixtures with: ```bash -node scripts/run-eval.mjs repos/angular-spotify --mode=discovery --fixture-a=tests/fixtures/discovery-angular-spotify.json --skip-reindex --output=results/codebase-context-angular-spotify.json -node scripts/run-eval.mjs repos/excalidraw --mode=discovery --fixture-a=tests/fixtures/discovery-excalidraw.json --skip-reindex --output=results/codebase-context-excalidraw.json -node scripts/benchmark-comparators.mjs --repos repos/angular-spotify,repos/excalidraw --output results/comparator-evidence.json -node scripts/run-eval.mjs repos/angular-spotify repos/excalidraw --mode=discovery --fixture-a=tests/fixtures/discovery-angular-spotify.json --fixture-b=tests/fixtures/discovery-excalidraw.json --competitor-results=results/comparator-evidence.json --skip-reindex --output=results/gate-evaluation.json +node scripts/contextbench-runner.mjs --validate-fixtures ``` -## Discovery Current Result - -From `results/gate-evaluation.json`: - -- `status`: `pending_evidence` -- `suiteStatus`: `complete` -- `claimAllowed`: `false` -- `totalTasks`: `24` -- `averageUsefulness`: `0.75` -- `averageEstimatedTokens`: `1827.0833` -- `bestExampleUsefulnessRate`: `0.125` - -Repo-level outputs from the same rerun: - -| Repo | Tasks | Avg usefulness | Avg estimated tokens | Best-example usefulness | -| --- | ---: | ---: | ---: | ---: | -| `angular-spotify` | 12 | 0.8333 | 2138.4167 | 0.25 | -| `excalidraw` | 12 | 0.6667 | 1506.0833 | 0 | - -## Discovery Gate Truth - -The gate is intentionally still blocked. +They show the test structure and task format. They are not a byte-for-byte copy of every file used by the measured sessions; all three file hashes differ from the later 30-session pilot. The manifest names tasks and base commits but does not contain their full problem statements or gold-context values. The exact machine-specific raw run directories behind the 100-attempt retrieval table, the 300-attempt repeatability history, and the 30-session agent pilot are not checked into the public repository, so this checkout cannot regenerate the reported tables with one command. The [original schema-2 retrieval summary](../results/contextbench-engineering-note-summary.json) is preserved from local commit `18e0a43a90fe67e9df0d728b2134e9d430469883`; the [sanitized evidence extract](../results/benchmark-presentation-evidence.json) preserves the exact values and hashes used for this presentation. Fixture validation checks fixture consistency, not the reported measurements. -- The combined suite covers both public repos. -- `claimAllowed` remains `false` because comparator evidence still does not support a benchmark-win claim. -- Two comparator artifacts now return `status: "ok"`, but that does not yet close the gate: - - `raw Claude Code` still leaves the baseline `pending_evidence` because `averageFirstRelevantHit` is `null` - - `codebase-memory-mcp` now has real current metrics, but the gate still marks it `failed` on the frozen tolerance rule -- Three comparator lanes still fail setup entirely: `GrepAI`, `jCodeMunch`, and `CodeGraphContext`. +Local session artifacts retained outside this public checkout record Windows x64 with 16 logical CPUs, about 24 GiB of RAM, Node.js 24.14.1, Claude Code 2.1.173, and `gpt-5.4-mini-high` for the later agent pilot. Readers cannot independently inspect those environment files from this checkout. Exact version pins for all five tools in the retrieval families were not captured consistently enough to publish a trustworthy version matrix. -## Discovery Comparator Reality +This makes the public evidence partially reproducible: the method and current fixtures are inspectable, while the exact reported runs are not independently replayable from this checkout alone. The tables should be read as a transparent engineering report, not a paper-grade reproduction package. -The current comparator artifact records incomplete comparator evidence, not benchmark wins. +## Method and limits -| Comparator | Status | Current reason | -| --- | --- | --- | -| `codebase-memory-mcp` | comparator artifact: `ok`; gate: `failed` | Runs through the repaired graph-backed path and now records real metrics (`averageUsefulness: 0.1875`, `averageFirstRelevantHit: 1.2857`, `bestExampleUsefulnessRate: 0.5`), but the frozen gate still fails it on the required usefulness comparisons | -| `jCodeMunch` | `setup_failed` | `MCP error -32000: Connection closed` | -| `GrepAI` | `setup_failed` | Local Go binary and Ollama model path not present | -| `CodeGraphContext` | `setup_failed` | `MCP error -32000: Connection closed` | -| `raw Claude Code` | comparator artifact: `ok`; gate: `pending_evidence` | The explicit Haiku CLI runner now returns current metrics (`averageUsefulness: 0.0278`, `averageEstimatedTokens: 32.1667`), but the baseline still lacks `averageFirstRelevantHit`, so the gate keeps this lane as missing evidence | +The retrieval percentages, precision values, and reported `peakPrivateGb` fields come from the corrected retrieval-only family. The repeated retrieval family is reported as completion and repeatability history rather than merged into that table. The full-agent pilot reports input, cached input, output, reasoning output, and task time for two frozen investigation tasks. The metered replay is retained as history only and excluded from effectiveness or token-saving claims while tool-use validity is unresolved. -`CodeGraphContext` remains part of the frozen comparison frame. It is not omitted from the public story just because the lane still fails to start. +A historical fresh isolated Codebase Context indexing observation with embeddings enabled used the Transformers provider and `Xenova/bge-small-en-v1.5`. Its median was about 12 minutes 31 seconds. This is a historical observation in methodology and limits, not a normal-use estimate or a cross-tool speed ranking. -## Discovery Important Limitations +That index observation does not describe normal warm use, incremental indexing, query latency, install time, or cross-tool index-speed ranking. Shared model-cache state, actual GPU use, the exact scan, analysis, embedding, and storage stage durations, and equivalent cold work across competitors were not established by that observation. -- This benchmark measures discovery usefulness and payload cost only. -- It does not measure implementation correctness, patch quality, or end-to-end task completion. -- Comparator setup remains environment-sensitive, and the checked-in comparator outputs still do not satisfy the frozen claim gate. -- The reranker cache is currently corrupted on this machine. During the proof rerun, search fell back to original ordering after `Protobuf parsing failed` while still completing the harness. -- `averageFirstRelevantHit` remains `null` in the current gate output, which is enough to keep the raw-Claude baseline in `pending_evidence`. +The benchmark does not measure patch correctness, end-to-end task completion, or a universal token or time result. It also does not combine the four evidence families into a pooled score. Setup and run failures remain part of the record so that incomplete attempts are visible. -## Discovery Claims Supported +Historical note: the [earlier public report archive](./benchmark-prior-public-report.md) contains a one-task official pilot and a separate 24-task discovery report. Those records and their comparator statuses remain separate, are not folded into this current four-family summary, and retain their claim gates. -- It can support claims about the shipped discovery surfaces and their current measured outputs on the frozen public tasks. -- It can support claims that the proof gate is still blocked by comparator evidence. -- It cannot support claims that `codebase-context` beats the named comparators today. -- It cannot support claims about edit success, code quality, or implementation speed. +See the [comparison table](./comparison-table.md) for the retrieval family in a compact form. The repository walkthrough is in the [demo](./demo.md). diff --git a/docs/capabilities.md b/docs/capabilities.md index 7545148..d5d5c2f 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -6,10 +6,10 @@ Technical reference for what `codebase-context` ships today. The public product The server supports two transport modes: -| Mode | Command | MCP endpoint | -| ------------------- | ------------------------------------------- | ---------------------------- | -| **stdio** (default) | `npx -y codebase-context` | Spawned process stdin/stdout | -| **HTTP** | `npx -y codebase-context --http [--port N]` | `http://127.0.0.1:3100/mcp` | +| Mode | Command | MCP endpoint | +| ------------------- | ------------------------------------------------- | ---------------------------- | +| **stdio** (default) | `npx -y codebase-context@2.2.0` | Spawned process stdin/stdout | +| **HTTP** | `npx -y codebase-context@2.2.0 --http [--port N]` | `http://127.0.0.1:3100/mcp` | HTTP defaults to `127.0.0.1:3100`. Override with `--port`, `CODEBASE_CONTEXT_PORT`, or `server.port` in `~/.codebase-context/config.json`. @@ -26,7 +26,7 @@ Copy-pasteable client config templates are shipped in the package: - `templates/mcp/stdio/.mcp.json` — stdio setup for `.mcp.json`-style clients - `templates/mcp/http/.mcp.json` — HTTP setup for `.mcp.json`-style clients -Client transport support varies — see [README.md](../README.md) for a per-client matrix covering Claude Code, Cursor, Codex, Windsurf, VS Code, Claude Desktop, and OpenCode. +Use Node.js 22 or newer with the published 2.2.0 commands shown here. For client registration recipes and their verification limits, see the [client setup guide](./client-setup.md). ## CLI Reference @@ -49,15 +49,15 @@ For a command gallery with examples, see `docs/cli.md`. | `memory add` | `--type`, `--category`, `--memory`, `--reason` | `remember` | | `memory remove ` | — | — | -All commands accept `--json` for raw JSON output. Errors go to stderr with exit code 1. +Commands that list `--json` above support raw JSON output. For MCP client registration, follow the [published client setup recipes](./client-setup.md); do not use the broken `init` setup command in published 2.2.0. Errors go to stderr with exit code 1. ```bash # Quick examples -npx codebase-context status -npx codebase-context search --query "auth middleware" --intent edit -npx codebase-context refs --symbol "UserService" --limit 10 -npx codebase-context cycles --scope src/features -npx codebase-context reindex --incremental +npx -y codebase-context@2.2.0 status +npx -y codebase-context@2.2.0 search --query "auth middleware" --intent edit +npx -y codebase-context@2.2.0 refs --symbol "UserService" --limit 10 +npx -y codebase-context@2.2.0 cycles --scope src/features +npx -y codebase-context@2.2.0 reindex --incremental ``` ## Tool Surface @@ -282,29 +282,25 @@ Notes: ## Evaluation Harness -Current public proof bundle: [`docs/benchmark.md`](../docs/benchmark.md) and [`docs/comparison-table.md`](../docs/comparison-table.md). +Current public evidence: [`docs/benchmark.md`](../docs/benchmark.md) and [`docs/comparison-table.md`](../docs/comparison-table.md). -Reproducible evaluation is shipped as a CLI entrypoint backed by shared scoring/reporting code. +Reproducible evaluation is shipped as a CLI entrypoint backed by shared scoring and reporting code. -- **Command:** `npm run eval -- [codebaseB] --mode retrieval|discovery [--competitor-results ]` (builds first, then runs `scripts/run-eval.mjs`) +- **Command:** `pnpm run eval -- [codebaseB] --mode retrieval|discovery [--competitor-results ]` (builds first, then runs `scripts/run-eval.mjs`) +- **Contributor requirement:** this repository command uses the pinned `pnpm@10.27.0`; end users do not need pnpm for the `npx` CLI. - **Shared implementation:** `src/eval/harness.ts`, `src/eval/discovery-harness.ts`, and `src/eval/types.ts` -- **Frozen retrieval fixtures:** - - `tests/fixtures/eval-angular-spotify.json` - - `tests/fixtures/eval-controlled.json` + `tests/fixtures/codebases/eval-controlled/` -- **Frozen discovery fixtures:** - - `tests/fixtures/discovery-angular-spotify.json` - - `tests/fixtures/discovery-excalidraw.json` - - `tests/fixtures/discovery-benchmark-protocol.json` -- **Retrieval metrics:** Top-1 accuracy, Top-3 recall, spec contamination rate, and a gate pass/fail +- **Retrieval fixtures:** `tests/fixtures/eval-angular-spotify.json` and `tests/fixtures/eval-controlled.json` with `tests/fixtures/codebases/eval-controlled/` +- **Discovery fixtures:** `tests/fixtures/discovery-angular-spotify.json`, `tests/fixtures/discovery-excalidraw.json`, and `tests/fixtures/discovery-benchmark-protocol.json` +- **Retrieval metrics:** Top-1 accuracy, Top-3 recall, spec contamination rate, and a pass/fail result for the configured retrieval check - **Discovery metrics:** usefulness score, payload bytes, estimated tokens, first relevant hit, and best-example usefulness -- **Discovery gate:** discovery mode evaluates the frozen ship gate only when the full public suite and comparator metrics are available; missing comparator evidence is reported as pending, not silently treated as pass/fail -- **Current checked-in gate truth:** `results/gate-evaluation.json` remains `pending_evidence` with `claimAllowed: false`; the raw-Claude baseline still lacks `averageFirstRelevantHit`, `codebase-memory-mcp` still fails the frozen usefulness comparisons, and the remaining named lanes are still `setup_failed` -- **Limits:** discovery mode is discovery-only, uses current shipped surfaces only, and does not claim implementation quality; named competitor runs remain a documented hybrid/manual lane rather than a built-in automated benchmark +- **Public benchmark families:** the current documentation keeps 100 corrected retrieval attempts, 300 repeated retrieval attempts, 30 full-agent pilot attempts, and 100 invalid-token replays separate across five local code-context tools +- **Discovery limits:** discovery mode describes the shipped discovery surface and does not measure patch correctness or end-to-end task completion; named competitor runs remain a documented hybrid/manual lane rather than a built-in automated benchmark +- **Evidence limits:** missing or failed setup and comparator runs remain visible in their source artifacts and are not treated as a pass ## Limitations - **Symbol refs are not a call-graph.** `get_symbol_references` counts identifier-node occurrences in the AST (comments/strings excluded via Tree-sitter). It does not distinguish call sites from type annotations, variable assignments, or imports. Full call-site-specific analysis (`call_expression` nodes only) is a roadmap item. - **Impact is 2-hop max.** `computeImpactCandidates` walks direct importers then their importers. Full BFS reachability is on the roadmap. -- **Angular, React, Next.js, and NestJS have dedicated analyzers.** All other languages go through the Generic analyzer (30+ languages, chunking + import graph, no framework-specific signal extraction). +- **Published 2.2.0 has dedicated Angular, React, and Next.js analyzers.** NestJS support is in a newer source candidate and is not part of published 2.2.0. Other languages use the Generic analyzer (30+ languages, chunking + import graph, no framework-specific signal extraction). - **Default embedding model is `bge-small-en-v1.5` (512-token context).** Granite (8192 context) is opt-in via `EMBEDDING_MODEL`. OpenAI is opt-in via `EMBEDDING_PROVIDER=openai` — sends code externally. - **Patterns are file-level frequency counts.** Not semantic clustering. Rising/Declining trend is derived from git commit recency for files using each pattern, not from usage semantics. diff --git a/docs/cli.md b/docs/cli.md index 5025f65..dfda760 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -10,16 +10,18 @@ > Output depends on the repo you run it against. The examples below are illustrative (paths, counts, and detected frameworks will vary). > -> The CLI is intentionally single-project per invocation. MCP multi-project routing and trusted-root auto-discovery are only for the MCP server; the CLI still targets one root via `CODEBASE_ROOT` or the current working directory. +> These examples show an earlier CLI run; they do not verify the current package or an MCP connection. +> +> The commands below use published `codebase-context@2.2.0` and require Node.js 22 or newer. The CLI is intentionally single-project per invocation and targets one root via `CODEBASE_ROOT` or the current working directory. MCP registration and project selection are covered in the [client setup guide](./client-setup.md). ## How to run ```bash -# Run from a repo root, or set CODEBASE_ROOT explicitly: -CODEBASE_ROOT=/path/to/repo npx -y codebase-context status +# CLI only: run from a repo root, or set CODEBASE_ROOT explicitly: +CODEBASE_ROOT=/path/to/repo npx -y codebase-context@2.2.0 status -# Every command supports --json (machine output). Human mode is default. -npx -y codebase-context patterns --json +# Commands that show --json in their help support machine output. Human mode is default. +npx -y codebase-context@2.2.0 patterns --json ``` ### ASCII fallback @@ -27,7 +29,7 @@ npx -y codebase-context patterns --json If your terminal doesn’t render Unicode box-drawing cleanly: ```bash -CODEBASE_CONTEXT_ASCII=1 npx -y codebase-context patterns +CODEBASE_CONTEXT_ASCII=1 npx -y codebase-context@2.2.0 patterns ``` ## Commands @@ -42,13 +44,27 @@ CODEBASE_CONTEXT_ASCII=1 npx -y codebase-context patterns - `reindex` — rebuild index (full or incremental) - `style-guide` — find style guide sections in docs - `memory list|add|remove` — manage team memory (stored in `.codebase-context/memory.json`) +- MCP client registration — use the published 2.2.0 commands in the [client setup guide](./client-setup.md); published `init` is not a reliable setup route --- +## First use + +Run these CLI commands from the project root. Build or refresh the local index first; the first index may take a while while the project is scanned and local embeddings are created. Then inspect the conventions map and search for the code you need: + +```bash +npx -y codebase-context@2.2.0 reindex +npx -y codebase-context@2.2.0 map +npx -y codebase-context@2.2.0 search --query "auth middleware" +``` + +This is the CLI preparation flow. It is separate from MCP client registration and does not by itself show that an MCP client connected successfully. + ## `map` ```bash -npx -y codebase-context map +npx -y codebase-context@2.2.0 reindex +npx -y codebase-context@2.2.0 map ``` The conventions map - run this first on an unfamiliar repo. It shows architecture layers, active patterns with adoption rates and trend direction, and the golden files the team treats as the strongest examples. This is also what the MCP server delivers to AI agents via the `codebase://context` resource on first call, before search narrows to a specific local example. @@ -83,7 +99,7 @@ Example output (truncated): ## `metadata` ```bash -npx -y codebase-context metadata +npx -y codebase-context@2.2.0 metadata ``` Example output: @@ -104,7 +120,7 @@ Example output: ## `patterns` ```bash -npx -y codebase-context patterns +npx -y codebase-context@2.2.0 patterns ``` Example output (truncated): @@ -126,7 +142,8 @@ Example output (truncated): ## `search` ```bash -npx -y codebase-context search --query "file watcher" --intent edit --limit 3 +npx -y codebase-context@2.2.0 reindex +npx -y codebase-context@2.2.0 search --query "file watcher" --intent edit --limit 3 ``` Example output (truncated): @@ -147,7 +164,7 @@ Example output (truncated): ## `refs` ```bash -npx -y codebase-context refs --symbol "startFileWatcher" --limit 10 +npx -y codebase-context@2.2.0 refs --symbol "startFileWatcher" --limit 10 ``` Example output (truncated): @@ -166,7 +183,7 @@ Example output (truncated): ## `cycles` ```bash -npx -y codebase-context cycles --scope src +npx -y codebase-context@2.2.0 cycles --scope src ``` Example output: @@ -182,7 +199,7 @@ Example output: ## `status` ```bash -npx -y codebase-context status +npx -y codebase-context@2.2.0 status ``` Example output: @@ -201,8 +218,8 @@ Example output: ## `reindex` ```bash -npx -y codebase-context reindex -npx -y codebase-context reindex --incremental --reason "changed watcher logic" +npx -y codebase-context@2.2.0 reindex +npx -y codebase-context@2.2.0 reindex --incremental --reason "changed watcher logic" ``` > **MCP server mode**: if you're running codebase-context as an MCP server (long-running process), the index auto-refreshes via a file watcher — you don't need to call `reindex` between edits. Use `reindex` for one-shot CLI runs or to force a full rebuild. @@ -210,7 +227,7 @@ npx -y codebase-context reindex --incremental --reason "changed watcher logic" ## `style-guide` ```bash -npx -y codebase-context style-guide --query "naming" +npx -y codebase-context@2.2.0 style-guide --query "naming" ``` Example output: @@ -223,14 +240,14 @@ No style guides found. ## `memory` ```bash -npx -y codebase-context memory list -npx -y codebase-context memory list --query "watcher" +npx -y codebase-context@2.2.0 memory list +npx -y codebase-context@2.2.0 memory list --query "watcher" -npx -y codebase-context memory add \ +npx -y codebase-context@2.2.0 memory add \ --type gotcha \ --category tooling \ --memory "Use pnpm, not npm" \ --reason "Workspace support and speed" -npx -y codebase-context memory remove +npx -y codebase-context@2.2.0 memory remove ``` diff --git a/docs/client-setup.md b/docs/client-setup.md index 032b0cf..7770e7f 100644 --- a/docs/client-setup.md +++ b/docs/client-setup.md @@ -1,146 +1,265 @@ # Client Setup -Full setup instructions for each AI client. This guide is about transport and wiring, not a different product mode: each client gets the same bounded conventions map first and local-pattern discovery second. For the quick-start summary, see [README.md](../README.md). +Codebase Context runs as a local MCP server: a tool your AI editor or command-line client can call while it works. Register it without a fixed folder, then let the agent supply its current repository through the `project` tool parameter. These examples use published npm `2.2.0`. -## Transport modes +Use Node.js 22 or newer for these recipes. The package declares Node 18+, but native dependency requirements are narrower than that broad declaration; these checks do not establish Node 18.0 compatibility. -| Mode | How it runs | When to use | -| ---- | ----------- | ------------ | -| **stdio** (default) | Process spawned by the client | One AI client, simple setup | -| **HTTP** | Long-lived server at `http://127.0.0.1:3100/mcp` | Multiple clients sharing one server | +The copyable registration commands target PowerShell or a POSIX terminal. Windows Command Prompt was not tested; OpenCode's quoted separator is shell-dependent. -Start the HTTP server: +## Current published-package recipes (updated 2026-10-06) -```bash -npx -y codebase-context --http # default port 3100 -npx -y codebase-context --http --port 4000 -``` +Claude Code, Codex CLI and OpenCode registration commands were checked again on October 6 without a project argument. The other clients retain their older config-shape evidence; no-folder native agent use is unverified for them. See the dated notes below. -Copy-pasteable templates: [`templates/mcp/stdio/.mcp.json`](../templates/mcp/stdio/.mcp.json) and [`templates/mcp/http/.mcp.json`](../templates/mcp/http/.mcp.json). +Each client section says how far its isolated Windows check went: -## Project routing contract +| Status | What it means | +| ------------------------------ | ------------------------------------------------------------------------------------------ | +| **Connected locally** | The isolated client started and connected to the server. | +| **Config recognized locally** | The isolated client found the config but still required approval before loading it. | +| **Config written locally** | The isolated command wrote the expected config. A live agent session was not claimed. | +| **Official instructions only** | The config follows current official docs, but this client was not fully exercised locally. | -Automatic multi-project routing is evidence-backed only when the MCP host announces workspace roots. Treat that as the primary path. +These checks cover setup and startup state. They do not measure how an agent handles a coding task. -If the host does not send roots, or still cannot tell which project is active, use one of the explicit fallbacks instead: +After registration, start a new agent session in your project and ask: -- start the server with a single bootstrap path -- set `CODEBASE_ROOT` -- retry tool calls with `project` +> Use Codebase Context to find [feature] in this repository. Pass this repository's absolute path as project when checking get_indexing_status and searching. Wait for indexing if needed, read codebase://context, then search_codebase and open a returned source file. Show me the relevant files. -If multiple projects are available and no active project can be inferred safely, the server returns `selection_required` instead of guessing. +The first index may take a while and download a local model. You can prepare it separately with `npx -y codebase-context@2.2.0 reindex` from the project folder. Index readiness, client connection and useful agent behavior are separate checks. ## Claude Code +**Status:** Connected locally + +**Checked:** 2026-10-06 with Claude Code 2.1.291 + +**Scope:** User scope makes the registration available across projects. The agent selects the repository in tool calls. + +User setup: + ```bash -claude mcp add codebase-context -- npx -y codebase-context +claude mcp add --scope user --transport stdio codebase-context -- npx -y codebase-context@2.2.0 ``` -Claude Code only supports stdio. HTTP is not available for this client. +Optional shared project setup: -## Claude Desktop +```bash +claude mcp add --scope project --transport stdio codebase-context -- npx -y codebase-context@2.2.0 +``` -Add to `claude_desktop_config.json`: +`--scope project` writes shared `.mcp.json` and requires project-server approval. `--scope local` instead stores a private association for the current project in `~/.claude.json`; it does not write `.mcp.json`. -```json -{ - "mcpServers": { - "codebase-context": { - "command": "npx", - "args": ["-y", "codebase-context"] - } - } -} +**What was checked:** With isolated user configuration, the no-folder user command wrote the expected stdio arguments and `claude mcp get` reported Connected. + +**Limit:** Connection proof does not establish an agent investigation. The optional shared-project variant follows the official scope contract; it was not exercised in this October check. + +**Official docs:** https://code.claude.com/docs/en/mcp + +## Codex CLI + +**Status:** Config written locally + +**Checked:** 2026-10-06 with Codex CLI 0.160.0 + +**Scope:** The command writes user config. A trusted project can use a project `.codex/config.toml` instead. + +```bash +codex mcp add codebase-context -- npx -y codebase-context@2.2.0 ``` -Claude Desktop only supports stdio. +Equivalent project config: -## Cursor +```toml +[mcp_servers.codebase-context] +command = "npx" +args = ["-y", "codebase-context@2.2.0"] +startup_timeout_sec = 120 +``` -**Stdio** — add to `.cursor/mcp.json` in your project (copy from [`templates/mcp/stdio/.mcp.json`](../templates/mcp/stdio/.mcp.json)): +**What was checked:** The command ran with an isolated `CODEX_HOME`, wrote `config.toml`, and `codex mcp get` returned the same command and arguments. -```json -{ - "mcpServers": { - "codebase-context": { - "command": "npx", - "args": ["-y", "codebase-context"] - } - } -} +**Limit:** The CLI command is user-scoped. A trusted project can instead use .codex/config.toml. The double dash is required. + +**Official docs:** [Codex MCP configuration](https://learn.chatgpt.com/docs/extend/mcp?surface=cli). The supported `startup_timeout_sec` option defaults to ten seconds; the project example allows 120 seconds for initial `npx` package preparation. This does not prove Desktop startup or guarantee completion of a cold download. + +## Codex Desktop: published 2.2.0 + +Create or merge `.codex/config.toml` in the project you want to search. Preserve any existing settings: + +```toml +[mcp_servers.codebase-context] +command = "npx" +args = ["-y", "codebase-context@2.2.0"] +startup_timeout_sec = 120 ``` -**HTTP** — start the server first, then add to `.cursor/mcp.json` (copy from [`templates/mcp/http/.mcp.json`](../templates/mcp/http/.mcp.json)): +Trust the project if asked, restart Codex and start a new task there. Use the first-use prompt above so the agent selects the current repository. The CLI command in the preceding section writes user-level config; this recipe uses a trusted project's config instead. -```json -{ - "mcpServers": { - "codebase-context": { - "type": "http", - "url": "http://127.0.0.1:3100/mcp" - } - } -} +**Verification boundary:** isolated Windows CLI and SDK/MCP checks on October 5 indexed a six-file fixture with published `2.2.0`, selected its project, retrieved context and found a source that the host could open. The exact `npx -y codebase-context@2.2.0 ` entrypoint also passed using a warmed isolated npm cache and prepared index. This does not establish a fully cold download or a fresh Codex Desktop agent session. The September Desktop session below used a packed candidate; keep it separate from published-package acceptance. + +On October 6 the exact published `npx` entrypoint was also tested without a folder, environment root or MCP roots, launched from an unrelated directory. Its first tool call returned `selection_required`; an explicit `project` call selected the prepared fixture, search found the expected symbol, context returned its map, and the runner host opened the returned source. A later call without `project` retained that selection. This is SDK/protocol and runner-host evidence with warmed caches and a pre-indexed fixture, not a native Desktop task. + +## Unreleased installer candidate + +The next release is expected to support one project-only command from the target project: + +The `init --client codex --yes` flow is an unreleased source candidate. Published npm `2.2.0` does not include these flags. Use the published configuration above rather than an `@latest` installer command; candidate verification uses an exact built or installed candidate. + +A successful `init --client codex --yes` writes the project-scoped `.codex/config.toml` and prepares the index for that root. If config writing or index preparation fails, setup is incomplete and Codex is not ready to use. + +For candidate verification before publication, build the source checkout and run its absolute `dist/index.js` path from the project you want to configure: + +```powershell +# In the codebase-context source checkout +npm run build + +# In the target project, using the built CLI path +node C:\path\to\codebase-context\dist\index.js init --client codex --yes ``` -## Windsurf +Use `--root C:\path\to\project` when the command is run from elsewhere. Here, `root` means the project folder to configure and index. The candidate writes the project-scoped `/.codex/config.toml`, preserves unrelated Codex settings, prepares the index for that folder, and does not write `AGENTS.md` or global client configuration. After a successful result, trust the project in Codex Desktop if prompted, restart or reopen Codex, start a new task in that project, and ask a small project question. A written config and ready index do not prove that Codex loaded or queried the server. -Open Settings > MCP and add (stdio only — HTTP is not documented for Windsurf yet): +Before release, the generated `npx` server command still resolves the published package unless the candidate is installed in the target project's `node_modules`. Our packaged test installs the candidate in a disposable project and checks its file hash. Running the built setup command alone does not prove that Codex will launch that same candidate. -```json -{ - "mcpServers": { - "codebase-context": { - "command": "npx", - "args": ["-y", "codebase-context"] - } - } -} +**Checked on 2026-09-15:** the packed candidate installed and indexed an isolated Windows project. A fresh Codex Desktop task then called the actual `search_codebase` MCP tool and located the requested chart component with its source file and lines. The normal Desktop task flow established project trust; no trust-file edits or replacement CLI/SDK search were used for that answer. This covers the small prepared project with shared caches, not a cold-machine install or the published npm package. + +## Optional fixed-folder setup and CLI preparation + +To set a default project before the first tool call, append its full path to the server arguments, for example `args = ["-y", "codebase-context@2.2.0", "C:/projects/my-app"]`. Use a quoted literal shell argument in registration commands. This makes the default fixed across repositories when the registration is user-scoped; explicit tool selection can still choose another allowed project. + +To prepare the index separately, run from the project folder: + +```powershell +npx -y codebase-context@2.2.0 reindex ``` -## Codex +Do not use published `2.2.0`'s interactive `init` as the setup route. October 6 isolated checks found that it ignores `--help`, `--client` and `--yes`; its Claude/Codex registration executes `mcp` instead of the client executable, and its HTTP recipes require a separately started server. It can exit zero after registration fails. The unreleased installer above is a separate implementation. + +## Gemini CLI + +**Status:** Config written locally + +**Checked:** 2026-08-11 with Gemini CLI 0.41.1 + +**Scope:** User or project. -**Stdio:** +User setup: ```bash -codex mcp add codebase-context npx -y codebase-context +gemini mcp add --scope user codebase-context npx -y codebase-context@2.2.0 ``` -**HTTP** — start the server first (`npx -y codebase-context --http`), then save a config file and pass it: +Project setup: + +```bash +gemini mcp add --scope project codebase-context npx -y codebase-context@2.2.0 +``` + +Gemini writes an `mcpServers` entry in its settings. Claude, Codex, and Copilot CLI use `--`; Gemini does not. + +**What was checked:** Both commands wrote the expected config in isolated homes and projects. The user-scoped list found the server as disconnected; no live agent connection was claimed. + +**Limit:** Gemini does not use the double-dash separator. The isolated project-scoped list did not enumerate the project entry, so only config generation is proved. + +**Official docs:** https://geminicli.com/docs/tools/mcp-server/ + +## Cursor + +**Status:** Config recognized locally + +**Checked:** 2026-08-11 with Cursor 3.15.6 and Cursor Agent 2026.04.17-787b533 + +**Scope:** Project. + +Create `.cursor/mcp.json`: ```json { "mcpServers": { "codebase-context": { - "type": "http", - "url": "http://127.0.0.1:3100/mcp" + "command": "npx", + "args": ["-y", "codebase-context@2.2.0"] } } } ``` -```bash -codex --mcp-config /path/to/mcp-http.json -``` +Checked config shape: `.cursor/mcp.json with mcpServers.codebase-context command npx and args [-y, codebase-context]`. + +**What was checked:** Cursor Agent loaded the isolated workspace config and showed the server pending approval. + +**Limit:** User approval is required. The official one-click route exists, but a generated deep link was not accepted as proof. + +**Official docs:** https://docs.cursor.com/context/model-context-protocol + +## VS Code with GitHub Copilot -## VS Code (Copilot) +**Status:** Official instructions only -Add `.vscode/mcp.json` to your project root. VS Code uses `servers` instead of `mcpServers`: +**Checked:** 2026-08-11 against VS Code 1.125.1 help and current official docs + +**Scope:** User command or workspace config; not locally runtime-tested. + +Official user command: + +```powershell +code --add-mcp '{"name":"codebase-context","command":"npx","args":["-y","codebase-context@2.2.0"]}' +``` + +Deterministic workspace config at `.vscode/mcp.json`: ```json { "servers": { "codebase-context": { "command": "npx", - "args": ["-y", "codebase-context"] + "args": ["-y", "codebase-context@2.2.0"] } } } ``` +**What was checked:** Local CLI help exposes `--add-mcp`. An isolated GUI launch could not provide deterministic completion, so it was not counted as a local runtime check. + +**Limit:** The command is official but not accepted as a completed local runtime test. Workspace config remains the deterministic manual fallback. + +**Official docs:** https://code.visualstudio.com/docs/agent-customization/mcp-servers + +## GitHub Copilot CLI + +**Status:** Config written locally + +**Checked:** 2026-08-11 with GitHub Copilot CLI 1.0.24 + +**Scope:** User config. + +```bash +copilot mcp add codebase-context -- npx -y codebase-context@2.2.0 +``` + +**What was checked:** The command ran with an isolated `--config-dir` and wrote the expected `mcp-config.json` command, arguments, and tool filter. + +**Limit:** The CLI writes user configuration by default. VS Code workspace mcp.json is a separate surface. + +**Official docs:** https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/add-mcp-servers + ## OpenCode -Add `opencode.json` to your project root: +**Status:** Connected locally + +**Checked:** 2026-10-06 with stable OpenCode 1.18.34 + +**Scope:** User registration by default. + +Run once: + +```bash +opencode mcp add codebase-context '--' npx -y codebase-context@2.2.0 +``` + +Keep the quoted `'--'`: the installed Windows PowerShell npm shim consumes an unquoted separator. The quoted form also works in POSIX shells. + +For a project-local alternative, merge this into `opencode.json`: ```json { @@ -148,73 +267,103 @@ Add `opencode.json` to your project root: "mcp": { "codebase-context": { "type": "local", - "command": ["npx", "-y", "codebase-context"], - "enabled": true + "command": ["npx", "-y", "codebase-context@2.2.0"], + "enabled": true, + "timeout": 120000 } } } ``` -OpenCode also supports interactive setup via `opencode mcp add`. +Checked config shape: `opencode.json with a local command array and timeout 120000`. -## Single-project fallback +The longer timeout is intentional because the isolated cold start took about 38 seconds while `npx` prepared the package and server. -If you only use one repo, append a project path: +**What was checked:** With home, config, data, cache, temporary and workspace paths redirected, the no-folder command wrote user configuration and `opencode mcp list` reported Connected. An August 11 check with OpenCode 1.18.5 also connected using the project config shape and an explicit folder; the no-folder project variant is not a new native acceptance check. -```bash -codex mcp add codebase-context npx -y codebase-context "/path/to/your/project" -``` +**Limit:** Stable OpenCode 1.x and beta OpenCode 2 use different native config shapes. The stable V1 shape is the public default; do not mix schemas. -Or set an environment variable: +**Official docs:** https://opencode.ai/docs/mcp-servers/ -```bash -CODEBASE_ROOT=/path/to/your/project -``` +## Windsurf -## Test a local build +**Status:** Official instructions only -Build the local branch first: +**Checked:** 2026-08-11 against current official docs; Windsurf was not installed locally -```bash -pnpm build -``` +**Scope:** User config. -Then point your MCP client at the local build: +Create `~/.codeium/windsurf/mcp_config.json`: ```json { "mcpServers": { "codebase-context": { - "command": "node", - "args": ["/dist/index.js"] + "command": "npx", + "args": ["-y", "codebase-context@2.2.0"] } } } ``` -If the default setup is not enough for your client, pass a project path explicitly: +Checked config shape: `~/.codeium/windsurf/mcp_config.json with mcpServers.codebase-context command npx and args [-y, codebase-context]`. + +**What was checked:** The config shape was checked against the current official page. The local client was unavailable. + +**Limit:** Current ownership and local behavior are not independently verified on this machine; publish as official config, not locally tested. + +**Official docs:** https://docs.windsurf.com/windsurf/cascade/mcp + +## Optional advanced HTTP server + +Use `stdio` for the simplest setup. HTTP can let several clients share one long-running local process, but support depends on the client and this path is not part of the default setup proof: + +```bash +npx -y codebase-context@2.2.0 --http +npx -y codebase-context@2.2.0 --http --port 4000 +``` + +The documented default endpoint is `http://127.0.0.1:3100/mcp`. Config-shape templates are available in [`templates/mcp/stdio/.mcp.json`](../templates/mcp/stdio/.mcp.json) and [`templates/mcp/http/.mcp.json`](../templates/mcp/http/.mcp.json); pin the published package as in the recipes above. HTTP does not discover a client's project folder; the agent still has to select its repository. + +The [MCP July 28, 2026 HTTP revision](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports/streamable-http) removed protocol sessions and the GET stream endpoint. This source checkout uses SDK 1.27.1 and the earlier protocol/session model. The stdio checks do not validate its concurrent HTTP project isolation or compatibility with the new revision. A protocol migration and host testing are separate from these setup recipes. + +Client support for HTTP differs. Follow that client's official docs and keep stdio as the fallback. + +## Project routing contract + +The documented default is agent-directed selection. Start the stdio server without a folder; call `get_indexing_status` with the current repository's absolute path as `project`, wait for readiness and pass the same `project` when searching. A successful selection becomes the default for later calls in that server process, including `codebase://context`. Select another project explicitly when changing repositories. This lets one registration serve multiple projects without changing client configuration. + +CBC deliberately does not index the server process's working directory when no project is configured. Without a configured folder, known root or explicit selection, tools return `selection_required`. They do not guess or index the user's home directory. + +Some clients also announce workspace roots. One valid root can auto-select; several roots with no active selection return `selection_required` and available projects. Explicit selectors must stay within announced roots. Relative monorepo paths require a known workspace context; use absolute repository paths for the portable first-use recipe. + +[MCP deprecated Roots on July 28, 2026](https://modelcontextprotocol.io/specification/2026-07-28/client/roots), recommending explicit directories/files in tool parameters, resource URIs or server configuration. CBC's older Roots support remains a compatibility aid rather than a required setup step. + +October 6 verification: 18 focused current-source routing tests passed, including ambiguous roots, explicit selection, subsequent calls and root changes. The exact published server also switched between two isolated prepared repositories A to B to A in one no-roots stdio process: search paths, context map and subsequent status calls followed each selection. The runner host opened the expected distinct sources. A native agent session across real repositories remains a validation step; stdio process-local selection does not establish concurrent HTTP isolation. + +A follow-up recorded raw `search_codebase` calls containing only `query` and `mode`, with no `project`: they returned A, B and A after those explicit selections. The caller does not automatically add the full path; CBC routes omitted-project calls using its selected project. This state lasts for that stdio process. Select again after a server restart or when changing repositories. + +A native Codex CLI controlled-fixture trial with an ordinary question supplied the correct absolute `project` without path instructions. Its MCP call was denied by the noninteractive run's `never` approval policy, so native retrieval and continuation did not pass. If a host refuses a tool call for approval, repeating the folder path will not resolve that permission issue; use the client's trust/approval flow. The test retained the denial and did not count fallback source/index reads as CBC acceptance. + +## Test a local build + +Build the source checkout: + +```bash +pnpm build +``` + +Then replace the `npx` command in your client config with the local entry point: ```json { "mcpServers": { "codebase-context": { "command": "node", - "args": ["/dist/index.js", "/path/to/your/project"] + "args": ["/dist/index.js"] } } } ``` -Check these three flows: - -1. **Single project** — call `search_codebase` or `metadata`. Routing is automatic. - -2. **Multiple projects on a roots-capable host** — open two repos or a monorepo. Call `codebase://context`. Expected: workspace overview, then automatic routing once a project is active. - -3. **Ambiguous or no-roots selection** — start without a bootstrap path, call `search_codebase`. Expected: `selection_required`. Retry with `project` set to `apps/dashboard` or `/repos/customer-portal`. - -For monorepos, test all three selector forms: - -- relative subproject path: `apps/dashboard` -- repo path: `/repos/customer-portal` -- file path: `/repos/monorepo/apps/dashboard/src/auth/guard.ts` +For one explicit project, append its path to the arguments or set `CODEBASE_ROOT`. diff --git a/docs/comparison-table.md b/docs/comparison-table.md index aee2eaa..1b55f11 100644 --- a/docs/comparison-table.md +++ b/docs/comparison-table.md @@ -1,35 +1,26 @@ -# Comparator Summary +# Corrected retrieval comparison -This table summarizes the current comparator evidence from `results/comparator-evidence.json`. -It is a setup-status table first, not a marketing scoreboard. +This table summarizes the corrected retrieval-only family across five local code-context tools. In this fixed setup, Codebase Context recovered 25.7% of expected gold files at 11.5% file precision; this does not show better completed coding tasks. Expected-file coverage means expected gold files recovered divided by expected gold files. File precision means expected gold files recovered divided by all files returned. The reported `peakPrivateGb` field is copied from retained telemetry; its underlying unit was not independently verified. -| Comparator | Intended role in gate | Current status | Evidence summary | -| --- | --- | --- | --- | -| `raw Claude Code` | Baseline for payload cost and at least one usefulness comparison | comparator artifact: `ok`; gate: `pending_evidence` | The Haiku-backed Claude CLI runner now returns current payloads, but the checked-in baseline still has `averageFirstRelevantHit: null`, so the gate still records missing baseline metrics. | -| `GrepAI` | Named MCP comparator | `setup_failed` | Requires the GrepAI binary plus a local Ollama embedding setup that is not present in this proof environment. | -| `jCodeMunch` | Named MCP comparator | `setup_failed` | The MCP server still closes on startup during the current rerun, so no comparable discovery metrics were produced. | -| `codebase-memory-mcp` | Named MCP comparator | comparator artifact: `ok`; gate: `failed` | The repaired graph-backed runner now produces real current metrics, but the frozen gate still fails this lane because `codebase-context` does not stay within tolerance on every required usefulness metric. | -| `CodeGraphContext` | Graph-native comparator in the relaunch frame | `setup_failed` | The MCP server still closes on startup during the current rerun, so this lane remains missing evidence. | +| Tool | Completed | Failed | Expected-file coverage | File precision | Reported `peakPrivateGb` | +| --- | ---: | ---: | ---: | ---: | ---: | +| Raw search | 20/20 | 0 | 4.3% | 4.0% | 0.06 | +| Codebase Context | 20/20 | 0 | 25.7% | 11.5% | 2.90 | +| jCodeMunch | 20/20 | 0 | 27.1% | 11.0% | 0.33 | +| Repowise | 19/20 | 1 | 20.1% (failed run = 0) | 12.0% | 4.82 | +| context-mode | 20/20 | 0 | 16.8% | 11.7% | 0.15 | -## Reading This Table +## Reading notes -- `setup_failed` means the lane was attempted and did not reach a credible metric-producing state. -- `pending_evidence` in the gate means the lane is still missing one or more required metrics. -- `failed` in the gate means the lane has real metrics, but the frozen comparison rule still does not pass. -- A missing metric is not treated as a win for `codebase-context`. -- The combined gate in `results/gate-evaluation.json` remains `pending_evidence`, and `claimAllowed` stays `false`, until these lanes produce real metrics. +- Expected-file coverage is the percentage of expected gold files recovered, averaged within the corrected retrieval-only family. +- File precision is the percentage of returned files that matched the expected gold files. +- The table covers 100 attempts: 99 completed and 1 failed. Repowise completed 19 of 20 attempts; the other tools completed 20 of 20. +- Repowise's 20.1% is failure-inclusive: its failed attempt is counted as zero. +- Raw search is a deterministic lexical adapter in this protocol, not a full normal coding-agent baseline. +- `peakPrivateGb` is a reported telemetry field, not a verified unit claim, install size, or steady-state idle memory. +- The full-agent pilot, paired token observations, repeated retrieval history, and metered replay are separate evidence families. Their values should not be added to this table; the replay remains excluded while tool-use validity is unresolved. +- The measurements do not establish patch correctness or end-to-end task completion. They also do not establish a universal token, time, or coding result. -## Current codebase-context result +The [sanitized evidence extract](../results/benchmark-presentation-evidence.json) records the exact values and source hashes. -For reference, the current combined discovery output across `angular-spotify` and `excalidraw` is: - -| Metric | codebase-context | -| --- | ---: | -| `totalTasks` | 24 | -| `averageUsefulness` | 0.75 | -| `averagePayloadBytes` | 7306.4583 | -| `averageEstimatedTokens` | 1827.0833 | -| `bestExampleUsefulnessRate` | 0.125 | -| `gate.status` | `pending_evidence` | - -Those numbers are not compared here as head-to-head wins because the comparator lanes above did not produce matching metrics. +Historical note: the [earlier public report archive](./benchmark-prior-public-report.md) contains a one-task official pilot and a separate 24-task discovery report; it remains separate and keeps its existing claim gates. diff --git a/docs/demo.md b/docs/demo.md index 825b2ce..a122dc1 100644 --- a/docs/demo.md +++ b/docs/demo.md @@ -1,13 +1,23 @@ # Demo Script -This walkthrough uses real CLI output captured against `repos/angular-spotify` during the Phase 10 proof rerun. -Run it from the repo root with `CODEBASE_ROOT` pointed at the frozen sample repo. The public flow is simple: start with the conventions map, then search for the local example you need. +This walkthrough shows real CLI output captured from the open-source `angular-spotify` repository during a local proof run. That sample checkout is not bundled here. + +To try the same CLI flow, use Node.js 22 or newer and open a terminal in the repository you want to inspect. These commands use published `codebase-context@2.2.0`; the CLI uses the current directory as the project root. MCP registration and project selection are covered in the [client setup guide](./client-setup.md). + +The saved output below is from an earlier CLI run. It does not verify the current package or an MCP connection. + +## 0. Build The Local Index + +```bash +npx -y codebase-context@2.2.0 reindex +``` + +Run this before the first map or search in a repository. Later runs can use `reindex --incremental` after the code changes. ## 1. Start With The Conventions Map ```bash -$env:CODEBASE_ROOT='C:\Users\bitaz\Repos\codebase-context\repos\angular-spotify' -node dist/index.js map --json +npx -y codebase-context@2.2.0 map --json ``` Captured output excerpt: @@ -40,8 +50,7 @@ What this shows: ## 2. Search With Edit Intent ```bash -$env:CODEBASE_ROOT='C:\Users\bitaz\Repos\codebase-context\repos\angular-spotify' -node dist/index.js search --query "auth headers" --intent edit --limit 3 --json +npx -y codebase-context@2.2.0 search --query "auth headers" --intent edit --limit 3 --json ``` Captured output excerpt: @@ -66,7 +75,7 @@ Captured output excerpt: }, "results": [ { - "file": "C:\\Users\\bitaz\\Repos\\codebase-context\\repos\\angular-spotify\\libs\\web\\auth\\util\\src\\lib\\interceptors\\auth.interceptor.ts:10-42", + "file": "repos/angular-spotify/libs/web/auth/util/src/lib/interceptors/auth.interceptor.ts:10-42", "type": "interceptor:core" } ] @@ -82,8 +91,7 @@ What this shows: ## 3. Check A Team Pattern Directly ```bash -$env:CODEBASE_ROOT='C:\Users\bitaz\Repos\codebase-context\repos\angular-spotify' -node dist/index.js patterns --category state --json +npx -y codebase-context@2.2.0 patterns --category state --json ``` Captured output excerpt: @@ -117,4 +125,5 @@ What this shows: ## Caveats - These excerpts were captured from the current local proof run and will change if the frozen sample repo or index state changes. -- The discovery benchmark gate is still `pending_evidence`, and `claimAllowed` remains `false`, so this walkthrough demonstrates shipped behavior, not a released performance claim. +- File paths in the excerpts belong to that capture; your output will use the repository you run the command from. +- The benchmark documents observed local evidence and its limitations; this walkthrough demonstrates shipped behavior, not a universal performance or coding result. diff --git a/results/benchmark-presentation-evidence.json b/results/benchmark-presentation-evidence.json new file mode 100644 index 0000000..7544ac2 --- /dev/null +++ b/results/benchmark-presentation-evidence.json @@ -0,0 +1,1251 @@ +{ + "schemaVersion": 1, + "artifactType": "benchmark-presentation-source-extract", + "description": "Allowlisted fields extracted from retained reports. This is not a new benchmark run or independent validation of those reports. Full source byte hashes identify the inputs; private filesystem paths and orchestration costs are omitted.", + "provenance": { + "corrected": { + "file": "contextbench-engineering-note-summary.json", + "sha256": "ad8cac1ad86747463212b1665c3458e8f9560fbcde728d3dc2d74d1d6fe3e17f", + "generatedAt": "2026-06-04T08:28:24Z", + "schemaVersion": 2, + "gitRef": "18e0a43a90fe67e9df0d728b2134e9d430469883", + "gitBlobSha256": "a26599e1302ebdbb34e259530c8d775a9325bd42714e5b03398c545ad1abaf6c", + "checkoutByteDifference": "CRLF versus LF only" + }, + "history": { + "file": "contextbench-engineering-note-summary.json", + "sha256": "29c1a03bff051d9bc90284c5585e4f2cc1e94f9cc9da5d01d1b36e73dac4e87e", + "generatedAt": "2026-06-07T19:09:33Z", + "schemaVersion": 4, + "gitRef": "67b7ae2d1541eef3f591a9a58daf5d09ead8dcf1", + "gitBlobSha256": "29c1a03bff051d9bc90284c5585e4f2cc1e94f9cc9da5d01d1b36e73dac4e87e", + "checkoutByteDifference": "CRLF versus LF only" + }, + "pilot": { + "file": "REPORT.claim-session.json", + "sha256": "1c3368185c01a5e52bb7ba55466f9e5a053a26afe896a60b16b80c0743a5bd69", + "generatedAt": "2026-06-12T02:23:14.817Z", + "schemaVersion": 1, + "sessionId": "claim-bearing-2task-5lane-repeat3-claimclean2e21-20260612T002612Z" + } + }, + "correctedRetrieval": { + "sourceFields": [ + "rows", + "qualitySnapshot", + "costSnapshot" + ], + "publicationReadiness": { + "claimBearingPublicationAllowed": false, + "claimScope": "20x1_retrieval_overlap_only", + "unresolvedBlockers": [ + "single_repeat_no_stability_claim", + "retrieval_overlap_only_no_patch_quality", + "provider_dollar_cost_not_measured_because_no_paid_provider_use", + "repowise_one_no_context_evaluator_failure", + "cold_vs_warm_install_dependency_cache_not_fully_normalized" + ], + "publicUseLevel": "narrow_retrieval_overlap_case_study", + "status": "engineering_note_ready_with_claim_blocks", + "blockedClaimTypes": [ + "benchmark winner", + "best tool", + "paper-grade benchmark", + "resource-efficiency claim", + "generalized agent coding quality claim", + "agent patch correctness claim", + "developer productivity claim" + ] + }, + "rows": { + "evaluatorFailedRows": 1, + "repeatsPerTaskLane": 1, + "lanes": 5, + "evaluatorCompletedRows": 99, + "retrievalOverlapRows": 100, + "appendixOnlyLanes": [ + { + "reason": "bounded local indexing readiness failed earlier; not part of this corrected 20x1 headline table", + "laneId": "GrepAI" + } + ], + "sameDenominatorRowsPresent": true, + "tasks": 20 + }, + "qualitySnapshot": [ + { + "laneId": "codebase-context", + "rows": 20, + "evaluatorCompletedRows": 20, + "status": "completed:20", + "fileCoverageMean": 0.25722222222222224, + "filePrecisionMean": 0.11499999999999999, + "symbolCoverageMean": 0.1396239637555427, + "spanCoverageMean": 0.08573378715275093, + "lineCoverageMean": 0.0878787249873908 + }, + { + "laneId": "context-mode", + "rows": 20, + "evaluatorCompletedRows": 20, + "status": "completed:20", + "fileCoverageMean": 0.16833333333333333, + "filePrecisionMean": 0.11666666666666665, + "symbolCoverageMean": 0.02731443994601889, + "spanCoverageMean": 0.011708371581594209, + "lineCoverageMean": 0.01103832178724339 + }, + { + "laneId": "jcodemunch-full-mcp", + "rows": 20, + "evaluatorCompletedRows": 20, + "status": "completed:20", + "fileCoverageMean": 0.2713888888888889, + "filePrecisionMean": 0.11000000000000001, + "symbolCoverageMean": 0.19046586746586747, + "spanCoverageMean": 0.08523281976808295, + "lineCoverageMean": 0.08706990517203586 + }, + { + "laneId": "raw-native", + "rows": 20, + "evaluatorCompletedRows": 20, + "status": "completed:20", + "fileCoverageMean": 0.043333333333333335, + "filePrecisionMean": 0.04, + "symbolCoverageMean": 0.007186234817813765, + "spanCoverageMean": 0.00461426238619191, + "lineCoverageMean": 0.003664930124865624 + }, + { + "laneId": "repowise-selfhosted-mcp", + "rows": 20, + "evaluatorCompletedRows": 19, + "status": "completed:19,judge_failed:1", + "fileCoverageMean": 0.2013888888888889, + "filePrecisionMean": 0.12, + "symbolCoverageMean": 0.046063588114671705, + "spanCoverageMean": 0.03033601885780799, + "lineCoverageMean": 0.024685983349898594 + } + ], + "costSnapshot": [ + { + "laneId": "codebase-context", + "setupStatuses": "not_required:20", + "indexStatuses": "completed:20", + "indexExitStatuses": "unknown:20", + "indexMedianSeconds": 750.8, + "indexSumMinutes": 226.6, + "rowWallSumMinutes": 231.5, + "peakPrivateGb": 2.9, + "modelEvidence": "Xenova/bge-small-en-v1.5:20" + }, + { + "laneId": "context-mode", + "setupStatuses": "completed:20", + "indexStatuses": "completed:20", + "indexExitStatuses": "unknown:20", + "indexMedianSeconds": 18.6, + "indexSumMinutes": 12.5, + "rowWallSumMinutes": 16, + "peakPrivateGb": 0.15, + "modelEvidence": "unknown:20" + }, + { + "laneId": "jcodemunch-full-mcp", + "setupStatuses": "completed:20", + "indexStatuses": "completed:20", + "indexExitStatuses": "unknown:20", + "indexMedianSeconds": 11.3, + "indexSumMinutes": 4.9, + "rowWallSumMinutes": 10.9, + "peakPrivateGb": 0.33, + "modelEvidence": "unknown:20" + }, + { + "laneId": "raw-native", + "setupStatuses": "not_required:20", + "indexStatuses": "not_required:20", + "indexExitStatuses": "unknown:20", + "indexMedianSeconds": 0, + "indexSumMinutes": 0, + "rowWallSumMinutes": 6.8, + "peakPrivateGb": 0.06, + "modelEvidence": "unknown:20" + }, + { + "laneId": "repowise-selfhosted-mcp", + "setupStatuses": "completed:20", + "indexStatuses": "completed:20", + "indexExitStatuses": "1:20", + "indexMedianSeconds": 294.3, + "indexSumMinutes": 154.2, + "rowWallSumMinutes": 200.7, + "peakPrivateGb": 4.82, + "modelEvidence": "unknown:20" + } + ], + "failedRows": [ + { + "taskId": "Multi-SWE-Bench__typescript__maintenance__bugfix__8d2ac221", + "laneId": "repowise-selfhosted-mcp", + "status": "judge_failed", + "reason": "official_result_error:no_context_extracted" + } + ] + }, + "repeatedRetrievalAndReplayHistory": { + "sourceFields": [ + "retrievalChecks.sourceRuns", + "publicationReadiness" + ], + "sourceRuns": [ + { + "name": "corrected-20x1", + "checks": 100, + "checksScored": 99, + "checksWithNoUsableContext": 1 + }, + { + "name": "repeat-20x3", + "checks": 300, + "checksScored": 288, + "checksWithNoUsableContext": 12 + }, + { + "name": "metered-context-consumption", + "checks": 100, + "checksScored": 100, + "checksWithNoUsableContext": 0, + "tokenCountersCaptured": 100 + } + ], + "publicationReadiness": { + "claimBearingPublicationAllowed": false, + "claimScope": "500_check_practical_code_context_case_study", + "unresolvedBlockers": [ + "combined_local_evidence_sets_not_preregistered_repeated_protocol", + "retrieval_overlap_and_context_consumption_only_no_patch_quality", + "token_metering_is_context_consumption_not_patch_workflow", + "provider_dollar_cost_not_calculated_from_model_pricing", + "repowise_thirteen_checks_with_no_usable_context", + "cold_vs_warm_install_dependency_cache_not_fully_normalized" + ], + "publicUseLevel": "narrow_code_context_case_study", + "status": "engineering_note_ready_with_claim_blocks", + "blockedClaimTypes": [ + "benchmark winner", + "best tool", + "research-paper benchmark", + "resource-efficiency claim", + "generalized agent coding quality claim", + "agent patch correctness claim", + "developer productivity claim", + "token-savings claim" + ] + }, + "replayCaution": "The historical schema4 summary records completed metering; the later presentation describes no-tool-use replay exclusion. This extract does not reconcile the validity of that replay. No replay token results are presented." + }, + "twoTaskAgentPilot": { + "publicationReadiness": { + "construction": "Publication readiness is a claim gate derived from existing report evidence. It is not owner approval and not a new benchmark score.", + "status": "not_publication_ready", + "publicUseLevel": "engineering_note_or_internal_audit_only", + "claimBearingPublicationAllowed": false, + "claimScope": "pilot_only", + "qualityRowCount": 30, + "qualityTaskCount": 2, + "scoreableQualityLaneCount": 5, + "missingQualityLaneCount": 0, + "missingQualityLaneIds": [], + "reliabilityRowCount": 0, + "repeatConfidenceStatus": "repeat_ready", + "resourceTelemetryGapRows": [], + "unresolvedBlockers": [ + "pilot_task_count" + ], + "blockerActions": [ + { + "blocker": "pilot_task_count", + "action": "Add scoreable quality rows across the required task count before broad benchmark claims." + } + ], + "allowedNonClaimUses": [ + "Describe scoreable quality rows only within their narrow denominator." + ], + "blockedClaimTypes": [ + "benchmark winner", + "paper-grade benchmark", + "four-lane quality comparison", + "resource-efficiency claim", + "generalized agent coding quality claim" + ] + }, + "summary": { + "totalRows": 30, + "taskCount": 2, + "qualityTaskCount": 2, + "laneCount": 5, + "qualityRowCount": 30, + "reliabilityRowCount": 0, + "statusCounts": { + "completed": 30 + }, + "taskIds": [ + "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27" + ], + "qualityTaskIds": [ + "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27" + ], + "laneIds": [ + "codebase-context", + "context-mode", + "jcodemunch-full-mcp", + "raw-native", + "repowise-selfhosted-mcp" + ] + }, + "sourceFields": [ + "qualityRows", + "costRows" + ], + "qualityRows": [ + { + "laneId": "jcodemunch-full-mcp", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 1, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.5, + "filePrecision": 0.5, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0017401657736868617, + "spanPrecision": 0.475, + "lineCoverage": 0.002398081534772182, + "linePrecision": 0.5, + "editlocRecall": 1, + "editlocPrecision": 1 + } + }, + { + "laneId": "jcodemunch-full-mcp", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 2, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.5, + "filePrecision": 0.4, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0017401657736868617, + "spanPrecision": 0.3619047619047619, + "lineCoverage": 0.002398081534772182, + "linePrecision": 0.4, + "editlocRecall": 1, + "editlocPrecision": 1 + } + }, + { + "laneId": "jcodemunch-full-mcp", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 3, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.5, + "filePrecision": 0.5, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0017401657736868617, + "spanPrecision": 0.475, + "lineCoverage": 0.002398081534772182, + "linePrecision": 0.5, + "editlocRecall": 1, + "editlocPrecision": 1 + } + }, + { + "laneId": "jcodemunch-full-mcp", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 1, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.6666666666666666, + "filePrecision": 0.4, + "symbolCoverage": 0.02857142857142857, + "symbolPrecision": 0.16666666666666666, + "spanCoverage": 0.00467177045029839, + "spanPrecision": 0.5272108843537415, + "lineCoverage": 0.002026342451874367, + "linePrecision": 0.4, + "editlocRecall": 0.5714285714285714, + "editlocPrecision": 0.5714285714285714 + } + }, + { + "laneId": "jcodemunch-full-mcp", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 2, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.6666666666666666, + "filePrecision": 0.4, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0021399722707818435, + "spanPrecision": 0.2591240875912409, + "lineCoverage": 0.0010131712259371835, + "linePrecision": 0.2, + "editlocRecall": 0.5714285714285714, + "editlocPrecision": 0.5714285714285714 + } + }, + { + "laneId": "jcodemunch-full-mcp", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 3, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.6666666666666666, + "filePrecision": 0.4, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0021399722707818435, + "spanPrecision": 0.2591240875912409, + "lineCoverage": 0.0010131712259371835, + "linePrecision": 0.2, + "editlocRecall": 0.5714285714285714, + "editlocPrecision": 0.5714285714285714 + } + }, + { + "laneId": "repowise-selfhosted-mcp", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 1, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.5, + "filePrecision": 0.3333333333333333, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0017401657736868617, + "spanPrecision": 0.2923076923076923, + "lineCoverage": 0.002398081534772182, + "linePrecision": 0.3333333333333333, + "editlocRecall": 1, + "editlocPrecision": 1 + } + }, + { + "laneId": "repowise-selfhosted-mcp", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 2, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.5, + "filePrecision": 0.5, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0017401657736868617, + "spanPrecision": 0.475, + "lineCoverage": 0.002398081534772182, + "linePrecision": 0.5, + "editlocRecall": 1, + "editlocPrecision": 1 + } + }, + { + "laneId": "repowise-selfhosted-mcp", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 3, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.5, + "filePrecision": 0.4, + "symbolCoverage": 0.10526315789473684, + "symbolPrecision": 0.4, + "spanCoverage": 0.006640106241699867, + "spanPrecision": 0.5272727272727272, + "lineCoverage": 0.002398081534772182, + "linePrecision": 0.4, + "editlocRecall": 1, + "editlocPrecision": 1 + } + }, + { + "laneId": "repowise-selfhosted-mcp", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 1, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.6666666666666666, + "filePrecision": 0.5, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0021399722707818435, + "spanPrecision": 0.3287037037037037, + "lineCoverage": 0.0010131712259371835, + "linePrecision": 0.25, + "editlocRecall": 0.5714285714285714, + "editlocPrecision": 0.5714285714285714 + } + }, + { + "laneId": "repowise-selfhosted-mcp", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 2, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.6666666666666666, + "filePrecision": 0.5, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0021399722707818435, + "spanPrecision": 0.3287037037037037, + "lineCoverage": 0.0010131712259371835, + "linePrecision": 0.25, + "editlocRecall": 0.5714285714285714, + "editlocPrecision": 0.5714285714285714 + } + }, + { + "laneId": "repowise-selfhosted-mcp", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 3, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.6666666666666666, + "filePrecision": 0.5, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0021399722707818435, + "spanPrecision": 0.3287037037037037, + "lineCoverage": 0.0010131712259371835, + "linePrecision": 0.25, + "editlocRecall": 0.5714285714285714, + "editlocPrecision": 0.5714285714285714 + } + }, + { + "laneId": "codebase-context", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 1, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.5, + "filePrecision": 1, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0017401657736868617, + "spanPrecision": 1, + "lineCoverage": 0.002398081534772182, + "linePrecision": 1, + "editlocRecall": 1, + "editlocPrecision": 1 + } + }, + { + "laneId": "codebase-context", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 2, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.5, + "filePrecision": 0.6666666666666666, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0017401657736868617, + "spanPrecision": 0.6909090909090909, + "lineCoverage": 0.002398081534772182, + "linePrecision": 0.6666666666666666, + "editlocRecall": 1, + "editlocPrecision": 1 + } + }, + { + "laneId": "codebase-context", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 3, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.5, + "filePrecision": 0.4, + "symbolCoverage": 0.05263157894736842, + "symbolPrecision": 0.25, + "spanCoverage": 0.006731693913999175, + "spanPrecision": 0.5856573705179283, + "lineCoverage": 0.002398081534772182, + "linePrecision": 0.4, + "editlocRecall": 1, + "editlocPrecision": 1 + } + }, + { + "laneId": "codebase-context", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 1, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.6666666666666666, + "filePrecision": 0.4, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0021399722707818435, + "spanPrecision": 0.2591240875912409, + "lineCoverage": 0.0010131712259371835, + "linePrecision": 0.2, + "editlocRecall": 0.5714285714285714, + "editlocPrecision": 0.5714285714285714 + } + }, + { + "laneId": "codebase-context", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 2, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.6666666666666666, + "filePrecision": 0.4, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0021399722707818435, + "spanPrecision": 0.2591240875912409, + "lineCoverage": 0.0010131712259371835, + "linePrecision": 0.2, + "editlocRecall": 0.5714285714285714, + "editlocPrecision": 0.5714285714285714 + } + }, + { + "laneId": "codebase-context", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 3, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.6666666666666666, + "filePrecision": 0.4, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0021399722707818435, + "spanPrecision": 0.2591240875912409, + "lineCoverage": 0.0010131712259371835, + "linePrecision": 0.2, + "editlocRecall": 0.5714285714285714, + "editlocPrecision": 0.5714285714285714 + } + }, + { + "laneId": "raw-native", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 1, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.5, + "filePrecision": 0.5, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0017401657736868617, + "spanPrecision": 0.475, + "lineCoverage": 0.002398081534772182, + "linePrecision": 0.5, + "editlocRecall": 1, + "editlocPrecision": 1 + } + }, + { + "laneId": "raw-native", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 2, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.5, + "filePrecision": 0.6666666666666666, + "symbolCoverage": 0.10526315789473684, + "symbolPrecision": 0.6666666666666666, + "spanCoverage": 0.004258826761917846, + "spanPrecision": 0.7560975609756098, + "lineCoverage": 0.002398081534772182, + "linePrecision": 0.6666666666666666, + "editlocRecall": 1, + "editlocPrecision": 1 + } + }, + { + "laneId": "raw-native", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 3, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.5, + "filePrecision": 0.6666666666666666, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0017401657736868617, + "spanPrecision": 0.7169811320754716, + "lineCoverage": 0.002398081534772182, + "linePrecision": 0.6666666666666666, + "editlocRecall": 1, + "editlocPrecision": 1 + } + }, + { + "laneId": "raw-native", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 1, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.6666666666666666, + "filePrecision": 0.3333333333333333, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0021399722707818435, + "spanPrecision": 0.21385542168674698, + "lineCoverage": 0.0010131712259371835, + "linePrecision": 0.16666666666666666, + "editlocRecall": 0.5714285714285714, + "editlocPrecision": 0.5714285714285714 + } + }, + { + "laneId": "raw-native", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 2, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.6666666666666666, + "filePrecision": 0.5, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0021399722707818435, + "spanPrecision": 0.3287037037037037, + "lineCoverage": 0.0010131712259371835, + "linePrecision": 0.25, + "editlocRecall": 0.5714285714285714, + "editlocPrecision": 0.5714285714285714 + } + }, + { + "laneId": "raw-native", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 3, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.6666666666666666, + "filePrecision": 0.4, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0021399722707818435, + "spanPrecision": 0.2591240875912409, + "lineCoverage": 0.0010131712259371835, + "linePrecision": 0.2, + "editlocRecall": 0.5714285714285714, + "editlocPrecision": 0.5714285714285714 + } + }, + { + "laneId": "context-mode", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 1, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.5, + "filePrecision": 0.6666666666666666, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0017401657736868617, + "spanPrecision": 0.6909090909090909, + "lineCoverage": 0.002398081534772182, + "linePrecision": 0.6666666666666666, + "editlocRecall": 1, + "editlocPrecision": 1 + } + }, + { + "laneId": "context-mode", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 2, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.5, + "filePrecision": 0.5, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0017401657736868617, + "spanPrecision": 0.475, + "lineCoverage": 0.002398081534772182, + "linePrecision": 0.5, + "editlocRecall": 1, + "editlocPrecision": 1 + } + }, + { + "laneId": "context-mode", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 3, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.25, + "filePrecision": 0.3333333333333333, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0009158767229930851, + "spanPrecision": 0.31746031746031744, + "lineCoverage": 0.001199040767386091, + "linePrecision": 0.3333333333333333, + "editlocRecall": 1, + "editlocPrecision": 1 + } + }, + { + "laneId": "context-mode", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 1, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.6666666666666666, + "filePrecision": 0.6666666666666666, + "symbolCoverage": 0.02857142857142857, + "symbolPrecision": 0.2, + "spanCoverage": 0.0028633431792151427, + "spanPrecision": 0.6643356643356644, + "lineCoverage": 0.002026342451874367, + "linePrecision": 0.6666666666666666, + "editlocRecall": 0.5714285714285714, + "editlocPrecision": 0.5714285714285714 + } + }, + { + "laneId": "context-mode", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 2, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.6666666666666666, + "filePrecision": 0.4, + "symbolCoverage": 0.02857142857142857, + "symbolPrecision": 0.14285714285714285, + "spanCoverage": 0.0028633431792151427, + "spanPrecision": 0.521978021978022, + "lineCoverage": 0.002026342451874367, + "linePrecision": 0.4, + "editlocRecall": 0.5714285714285714, + "editlocPrecision": 0.5714285714285714 + } + }, + { + "laneId": "context-mode", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 3, + "rowStatus": "completed", + "metrics": { + "fileCoverage": 0.6666666666666666, + "filePrecision": 0.4, + "symbolCoverage": 0, + "symbolPrecision": 1, + "spanCoverage": 0.0021399722707818435, + "spanPrecision": 0.2591240875912409, + "lineCoverage": 0.0010131712259371835, + "linePrecision": 0.2, + "editlocRecall": 0.5714285714285714, + "editlocPrecision": 0.5714285714285714 + } + } + ], + "costRows": [ + { + "laneId": "jcodemunch-full-mcp", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 1, + "status": "completed", + "inputTokens": 651991, + "cachedInputTokens": 562688, + "outputTokens": 8569, + "reasoningOutputTokens": 5477, + "taskWallTimeMs": 158527, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "jcodemunch-full-mcp", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 2, + "status": "completed", + "inputTokens": 682131, + "cachedInputTokens": 642048, + "outputTokens": 10057, + "reasoningOutputTokens": 5312, + "taskWallTimeMs": 224679, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "jcodemunch-full-mcp", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 3, + "status": "completed", + "inputTokens": 400544, + "cachedInputTokens": 317952, + "outputTokens": 6954, + "reasoningOutputTokens": 4131, + "taskWallTimeMs": 131028, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "jcodemunch-full-mcp", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 1, + "status": "completed", + "inputTokens": 347800, + "cachedInputTokens": 314368, + "outputTokens": 13393, + "reasoningOutputTokens": 9227, + "taskWallTimeMs": 213283, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "jcodemunch-full-mcp", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 2, + "status": "completed", + "inputTokens": 273980, + "cachedInputTokens": 244224, + "outputTokens": 7323, + "reasoningOutputTokens": 4019, + "taskWallTimeMs": 140876, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "jcodemunch-full-mcp", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 3, + "status": "completed", + "inputTokens": 241423, + "cachedInputTokens": 206848, + "outputTokens": 5932, + "reasoningOutputTokens": 3550, + "taskWallTimeMs": 119511, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "repowise-selfhosted-mcp", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 1, + "status": "completed", + "inputTokens": 428418, + "cachedInputTokens": 398336, + "outputTokens": 7056, + "reasoningOutputTokens": 2584, + "taskWallTimeMs": 150948, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "repowise-selfhosted-mcp", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 2, + "status": "completed", + "inputTokens": 230103, + "cachedInputTokens": 200192, + "outputTokens": 5962, + "reasoningOutputTokens": 3079, + "taskWallTimeMs": 153087, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "repowise-selfhosted-mcp", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 3, + "status": "completed", + "inputTokens": 725790, + "cachedInputTokens": 665088, + "outputTokens": 9735, + "reasoningOutputTokens": 6607, + "taskWallTimeMs": 197729, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "repowise-selfhosted-mcp", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 1, + "status": "completed", + "inputTokens": 324871, + "cachedInputTokens": 296448, + "outputTokens": 9513, + "reasoningOutputTokens": 5744, + "taskWallTimeMs": 170065, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "repowise-selfhosted-mcp", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 2, + "status": "completed", + "inputTokens": 224215, + "cachedInputTokens": 177664, + "outputTokens": 4517, + "reasoningOutputTokens": 1669, + "taskWallTimeMs": 115595, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "repowise-selfhosted-mcp", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 3, + "status": "completed", + "inputTokens": 263549, + "cachedInputTokens": 243200, + "outputTokens": 10605, + "reasoningOutputTokens": 7467, + "taskWallTimeMs": 170770, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "codebase-context", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 1, + "status": "completed", + "inputTokens": 223558, + "cachedInputTokens": 190976, + "outputTokens": 4990, + "reasoningOutputTokens": 2899, + "taskWallTimeMs": 110708, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "codebase-context", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 2, + "status": "completed", + "inputTokens": 269851, + "cachedInputTokens": 234496, + "outputTokens": 6028, + "reasoningOutputTokens": 3024, + "taskWallTimeMs": 121854, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "codebase-context", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 3, + "status": "completed", + "inputTokens": 205432, + "cachedInputTokens": 130048, + "outputTokens": 5564, + "reasoningOutputTokens": 3530, + "taskWallTimeMs": 114664, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "codebase-context", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 1, + "status": "completed", + "inputTokens": 215407, + "cachedInputTokens": 167424, + "outputTokens": 7930, + "reasoningOutputTokens": 4784, + "taskWallTimeMs": 143643, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "codebase-context", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 2, + "status": "completed", + "inputTokens": 247868, + "cachedInputTokens": 206848, + "outputTokens": 5463, + "reasoningOutputTokens": 3246, + "taskWallTimeMs": 106404, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "codebase-context", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 3, + "status": "completed", + "inputTokens": 534029, + "cachedInputTokens": 498688, + "outputTokens": 13790, + "reasoningOutputTokens": 10169, + "taskWallTimeMs": 238528, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "raw-native", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 1, + "status": "completed", + "inputTokens": 541233, + "cachedInputTokens": 497152, + "outputTokens": 9011, + "reasoningOutputTokens": 4494, + "taskWallTimeMs": 185061, + "contextToolCallCount": null, + "contextToolCompletedCallCount": null + }, + { + "laneId": "raw-native", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 2, + "status": "completed", + "inputTokens": 856202, + "cachedInputTokens": 798720, + "outputTokens": 12581, + "reasoningOutputTokens": 7297, + "taskWallTimeMs": 257215, + "contextToolCallCount": null, + "contextToolCompletedCallCount": null + }, + { + "laneId": "raw-native", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 3, + "status": "completed", + "inputTokens": 653348, + "cachedInputTokens": 610304, + "outputTokens": 6730, + "reasoningOutputTokens": 3640, + "taskWallTimeMs": 179577, + "contextToolCallCount": null, + "contextToolCompletedCallCount": null + }, + { + "laneId": "raw-native", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 1, + "status": "completed", + "inputTokens": 826946, + "cachedInputTokens": 740864, + "outputTokens": 17144, + "reasoningOutputTokens": 12215, + "taskWallTimeMs": 294443, + "contextToolCallCount": null, + "contextToolCompletedCallCount": null + }, + { + "laneId": "raw-native", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 2, + "status": "completed", + "inputTokens": 251601, + "cachedInputTokens": 220672, + "outputTokens": 7386, + "reasoningOutputTokens": 3991, + "taskWallTimeMs": 141981, + "contextToolCallCount": null, + "contextToolCompletedCallCount": null + }, + { + "laneId": "raw-native", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 3, + "status": "completed", + "inputTokens": 290186, + "cachedInputTokens": 272384, + "outputTokens": 7567, + "reasoningOutputTokens": 4024, + "taskWallTimeMs": 142002, + "contextToolCallCount": null, + "contextToolCompletedCallCount": null + }, + { + "laneId": "context-mode", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 1, + "status": "completed", + "inputTokens": 304532, + "cachedInputTokens": 271872, + "outputTokens": 7060, + "reasoningOutputTokens": 4975, + "taskWallTimeMs": 146469, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "context-mode", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 2, + "status": "completed", + "inputTokens": 610092, + "cachedInputTokens": 546304, + "outputTokens": 8915, + "reasoningOutputTokens": 5830, + "taskWallTimeMs": 236889, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "context-mode", + "taskId": "Multi-SWE-Bench__c__maintenance__bugfix__5e659108", + "repeatIndex": 3, + "status": "completed", + "inputTokens": 541331, + "cachedInputTokens": 480256, + "outputTokens": 10847, + "reasoningOutputTokens": 5649, + "taskWallTimeMs": 204901, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "context-mode", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 1, + "status": "completed", + "inputTokens": 182974, + "cachedInputTokens": 167424, + "outputTokens": 9561, + "reasoningOutputTokens": 7486, + "taskWallTimeMs": 146629, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "context-mode", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 2, + "status": "completed", + "inputTokens": 333030, + "cachedInputTokens": 292352, + "outputTokens": 12256, + "reasoningOutputTokens": 8613, + "taskWallTimeMs": 198105, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + }, + { + "laneId": "context-mode", + "taskId": "Multi-SWE-Bench__cpp__maintenance__bugfix__3e497d27", + "repeatIndex": 3, + "status": "completed", + "inputTokens": 518028, + "cachedInputTokens": 475136, + "outputTokens": 10446, + "reasoningOutputTokens": 6931, + "taskWallTimeMs": 200453, + "contextToolCallCount": 1, + "contextToolCompletedCallCount": 1 + } + ], + "aggregation": "Per-task, per-lane median of three costRows values for each counter independently. Cached input is a subset of input; component medians need not sum." + } +} diff --git a/results/contextbench-engineering-note-summary.json b/results/contextbench-engineering-note-summary.json new file mode 100644 index 0000000..03b0ebb --- /dev/null +++ b/results/contextbench-engineering-note-summary.json @@ -0,0 +1,213 @@ +{ + "schemaVersion": 2, + "artifactType": "contextbench-engineering-note-summary", + "generatedAt": "2026-06-04T08:28:24Z", + "source": { + "scorerScriptSha256": "sha256:13e859f6485735d76b1591a45a7211338f9fed9ff5566776ac8455db6e820e79", + "localSummary": "gate5-official-retrieval-scoring-current-script-20260604T1027/GATE5-SCORING-SUMMARY.json", + "localSummarySha256": "sha256:e5ecd3ee45b9577ae32d409de960edc1a7c2761159280cb7f8e7b7a85555387b", + "boundary": "The full run artifacts are local evidence. This checked-in summary is the public engineering-note artifact and does not make winner, leaderboard, paper-grade, resource-efficiency, or patch-quality claims.", + "localRunRoot": "benchmark-runs/contextbench/phase41/corrected-20x1-20260603T103632", + "localRunRootTracked": false + }, + "publicationReadiness": { + "claimBearingPublicationAllowed": false, + "claimScope": "20x1_retrieval_overlap_only", + "unresolvedBlockers": [ + "single_repeat_no_stability_claim", + "retrieval_overlap_only_no_patch_quality", + "provider_dollar_cost_not_measured_because_no_paid_provider_use", + "repowise_one_no_context_evaluator_failure", + "cold_vs_warm_install_dependency_cache_not_fully_normalized" + ], + "publicUseLevel": "narrow_retrieval_overlap_case_study", + "status": "engineering_note_ready_with_claim_blocks", + "blockedClaimTypes": [ + "benchmark winner", + "best tool", + "paper-grade benchmark", + "resource-efficiency claim", + "generalized agent coding quality claim", + "agent patch correctness claim", + "developer productivity claim" + ] + }, + "rows": { + "evaluatorFailedRows": 1, + "repeatsPerTaskLane": 1, + "lanes": 5, + "evaluatorCompletedRows": 99, + "retrievalOverlapRows": 100, + "appendixOnlyLanes": [ + { + "reason": "bounded local indexing readiness failed earlier; not part of this corrected 20x1 headline table", + "laneId": "GrepAI" + } + ], + "sameDenominatorRowsPresent": true, + "tasks": 20 + }, + "methodology": { + "notMeasured": "patch correctness, developer productivity, provider-dollar spend, multi-repeat confidence intervals", + "scoring": "official ContextBench evaluator on saved retrieval trajectories using per-task gold; scorer gold was not read during retrieval", + "lanes": [ + "raw-native", + "codebase-context", + "jcodemunch-full-mcp", + "repowise-selfhosted-mcp", + "context-mode" + ], + "measured": "retrieval overlap, setup/index time, row wall time, process memory telemetry, GPU availability probe, failure rows", + "metricView": "failure-inclusive means; judge_failed/no_context rows count as zero in the public summary", + "publicFrame": "we benchmarked these tools on open-source repos", + "taskSet": "20 frozen ContextBench-style open-source tasks" + }, + "qualitySnapshot": [ + { + "laneId": "codebase-context", + "rows": 20, + "evaluatorCompletedRows": 20, + "status": "completed:20", + "fileCoverageMean": 0.25722222222222224, + "filePrecisionMean": 0.11499999999999999, + "symbolCoverageMean": 0.1396239637555427, + "spanCoverageMean": 0.08573378715275093, + "lineCoverageMean": 0.0878787249873908 + }, + { + "laneId": "context-mode", + "rows": 20, + "evaluatorCompletedRows": 20, + "status": "completed:20", + "fileCoverageMean": 0.16833333333333333, + "filePrecisionMean": 0.11666666666666665, + "symbolCoverageMean": 0.02731443994601889, + "spanCoverageMean": 0.011708371581594209, + "lineCoverageMean": 0.01103832178724339 + }, + { + "laneId": "jcodemunch-full-mcp", + "rows": 20, + "evaluatorCompletedRows": 20, + "status": "completed:20", + "fileCoverageMean": 0.2713888888888889, + "filePrecisionMean": 0.11000000000000001, + "symbolCoverageMean": 0.19046586746586747, + "spanCoverageMean": 0.08523281976808295, + "lineCoverageMean": 0.08706990517203586 + }, + { + "laneId": "raw-native", + "rows": 20, + "evaluatorCompletedRows": 20, + "status": "completed:20", + "fileCoverageMean": 0.043333333333333335, + "filePrecisionMean": 0.04, + "symbolCoverageMean": 0.007186234817813765, + "spanCoverageMean": 0.00461426238619191, + "lineCoverageMean": 0.003664930124865624 + }, + { + "laneId": "repowise-selfhosted-mcp", + "rows": 20, + "evaluatorCompletedRows": 19, + "status": "completed:19,judge_failed:1", + "fileCoverageMean": 0.2013888888888889, + "filePrecisionMean": 0.12, + "symbolCoverageMean": 0.046063588114671705, + "spanCoverageMean": 0.03033601885780799, + "lineCoverageMean": 0.024685983349898594 + } + ], + "costSnapshot": [ + { + "laneId": "codebase-context", + "setupStatuses": "not_required:20", + "indexStatuses": "completed:20", + "indexExitStatuses": "unknown:20", + "indexMedianSeconds": 750.8, + "indexSumMinutes": 226.6, + "rowWallSumMinutes": 231.5, + "peakPrivateGb": 2.9, + "modelEvidence": "Xenova/bge-small-en-v1.5:20" + }, + { + "laneId": "context-mode", + "setupStatuses": "completed:20", + "indexStatuses": "completed:20", + "indexExitStatuses": "unknown:20", + "indexMedianSeconds": 18.6, + "indexSumMinutes": 12.5, + "rowWallSumMinutes": 16.0, + "peakPrivateGb": 0.15, + "modelEvidence": "unknown:20" + }, + { + "laneId": "jcodemunch-full-mcp", + "setupStatuses": "completed:20", + "indexStatuses": "completed:20", + "indexExitStatuses": "unknown:20", + "indexMedianSeconds": 11.3, + "indexSumMinutes": 4.9, + "rowWallSumMinutes": 10.9, + "peakPrivateGb": 0.33, + "modelEvidence": "unknown:20" + }, + { + "laneId": "raw-native", + "setupStatuses": "not_required:20", + "indexStatuses": "not_required:20", + "indexExitStatuses": "unknown:20", + "indexMedianSeconds": 0.0, + "indexSumMinutes": 0.0, + "rowWallSumMinutes": 6.8, + "peakPrivateGb": 0.06, + "modelEvidence": "unknown:20" + }, + { + "laneId": "repowise-selfhosted-mcp", + "setupStatuses": "completed:20", + "indexStatuses": "completed:20", + "indexExitStatuses": "1:20", + "indexMedianSeconds": 294.3, + "indexSumMinutes": 154.2, + "rowWallSumMinutes": 200.7, + "peakPrivateGb": 4.82, + "modelEvidence": "unknown:20" + } + ], + "failedRows": [ + { + "taskId": "Multi-SWE-Bench__typescript__maintenance__bugfix__8d2ac221", + "laneId": "repowise-selfhosted-mcp", + "status": "judge_failed", + "reason": "official_result_error:no_context_extracted", + "officialResults": "benchmark-runs/contextbench/phase41/corrected-20x1-20260603T103632/gate5-official-retrieval-scoring/rows/064-Multi-SWE-Bench__typescript__maintenance__bugfix__8d2ac221/repowise-selfhosted-mcp/official-results.jsonl", + "trajectory": "benchmark-runs/contextbench/phase41/corrected-20x1-20260603T103632/gate4-corrected-20x1/13-Multi-SWE-Bench-typescript-maintenance-bugfix-8d2ac221/repowise-selfhosted-mcp/runs/repowise-selfhosted-mcp-Multi-SWE-Bench__typescript__maintenance__bugfix__8d2ac221-1-retrieval/trajectory.json" + } + ], + "interpretation": [ + "jCodeMunch and codebase-context are close on file/span retrieval in this fixed adapter run; jCodeMunch has stronger symbol coverage but lower file precision.", + "codebase-context completed all 20 retrieval rows in this run but was expensive to index with local Transformers embeddings.", + "jCodeMunch is much cheaper to index in this adapter run, but its exact cold/warm install/cache boundary still needs care before cost claims.", + "context-mode is lightweight and low-cost, with lower recall ceiling in this run.", + "Repowise indexed and returned rows, but used the highest peak private memory, every index exited nonzero with a documented benign failure mode, and one row produced no extracted context.", + "raw-native is a deterministic lexical baseline here, not a free-form raw agent ceiling; it had low retrieval-overlap on these 20 tasks." + ], + "caveats": { + "publicDocs": "passed: README/docs/package-facing benchmark wording is aligned to the narrow 100-row retrieval-overlap engineering-note framing", + "overclaiming": "passed: public wording keeps winner, leaderboard, paper-grade, resource-efficiency, patch-quality, and productivity claims blocked", + "patchQuality": "blocked: retrieval-overlap only, no downstream patch/test success measured", + "costAccounting": "partial_pass: setup/index/resource telemetry captured; provider-dollar costs remain none/unknown because no paid provider use was approved", + "repeatStability": "blocked: 1 repeat per lane/task, no paper-grade stability or confidence claim", + "artifactCompleteness": "passed: 100 rows have summary, trajectory, raw trace, setup, telemetry, and gold", + "failureInclusion": "passed: Repowise judge_failed/no_context row is retained and zero-filled metric view exists", + "sameDenominator": "passed: 20 tasks x 5 lanes present" + }, + "metaCost": { + "note": "Benchmark orchestration cost is authoring/evaluation overhead, not a lane score.", + "latestKnownCodexGoalTokens": "owner-reported 2.29B input and 7.48M output tokens before this corrected scoring pass; exact all-run total not independently audited here", + "gate5ScoringPeakPrivateMb": 65.1, + "gate5ScoringRuntimeSeconds": 72.4 + } +} From f1035c2a78745fccd7b868e8749fc2343898f27e Mon Sep 17 00:00:00 2001 From: PatrickSys Date: Tue, 6 Oct 2026 10:54:33 +0200 Subject: [PATCH 2/2] fix(docs): preserve package links and evidence contracts --- README.md | 6 +- docs/capabilities.md | 8 +-- docs/demo.md | 2 +- results/benchmark-presentation-evidence.json | 2 +- tests/mcp-client-templates.test.ts | 61 ++++++++++++++------ tests/proof-truth-surfaces.test.ts | 56 ++++++++++++------ 6 files changed, 88 insertions(+), 47 deletions(-) diff --git a/README.md b/README.md index dbc2d3f..6249c02 100644 --- a/README.md +++ b/README.md @@ -133,9 +133,9 @@ Codebase Context runs locally through the Model Context Protocol (MCP), with ind Its ranked code search combines **Match words**, **Search by meaning**, and **Rank results** to give the agent ranked files, a best example, project patterns, relationships, and relevant memory before they edit. -The [benchmark](./docs/benchmark.md) reports a corrected 100-attempt retrieval family across five local code-context tools: 99 completed attempts and 1 failed attempt. Codebase Context recovered 25.7% of expected gold files with 11.5% file precision in that fixed adapter run; jCodeMunch recovered 27.1% with 11.0% file precision. Raw-native is a deterministic lexical adapter here, not a full normal coding-agent baseline. These are retrieval observations, not a winner or a coding-quality claim. The [sanitized evidence extract](./results/benchmark-presentation-evidence.json) records the exact values and source hashes. +The [benchmark](https://github.com/PatrickSys/codebase-context/blob/master/docs/benchmark.md) reports a corrected 100-attempt retrieval family across five local code-context tools: 99 completed attempts and 1 failed attempt. Codebase Context recovered 25.7% of expected gold files with 11.5% file precision in that fixed adapter run; jCodeMunch recovered 27.1% with 11.0% file precision. Raw-native is a deterministic lexical adapter here, not a full normal coding-agent baseline. These are retrieval observations, not a winner or a coding-quality claim. The [sanitized evidence extract](https://github.com/PatrickSys/codebase-context/blob/master/results/benchmark-presentation-evidence.json) records the exact values and source hashes. -The same report retains a separate 300-attempt repeatability history, a 30-run two-task full-agent pilot, and a metered replay whose tool-use validity remains unresolved. A historical Codebase Context indexing observation with embeddings enabled took about 12 minutes 31 seconds, but it does not establish warm-use speed, query latency, install cost, or a cross-tool index-speed ranking. The report does not combine these families into a pooled score and does not establish patch correctness or end-to-end coding quality. Earlier public reports are preserved separately in the [benchmark archive](./docs/benchmark-prior-public-report.md). +The same report retains a separate 300-attempt repeatability history, a 30-run two-task full-agent pilot, and a metered replay whose tool-use validity remains unresolved. A historical Codebase Context indexing observation with embeddings enabled took about 12 minutes 31 seconds, but it does not establish warm-use speed, query latency, install cost, or a cross-tool index-speed ranking. The report does not combine these families into a pooled score and does not establish patch correctness or end-to-end coding quality. Earlier public reports are preserved separately in the [benchmark archive](https://github.com/PatrickSys/codebase-context/blob/master/docs/benchmark-prior-public-report.md). The method and failures are documented so the measurements can be inspected with their limits. @@ -154,7 +154,7 @@ The method and failures are documented so the measurements can be inspected with - [Client setup](./docs/client-setup.md#client-setup) - commands, config, proof level, and client limits - [Capabilities](./docs/capabilities.md) - tools, response fields, routing, and configuration - [CLI](./docs/cli.md) - terminal commands and example output -- [Benchmark](./docs/benchmark.md) - method, measurements, and failures +- [Benchmark](https://github.com/PatrickSys/codebase-context/blob/master/docs/benchmark.md) - method, measurements, and failures - [Demo](https://github.com/PatrickSys/codebase-context/blob/master/docs/demo.md) - a complete repository walkthrough - [Motivation](https://github.com/PatrickSys/codebase-context/blob/master/MOTIVATION.md) - the design problem and research background - [Contributing](https://github.com/PatrickSys/codebase-context/blob/master/CONTRIBUTING.md) - local development and evaluation commands diff --git a/docs/capabilities.md b/docs/capabilities.md index d5d5c2f..748d5a1 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -104,12 +104,12 @@ Behavior matrix: Rules: -- If the client provides workspace context, that becomes the trusted workspace boundary for the session. In practice this usually comes from MCP roots. -- Treat seamless multi-project routing as evidence-backed only for roots-capable hosts. Without roots, explicit fallback is still required. -- If the server still cannot tell which project to use, a bootstrap path or explicit absolute `project` path remains the fallback. -- `project` is the canonical explicit selector when routing is ambiguous. +- The portable default is explicit selection: pass the repository's absolute path as `project`. A successful selection becomes the active project for that server process. +- Client-announced workspace roots remain a compatibility aid and constrain explicit selectors to that workspace. One valid root can auto-select; ambiguous roots require selection. Roots are not required for the documented setup. +- Without a configured folder, known root or active selection, the server returns `selection_required` instead of guessing a project. - `project` may point at a project path, file path, `file://` URI, or relative subproject path. - Later tool calls may omit `project`; the server falls back to the active project when one has already been established. +- Select again after a server restart or when changing repositories. Published-package A/B/A stdio routing was verified; it does not establish native agent continuation or concurrent HTTP isolation. See the [client setup guide](./client-setup.md#project-routing-contract) for the checks and their limits. - The server does not rely on `cwd` walk-up in MCP mode. - `codebase://context` serves the active project. Before selection in an unresolved multi-project session, it returns a workspace overview with candidate projects, readiness state, and project-scoped resource URIs. - `codebase://context/project/` serves a specific project directly and also makes that project active for later tool calls. diff --git a/docs/demo.md b/docs/demo.md index a122dc1..8b06f23 100644 --- a/docs/demo.md +++ b/docs/demo.md @@ -4,7 +4,7 @@ This walkthrough shows real CLI output captured from the open-source `angular-sp To try the same CLI flow, use Node.js 22 or newer and open a terminal in the repository you want to inspect. These commands use published `codebase-context@2.2.0`; the CLI uses the current directory as the project root. MCP registration and project selection are covered in the [client setup guide](./client-setup.md). -The saved output below is from an earlier CLI run. It does not verify the current package or an MCP connection. +The saved output below is from an earlier CLI run, not the published-package commands shown here. The machine-specific repository root is omitted from the returned file path. This capture does not verify the current package or an MCP connection. ## 0. Build The Local Index diff --git a/results/benchmark-presentation-evidence.json b/results/benchmark-presentation-evidence.json index 7544ac2..6f3e44f 100644 --- a/results/benchmark-presentation-evidence.json +++ b/results/benchmark-presentation-evidence.json @@ -10,7 +10,7 @@ "schemaVersion": 2, "gitRef": "18e0a43a90fe67e9df0d728b2134e9d430469883", "gitBlobSha256": "a26599e1302ebdbb34e259530c8d775a9325bd42714e5b03398c545ad1abaf6c", - "checkoutByteDifference": "CRLF versus LF only" + "checkoutByteDifference": "Source copy uses mixed CRLF/LF; normalizing line endings yields the Git blob byte-for-byte (LF)." }, "history": { "file": "contextbench-engineering-note-summary.json", diff --git a/tests/mcp-client-templates.test.ts b/tests/mcp-client-templates.test.ts index 03cf820..05baaa8 100644 --- a/tests/mcp-client-templates.test.ts +++ b/tests/mcp-client-templates.test.ts @@ -73,18 +73,21 @@ describe('templates/mcp/http/.mcp.json', () => { }); // --------------------------------------------------------------------------- -// README references templates and all four target clients +// README links the shipped guide for advanced templates and covers target clients // --------------------------------------------------------------------------- describe('README.md client setup documentation', () => { const readme = readText('README.md'); + const clientSetup = readText('docs/client-setup.md'); it('references the stdio template path', () => { - expect(readme).toContain('templates/mcp/stdio/.mcp.json'); + expect(readme).toContain('./docs/client-setup.md'); + expect(clientSetup).toContain('templates/mcp/stdio/.mcp.json'); }); it('references the HTTP template path', () => { - expect(readme).toContain('templates/mcp/http/.mcp.json'); + expect(readme).toContain('./docs/client-setup.md'); + expect(clientSetup).toContain('templates/mcp/http/.mcp.json'); }); it('mentions Claude Code', () => { @@ -103,8 +106,11 @@ describe('README.md client setup documentation', () => { expect(readme).toContain('Windsurf'); }); - it('includes the HTTP endpoint URL', () => { - expect(readme).toContain('127.0.0.1:3100/mcp'); + it('links the advanced HTTP endpoint without making it the default setup', () => { + expect(readme).toContain('./docs/client-setup.md'); + expect(readme).toContain('stdio'); + expect(clientSetup).toContain('127.0.0.1:3100/mcp'); + expect(clientSetup).toContain('not part of the default setup proof'); }); }); @@ -128,15 +134,21 @@ describe('docs/capabilities.md transport documentation', () => { }); it('covers all four target clients', () => { - expect(caps).toContain('Claude Code'); - expect(caps).toContain('Cursor'); - expect(caps).toContain('Codex'); - expect(caps).toContain('Windsurf'); - }); - - it('states the roots-first routing fallback explicitly', () => { - expect(caps).toContain('roots-capable hosts'); - expect(caps).toContain('explicit fallback is still required'); + expect(caps).toContain('./client-setup.md'); + const clientSetup = readText('docs/client-setup.md'); + for (const client of ['Claude Code', 'Cursor', 'Codex', 'Windsurf']) { + expect(clientSetup).toContain(client); + } + }); + + it('states explicit selection as the portable default and scopes compatibility behavior', () => { + expect(caps).toContain('portable default is explicit selection'); + expect(caps).toContain('absolute path as `project`'); + expect(caps).toContain('server returns `selection_required` instead of guessing'); + expect(caps).toContain('Roots are not required for the documented setup'); + expect(caps).toContain( + 'does not establish native agent continuation or concurrent HTTP isolation' + ); }); }); @@ -144,16 +156,27 @@ describe('docs/client-setup.md multi-project guidance', () => { const clientSetup = readText('docs/client-setup.md'); it('documents the project routing contract', () => { + expect(clientSetup).toContain("current repository's absolute path as `project`"); expect(clientSetup).toContain( - 'Automatic multi-project routing is evidence-backed only when the MCP host announces workspace roots.' + 'Without a configured folder, known root or explicit selection, tools return `selection_required`' ); + expect(clientSetup).toContain("They do not guess or index the user's home directory"); + expect(clientSetup).toContain('One valid root can auto-select'); expect(clientSetup).toContain( - 'the server returns `selection_required` instead of guessing' + 'several roots with no active selection return `selection_required`' ); + expect(clientSetup).toContain('Explicit selectors must stay within announced roots'); }); - it('keeps the three verification flows aligned with the roots-first contract', () => { - expect(clientSetup).toContain('Multiple projects on a roots-capable host'); - expect(clientSetup).toContain('Ambiguous or no-roots selection'); + it('separates process-local routing proof from native and HTTP acceptance', () => { + expect(clientSetup).toContain('A to B to A in one no-roots stdio process'); + expect(clientSetup).toContain('CBC routes omitted-project calls using its selected project'); + expect(clientSetup).toContain( + 'Select again after a server restart or when changing repositories' + ); + expect(clientSetup).toContain('native retrieval and continuation did not pass'); + expect(clientSetup).toContain( + 'stdio process-local selection does not establish concurrent HTTP isolation' + ); }); }); diff --git a/tests/proof-truth-surfaces.test.ts b/tests/proof-truth-surfaces.test.ts index cabb7a5..778f838 100644 --- a/tests/proof-truth-surfaces.test.ts +++ b/tests/proof-truth-surfaces.test.ts @@ -49,6 +49,7 @@ const gateArtifact = readJson('results/gate-evaluation.json'); const comparatorEvidence = readJson('results/comparator-evidence.json'); const benchmarkDoc = readText('docs/benchmark.md'); +const discoveryDoc = readText('docs/benchmark-prior-public-report.md'); const comparisonDoc = readText('docs/comparison-table.md'); const registryChecklist = readText('docs/registry-sync-checklist.md'); const readme = readText('README.md'); @@ -72,17 +73,19 @@ describe('proof truth surfaces', () => { expect(comparatorEvidence['codebase-memory-mcp']).toBeDefined(); }); - it('keeps the proof docs aligned to the current gate artifact', () => { - expectContains(benchmarkDoc, [ + it('keeps archived discovery proof aligned to its retained gate artifact', () => { + expectContains(discoveryDoc, [ 'discovery benchmark', `\`${gateArtifact.gate.status}\``, '`claimAllowed`' ]); - expectContains(comparisonDoc, [ - 'Comparator Summary', - `\`${gateArtifact.gate.status}\``, - `claimAllowed\` stays \`${String(gateArtifact.gate.claimAllowed)}\`` - ]); + expect(discoveryDoc).toContain( + `\`claimAllowed\` remains \`${String(gateArtifact.gate.claimAllowed)}\`` + ); + for (const currentDoc of [benchmarkDoc, comparisonDoc]) { + expect(currentDoc).toContain('./benchmark-prior-public-report.md'); + expect(currentDoc).toContain('separate 24-task discovery report'); + } expectContains(registryChecklist, [ `claimAllowed: ${String(gateArtifact.gate.claimAllowed)}`, gateArtifact.gate.status @@ -96,8 +99,8 @@ describe('proof truth surfaces', () => { if (rawClaude.averageFirstRelevantHit === null) { expect(rawClaudeGate.status).toBe('pending_evidence'); expect(rawClaudeGate.missingMetrics ?? []).toContain('averageFirstRelevantHit'); - expect(benchmarkDoc).toMatch(/raw Claude Code[\s\S]*averageFirstRelevantHit[\s\S]*null/i); - expect(comparisonDoc).toMatch(/raw Claude Code[\s\S]*pending_evidence/i); + expect(discoveryDoc).toMatch(/raw Claude Code[\s\S]*averageFirstRelevantHit[\s\S]*null/i); + expect(discoveryDoc).toMatch(/raw Claude Code[\s\S]*pending_evidence/i); expect(registryChecklist).toContain('averageFirstRelevantHit: null'); } }); @@ -108,8 +111,7 @@ describe('proof truth surfaces', () => { ); if (codebaseMemoryGate?.status === 'failed') { - expect(benchmarkDoc).toMatch(/codebase-memory-mcp[\s\S]*gate: `failed`/i); - expect(comparisonDoc).toMatch(/codebase-memory-mcp[\s\S]*gate: `failed`/i); + expect(discoveryDoc).toMatch(/codebase-memory-mcp[\s\S]*gate: `failed`/i); expect(registryChecklist).toContain('comparator artifact `ok` but gate `failed`'); } @@ -118,19 +120,35 @@ describe('proof truth surfaces', () => { .map(([name]) => name); for (const comparatorName of setupFailedComparators) { - expect(benchmarkDoc).toContain(`\`${comparatorName}\``); - expect(comparisonDoc).toContain(`\`${comparatorName}\``); + expect(discoveryDoc).toContain(`\`${comparatorName}\``); } }); - it('keeps package-facing proof mentions secondary and discovery-only', () => { - expectContains(readme, ['discovery-only proof', gateArtifact.gate.status, 'claimAllowed']); + it('keeps package-facing evidence scoped to retrieval and historical CLI observations', () => { + expectContains(readme, [ + '/docs/benchmark.md', + '/docs/benchmark-prior-public-report.md', + 'retrieval observations, not a winner or a coding-quality claim', + 'does not establish patch correctness or end-to-end coding quality' + ]); expectContains(capabilities, [ - 'discovery-only', - gateArtifact.gate.status, - `claimAllowed: ${String(gateArtifact.gate.claimAllowed)}` + 'docs/benchmark.md', + 'does not measure patch correctness or end-to-end task completion', + 'missing or failed setup and comparator runs remain visible' + ]); + expectContains(demo, [ + 'does not verify the current package or an MCP connection', + 'machine-specific repository root is omitted', + 'not a universal performance or coding result' + ]); + expectContains(benchmarkDoc, [ + 'does not measure patch correctness', + 'does not combine the four evidence families into a pooled score' + ]); + expectContains(comparisonDoc, [ + 'do not establish patch correctness', + 'failed attempt is counted as zero' ]); - expectContains(demo, [gateArtifact.gate.status, 'claimAllowed']); }); it('keeps shared planning summaries aligned to the same proof posture', () => {