Repository navigation
Publish historical benchmark and released setup docs - #130
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a0266a1030
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ## Discovery Gate Truth | ||
|
|
||
| The gate is intentionally still blocked. | ||
| They show the test structure and task format. They are not a byte-for-byte copy of every file used by the measured sessions; all three file hashes differ from the later 30-session pilot. The manifest names tasks and base commits but does not contain their full problem statements or gold-context values. The exact machine-specific raw run directories behind the 100-attempt retrieval table, the 300-attempt repeatability history, and the 30-session agent pilot are not checked into the public repository, so this checkout cannot regenerate the reported tables with one command. The [original schema-2 retrieval summary](../results/contextbench-engineering-note-summary.json) is preserved from local commit `18e0a43a90fe67e9df0d728b2134e9d430469883`; the [sanitized evidence extract](../results/benchmark-presentation-evidence.json) preserves the exact values and hashes used for this presentation. Fixture validation checks fixture consistency, not the reported measurements. |
There was a problem hiding this comment.
The newly published headline tables depend on raw run directories, full task/gold inputs, and exact tool versions that are unavailable in this repository; this paragraph explicitly acknowledges that the checkout cannot regenerate the reported measurements. This prevents external users from reproducing the results, contrary to the repository's non-negotiable public-evaluation requirement. Check in sufficient frozen inputs and artifacts with a runnable reproduction command, or withhold these measurements from the public benchmark.
AGENTS.md reference: AGENTS.md:L145-L150
Useful? React with 👍 / 👎.
| "schemaVersion": 2, | ||
| "gitRef": "18e0a43a90fe67e9df0d728b2134e9d430469883", | ||
| "gitBlobSha256": "a26599e1302ebdbb34e259530c8d775a9325bd42714e5b03398c545ad1abaf6c", | ||
| "checkoutByteDifference": "CRLF versus LF only" |
There was a problem hiding this comment.
The claim that the source and checked-in artifact differ only by CRLF versus LF is inconsistent with the recorded hashes: the tracked summary hashes to the stated gitBlobSha256 (a26599...), but converting those bytes to CRLF hashes to 615be5..., not the recorded source hash ad8cac.... Because the README and benchmark rely on this extract as exact evidence, regenerate it from the retained source or correct the hash/difference description before publication.
AGENTS.md reference: AGENTS.md:L112-L112
Useful? React with 👍 / 👎.
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
|
||
| # Then search for the local example you need | ||
| npx -y codebase-context search --query "auth middleware" | ||
| Check an existing same-name entry before replacing it. To give the server a default folder, append that folder's absolute path to the `npx` arguments. The [client setup guide](./docs/client-setup.md) covers scopes, optional fixed-folder configuration, verification limits and the unreleased installer. Published `2.2.0`'s interactive `init` has registration bugs; use the commands above. |
There was a problem hiding this comment.
tests/mcp-client-templates.test.ts. The rewritten setup guide also removes routing text and headings that the same test still requires. When the documentation tests run, these assertions will fail. Update the tests for the new guidance or retain the required references.
| "results": [ | ||
| { | ||
| "file": "C:\\Users\\bitaz\\Repos\\codebase-context\\repos\\angular-spotify\\libs\\web\\auth\\util\\src\\lib\\interceptors\\auth.interceptor.ts:10-42", | ||
| "file": "repos/angular-spotify/libs/web/auth/util/src/lib/interceptors/auth.interceptor.ts:10-42", |
There was a problem hiding this comment.
file value replaces the original absolute path with a relative-looking path. Readers cannot treat it as an exact example of the stated run. Keep the captured value or say that the path was normalized.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
1 participant
Published setup docs recommended an unreliable init path and mixed older benchmark claims with newer candidate capabilities. This publishes direct client setup for the released package, aligns capability/CLI/demo guidance, and preserves the prior public report beside the historical benchmark and source extracts.
The change contains ten documentation/evidence files and two documentation-contract tests against current master. It excludes runtime, installer, package, lockfile and workflow changes. The follow-up fixes unshipped README links, explains the normalized demo path and mixed source line endings, and keeps existing gate assertions tied to their archived evidence rather than another benchmark family.
Validation: relative links, setup recipes, source identities, archive bytes and benchmark extracts were independently checked. All 32 affected documentation-contract tests, scoped lint/format and whitespace checks pass locally. Initial CI passed quality and 86 test files but exposed eleven documentation assertions; the revised CI run passes Quality Checks and Functional Tests (89 test files). No frozen eval data, benchmark values or runtime behavior changed.
Historical output remains historical evidence. Native first-use continuation and complete public benchmark reproduction remain unverified. Historical-publication framing does not close the reproducibility gate; no version bump, tag or npm release is included.
The maintainer explicitly approved publication of this historical record with the public reproduction gap open on October 6. This does not close the reproducibility gate or authorize new comparative claims.