index: preserve overlapping file rank penalties - #1164
Merged
Merged
Conversation
Precomputed file categories retain only the first matching classification, so generated tests and similar overlaps lost ranking penalties after the indexing optimization. Compute independent ranking flags alongside the category before skipped content is discarded, keeping ranking cheap without having to reconstruct classifications later. The flags fit in existing Document padding on amd64 and are only needed for initial shard ordering. Keep the persisted category and index format unchanged: merge and explode preserve stored categories and relative document order rather than re-ranking reconstructed documents. Amp-Thread-ID: https://ampcode.com/threads/T-01a06c1b-9f70-777d-949b-61443d2d73c1 Co-authored-by: Amp <amp@ampcode.com>
keegancsmith
marked this pull request as ready for review
September 6, 2026 15:43
stefanhengl
approved these changes
Sep 9, 2026
Comment on lines
+643
to
+644
| b.todo = append(b.todo, &doc) | ||
|
|
Member
Author
There was a problem hiding this comment.
It is minor from a previous PR that landed. Logically it makes sense to only add the doc once we are done with it. The order we did it before was fine since we mutated a reference to do. But it feels like a bug waiting to happen having mutations to doc after we append it to TODO.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
April’s indexing optimization (#1036) reused a single precomputed category for ranking, losing overlapping penalties such as generated tests. Compute independent rank flags alongside the category before discarding skipped content, keeping ranking cheap without growing Document on amd64.
The index format stays unchanged. Merge/explode retain stored categories (as fixed in #1073) and document order; regression tests cover that round trip and skipped-content handling.