Skip to content

Add bounded knowledge retrieval - #179

Merged
Jesper Schulz-Wedde (JesperSchulz) merged 2 commits into
mainfrom
jesperschulz-integrate-bounded-retrieval
Sep 11, 2026
Merged

Add bounded knowledge retrieval#179
Jesper Schulz-Wedde (JesperSchulz) merged 2 commits into
mainfrom
jesperschulz-integrate-bounded-retrieval

Conversation

@JesperSchulz

Copy link
Copy Markdown
Contributor

Problem

Review skills repeatedly carried broad knowledge-index/model context and relied on report transport that could be incomplete or silently repaired. That made retrieval expensive, continuation state weak, and leaf-report acceptance difficult to validate deterministically.

Solution

  • Add lossless deterministic catalog paging capped at 16,000 serialized UTF-8 bytes, including the output newline.
  • Return every applicable or conditionally applicable catalog row without ranking, top-k filtering, query omission, or basename deduplication.
  • Retrieve exact article or sample bodies in stable chunks of at most eight paths, preserving full text, byte count, and SHA-256 with strict UTF-8 and path-safety checks.
  • Treat Entry's prepared v1 index (sourceSnapshot / sourceSha256) as the catalog freshness boundary; bind continuations to the exact raw index and normalized request/candidate state while validating selected live bodies on retrieval.
  • Make applicability and unknown dimensions explicit in the additive v2 catalog format.
  • Warn and skip invalid partner articles per file so valid siblings remain usable, while retaining complete search-time validation for corrupted or external prepared indexes.
  • Give each review leaf private GUID-scoped scratch/report paths and require deterministic validation of the exact Task return before rollup.
  • Apply the generic bounded Source workflow to all 16 canonical Microsoft review leaves without changing their Relevance, Worklist, Action, or Output rules.

Safeguards

  • No ranking, top-k selection, truncation, summarization, query-based omission, or other catalog/body data loss.
  • No runner framework, scanner, evidence ledger, model policy or allocation, domain shortcut, Shopify fixture, or domain-specific analyzer rule.
  • Invalid source metadata, noncanonical sentinels, invalid version ranges, unsafe paths, stale continuations, selected-body changes, broken sample links, oversized batches, and invalid UTF-8 fail explicitly or warn/skip at the documented boundary.
  • PR Separate deterministic analyzer rules from BCQuality #178's analyzer/knowledge separation is preserved: no deleted knowledge article or sample is restored, including the obsolete ApplicationArea guidance.

Validation

  • Knowledge index: 282 current articles, deterministic and complete.
  • Retrieval: all 282 articles and 457 samples round-tripped exactly; bounded pages, applicability, continuations, prepared-index boundaries, mutation detection, malformed/stale/unsafe/oversize failures, strict UTF-8, invalid sibling skip, semantic metadata parity, and layer validation covered.
  • Frontmatter validator: 0 errors, 0 warnings.
  • Review fixtures: 34 cases across 17 leaf domains.
  • All changed PowerShell files parse successfully.
  • Tracked and untracked whitespace checks and git diff --check pass.
  • All 16 leaf diffs remain Source-only, all 52 upstream deletions since the reviewed old base remain absent, and no generated artifacts are included.

Measured-experiment caveat

Experiments motivated these conservative bounded-retrieval and exact-transport mechanics, but they did not prove a production model mix or quality equivalence. This change therefore includes no model allocation or model-policy decision.

Jesper Schulz-Wedde and others added 2 commits September 11, 2026 11:46
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0d8b7764-f15a-49ea-8d50-d9147334f8af
Catalog matching compared an Int32 -BCVersion against a bigint range bound.
PowerShell coerces the right operand to the left operand's type, so a bound
wider than Int32 threw a conversion error and failed the whole domain catalog
rather than the single row. Metadata validation already accepts such bounds,
so compare as bigint on both sides.

The shared pager built its oversized-row message with $row.path, which
throws under Set-StrictMode -Version Latest when a row carries no path,
replacing the explicit bound failure with a property-lookup error. Resolve the
path defensively for dictionary and object rows so the offset-based fallback
is reachable.

Both paths gain regression coverage that fails without these fixes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@JesperSchulz
Jesper Schulz-Wedde (JesperSchulz) merged commit 5159706 into main Sep 11, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant