Skip to content

fix(search): bound retrieval and progressively read document evidence - #7789

Merged
icecrasher321 merged 2 commits into
stagingfrom
codex/progressive-document-search
Sep 11, 2026
Merged

fix(search): bound retrieval and progressively read document evidence#7789
icecrasher321 merged 2 commits into
stagingfrom
codex/progressive-document-search

Conversation

@icecrasher321

@icecrasher321 icecrasher321 commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Give live search SQL one shared deadline across connection acquisition, ranking, fallback, and hydration. Cancel expensive statements in PostgreSQL and expose incomplete retrieval explicitly; no usable evidence returns a retryable error.
  • Return verbatim search previews and let Assistant read more by chunk and character position, with bounded pages, current authorization, secret projection, and citations. Remove the unreleased ordinal-offset reader contract and use the same precise continuations on the organization document page.
  • Extend existing diagnostics to separate pool wait, SQL execution, document reads, and original versus returned text size. Extend the existing realistic harness rather than adding a separate reporting system.

Companion Assistant contract and telemetry change: https://github.com/simstudioai/mothership/pull/495. Deploy these changes together.

Type of Change

  • Bug fix

Testing

  • 1,085 search, authorization, provenance, reader, and route tests; includes protected query projection and stale continuation regressions.
  • Real PostgreSQL/pgvector harness with 45,000 synthetic chunks: server cancellation, exhausted pool, partial and fully timed-out retrieval, permission revocation, concurrent search, and organization scope.
  • Live local Go Assistant + Sim + configured model: search, targeted document read, correct facts, and citation. Synthetic search payload fell from 54 KB to 28 KB with the same 15 results.
  • Full workspace typecheck, lint, all 46 audits, API validation, block-registry audit, docs manifest, and generated contract checks.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel

vercel Bot commented Sep 11, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Sep 11, 2026 11:03pm UTC

Request Review

@greptile-apps

greptile-apps Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge, with no accepted new findings or outstanding previous findings.

Summary

  • Distinguishes complete, partial, and retryable timed-out retrieval.
  • Cancels expensive PostgreSQL work and records pool, SQL, hydration, and document-read diagnostics.
  • Returns projected, bounded search previews with citations and precise document continuations.
  • Adds authorization, provenance, stale-position, cancellation, and realistic integration coverage.

Diagram

sequenceDiagram
    participant A as Assistant
    participant S as Search tool
    participant R as Retrieval
    participant D as Document reader
    participant DB as PostgreSQL

    A->>S: search_workspace(query)
    S->>S: Project protected query
    S->>R: Search with shared deadline
    R->>DB: Vector and keyword retrieval
    DB-->>R: Ranked authorized rows
    R-->>S: Complete or partial retrieval
    S-->>A: Bounded previews + chunk indexes + citations
    A->>D: read_document(chunk index, offset)
    D->>DB: Load currently authorized enabled chunks
    D-->>A: Bounded verbatim page + precise next position
Loading

Reviews (2) · Last reviewed commit: "fix(search): protect query output and va..."

Comment thread apps/sim/lib/copilot/tools/server/knowledge/workspace-search.ts Outdated
Comment thread apps/sim/lib/knowledge/search/queries.ts
Comment thread apps/sim/lib/knowledge/application/read-search-document.ts
@icecrasher321
icecrasher321 force-pushed the codex/progressive-document-search branch from c8d09ca to 39fe2b8 Compare September 11, 2026 23:03
@icecrasher321

Copy link
Copy Markdown
Collaborator Author

@greptile

@icecrasher321

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

@icecrasher321
icecrasher321 merged commit 8e2623d into staging Sep 11, 2026
34 checks passed
@icecrasher321
icecrasher321 deleted the codex/progressive-document-search branch September 11, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant