fix(session-ingest): unblock catch-up convergence wedged by oversized reads and all-skipped refreshes

[spa5k]

Fixes #3183

Root causes

Two deterministic wedges combined to stall tracedecay sessions import historical catch-up on large host histories (repro: ~1,077 codex rollouts / ~3.4 GB raw; ingested store >10 GB).

1. Unbounded unoffered-record read in rewrite completion (tracedecay-global-db)

complete_observation_source_rewrite materialized every unoffered observation id for a source in one statement:

SELECT observation_id FROM observation_source_presence
WHERE source_key = ?1 AND generation <> ?2

The exact-SQL engine refuses queries that materialize more rows than its hard cap (MAX_QUERY_ROWS = 10_000). One long host session can hold more unoffered records than that cap (measured on the reproducing store: a source with 10,051 unoffered records; sources with 41k/21k/20k total records exist), so completion failed deterministically with projection storage operation read unoffered source records failed: exact SQL query materialization exceeded its limit.

The failure maps to a retryable projection_storage_retry_scheduled outcome, so catch-up re-attempted forever: each pass re-read the full transcript corpus, re-processed collision skips, and never committed a frontier. The retained session authority never mounted, so sessions search stayed unavailable.

Fix: keyset-paginate the unoffered ids (ordered pages of 1,000) inside the completion transaction. Retiring never touches observation_source_presence, so the cursor is stable across pages; the existing settle deletes remain the authoritative cleanup.

2. All-skipped refresh frontiers refused activation (tracedecay-session-temporal-store)

validate_candidate_frontier rejected any frontier advance that produced no canonical message outputs:

if expected.is_empty() && source_frontier != base_frontier { return Err(...) }

A refresh whose every new observation was deterministically skipped (effects recorded with output_count = 0 — e.g. host non-conversational records, 283k of them in the reproducing store) legitimately produces zero outputs. Activating that generation failed with candidate generation has no canonical message outputs past its base frontier and the temporal refresh worker retried the same session without bound (observed 332+ consecutive retries), stalling the convergence queue behind it.

Fix: keep the refusal only for an advance that processed no observation at all, detected via effects recorded past the base frontier. A frontier that moved over all-skipped observations activates normally.

Verification

• New regression test rewrite_completion_retires_unoffered_records_beyond_the_query_cap seeds 10,003 unoffered records (past both the page size and the engine cap); it fails against the old code with the exact production error and passes with the fix.
• Both crate test suites green: tracedecay-global-db (394), tracedecay-session-temporal-store (153).
• Validated against the original wedged 10.6 GB store with a debug build:
  • catch-up failures: ~4/min sustained retry storm → 0
  • collision-skip handling that previously could never complete now records durable skips and advances (the observation cursor progresses past colliding items)
  • a previously wedged session refresh now reaches session refresh complete; receipt complete and activates its generation (verified in session_temporal_generations)

Note on observability

The retry outcome (projection_storage_retry_scheduled) never logged the underlying storage error anywhere, which made this class of wedge undiagnosable from logs alone — the root cause here was found with a temporary durable_detail() log line. Not included in this PR to keep the diff minimal, but worth considering a debug/warn line on that mapping.

---

[changeset-bot]

⚠️ No Changeset found

Latest commit: e841b14

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[devin-ai-integration]

🔍 Devin Review: 2 flags

Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

[spa5k]

@codex review this

[chatgpt-codex-connector]

To use Codex here, create a Codex account and connect to github.