Repository navigation
Explain transcript pagination ending before its reported raw message count #3211
Description
Activity
Issue 3211: bounded read-only evidence
Observed against installed beta.76 on 2026-10-08. The same exact Codex session was used throughout. No database queries, profile changes, refreshes, or body dumps were used.
Public reads
- The dashboard traversal returned 64, 64, 64, 64, 34 messages: 290 distinct message IDs, terminal cursor. Every page reports raw message_count 478, summary_node_count 0. Initial pages report partial coverage; the terminal page reports ready with unknown coverage.
tracedecay_session_lookupin Forensic mode independently returned 64, 64, 64, 64, 34 anchors, 290 distinct anchors, terminal cursor. Its terminal MCP envelope reports complete coverage for 34 records. Therefore Current-only copy/correction suppression does not explain the discrepancy.- Exact lookup arguments for the first page:
{"session_id":"<session-id>","meta":{"order":"relevance","page":{"page_size":64},"projection":"summary","temporal":{"kind":"forensic"}},"format":"json"}. Continuations preserve these fields and setmeta.page.cursortooutcome.value.page.cursor.cursorfrom the prior response. Only anchor IDs and coverage were inspected. lcm_describefor this provider/session confirms raw_message_count 478, external_payload_count 0, first_store_id 10561, last_store_id 209718. Its truncated overview contains previews; only metadata fields were extracted. No stored-body concatenation was performed.- The exact-session
lcm_statusindependently confirms raw 478, lossy_records 0, missing_payload 0, schema 15. - Project
lcm_doctorreports stale projection, reasonhistorical_retry:history_admission_saturated, convergence epoch 0/converging, backlog 2, last_progress null, stuck_refresh 1, stuck_progress 1, cursor_key_absent 1. Its health probe is partial (bounded_row_probe_incomplete); zero missing-anchor/missing-receipt findings are not proof of absence. Projection state is project-wide, not proven to belong to this exact session.
Source findings
tracedecay-mcp/src/handlers/dashboard_lcm.rs: dashboard counts come fromdescription.raw_message_count; retrieval uses Current/Occurrence.tracedecay-lcm/src/query/status.rs: that count is the raw-message table count, not canonical representatives returned by retrieval.tracedecay-session-temporal-store/src/retrieval/queries.rs: session candidates use session/provider/generation/keyset bounds and identity-size bounds. No role/body filter explains the count difference. Empty/default semantic filters admit ordinary messages.tracedecay-temporal-query/src/resolution/resolver.rs: Forensic disables copy/supersession suppression. Ancestry lineage alone does not automatically create copy relations.- Candidate paging keeps the last emitted key when a budget leaves an un-emitted row; clauses advance only at EOF. No concrete pagination-loss defect was found in this bounded review.
tracedecay-dashboard-api/src/lcm_api.rs:574: Ready pages deliberately use complete coverage for aggregates but Unknown for Session/Search. This explains the terminal coverage presentation defect independently of the missing-count question.
Smallest proven coverage correction and test
Use existing
DashboardCoverageV1::complete(returned_count(&page), "canonical hydrated records")for Ready Session/Search pages as well. The count is the terminal page's canonical records (34 live), never raw 478 or traversed total 290. Preserve partial pages and continuation behavior.Extend the existing isolated
lcm_large_session_pages_preserve_continuation_and_timeline_countsHTTP test incrates/tracedecay/tests/dashboard_api_test/api.rs: assert partial/examined 64 on page one, then complete/eligible 6/examined 6/matched 6/omitted 0 on terminal page two; keep 70 unique IDs, raw count 70 and timeline count 70. The regression is being added after the #3210 page-capacity fix.Historical admission ownership and diagnostic limits
session_temporal_refresh_scheduler/registry.rs:82bounds daemon-wide historical admission at two permits (or fewer cores). Mounted project and profile schedulers share this semaphore.worker.rs:613reportshistory_admission_saturatedonly whentry_acquirefails before the historical pass. The same permits also cover LCM convergence, including summary work.- Refused work waits on semaphore release via
HistoryRelease::Admission; the idle loop also rechecks at 60 seconds. The refusal state alone does not demonstrate deadlock or identify a permit owner. - Read-only
session_refresh_statusrequires an existing opaque handle from begin (enforced in retained/session_refresh.rsadmitted_action). No such handle is available. Do not begin a refresh or invent frontier values to inspect the background scheduler. - Existing isolated regression:
session_suite/session_runtime/retained_history.rs::saturated_history_admission_defers_the_pass_and_resumes. - Relevant existing measurement spans:
daemon.scheduler.session_temporal.history,.history_release_wait,.history_idle_wait,.projection;sessions.ingest.project,sessions.ingest.project.codex,sessions.ingest.user,sessions.ingest.user.codex; summary convergence spans. Enable their modules at trace andTRACEDECAY_SPAN_TIMINGS=1only for a coordinated measurement, confirm spans emit, and remove overrides afterward. Event:retained historical session ingest pass will retrywithreason_code.
Current conclusion: projection/source lag is the leading hypothesis, but the exact session's missing 188 records have not been attributed to an incomplete frontier or a concrete projection defect. Do not label all raw records duplicates, change cursor semantics, raise limits, or force unsupported refresh frontiers without that evidence.
Commit 8a9bb56 fixes the independently proven terminal coverage defect: Ready transcript/search pages preserve complete coverage of the returned canonical records. The raw session count remains separate. The unused aggregate-only classifier was deleted.
The production HTTP regression failed with Unknown coverage before the change, then passed: the initial page remains partial with 64 records; the terminal page reports complete with eligible/examined/matched 6 and omitted 0 while raw message_count remains 70. The same test verifies 70 unique IDs and timeline count 70. Bazel production Clippy, pinned formatting, and diff checks pass.
A bounded 120-second trace of the installed beta.76 captured two completed profile-history passes (29.39s and 43.50s), proving progress and permit release during that window. A project-history span closed during shutdown and is not treated as a completed pass. Slow snapshot/query spans were observed, but cold startup and trace overhead prevent treating these as steady-state latency. No semaphore leak or missing-record cause has been proven. The temporary runtime logging override was removed and the normal service restored; no profile data was deleted.
Keep this issue open: the 478-raw/290-canonical discrepancy still needs attribution and post-release verification.
Further bounded source review: Forensic retrieval still uses the admitted generation and excludes SourceRecordRetired effects before resolver processing. However, completed retirement also deletes the raw output when it has no remaining active owner; shared output retains an active owner's rendering and requests projection reset. The simple explanation that all 188 extra raw rows are intentionally retained retired messages is therefore unsupported.
The exact original transcript path was not available through the inspected public metadata. A bounded metadata-only load probe encountered TemporalStoreReadFailed after approximately five seconds; no source bodies or direct database rows were inspected, and no data was changed. The normal daemon still shows history progress and intermittent reader contention. The independently reproduced background-priority defect is fixed in #3212, but its contribution to the count discrepancy remains unproven.
Keep this issue open for attribution and verification after the official updated binary is installed. No profile wipe is justified by the current evidence.
Post-release verification on official beta.79 (
82e9e85e3a89e7a68d72c7429b5966593b3bbc15), installed with verified release-workflow provenance: the real Rspack project's session tools negotiate correctly, and retained load/describe responses hydrate successfully throughtracedecay_retrieve.A bounded metadata-only walk of one authentic local Codex session advanced through 63 pages of 64 records: 4,032 distinct
(store_id, provider, message_id)identities, no repeats, and every opaque cursor advanced. Pages truthfully remained partial and still carried a continuation when the 180-second observation ended. No transcript contents were written to the diagnostic report. This is live pagination evidence, not a terminal-page result and not attribution of the original Linux session's 188-record discrepancy.Keep this issue open. The original session identity and its raw/canonical attribution remain unavailable; the independent coverage/count repairs must not be presented as proof that those 188 records were intentionally omitted or recovered. No raw-row backfill, profile wipe, or budget increase was used.
Closing. The question this issue asked now has a typed answer on every surface, and the stall behind it is fixed. Verified on origin/master
070489af58.Attribution. The gap between the raw count and the paged transcript is incomplete projection, not canonical dedup or pagination loss. #3420 traced it: the observation drain writes each raw row in the same transaction as its temporal effect, but occurrences become pageable only after a background refresh publishes a new generation. The original session's doctor showed exactly that refresh stuck (
historical_retry:history_admission_saturated). #3456 fixed why a refused project almost never got its history pass: the waiter dropped the permit its queue turn granted and retried withtry_acquire(), handing the permit to the next waiter.Truthful counts.
lcm_describe, the dashboard session page, the explorer and the Sessions inspector now reportunpublished_message_count: raw rows the active generation has not published, matched by projection provenance. The terminal page stays partial withomitted >= unpublished, soeligibleequalsmessage_countand a traversal that ends short of the raw count says why (#3420, #3444; terminal coverage from #3218/#3141).Bazel on master
070489af58://crates/tracedecay:dashboard_api_testapi::lcm_session_page_reports_messages_the_temporal_projection_has_not_published: 1 passed. Seeds 3 messages, publishes, seeds 2 more. The page reportsmessage_count5, 3 messages,unpublished_message_count2,omitted2,eligible5. Before fix(lcm): count transcript messages the projection has not published #3420 it reportedunpublished_message_countnull witheligible4.//crates/tracedecay:mcp_suitelcm_describe_behavior: 2 passed, includingtracedecay_lcm_describe_counts_messages_beyond_the_published_generation.//crates/tracedecay:session_suiteretained_history: 11 passed, includinga_saturated_history_pass_runs_with_the_permit_its_queue_turn_grants(fails if the granted permit is dropped).
The original beta.76 session cannot be re-read. Its identity is unavailable (see the 2026-10-10 comment), so its 188 rows are attributed by mechanism, not row by row. No raw-row backfill, budget increase or profile wipe was used. If a master-or-later build shows a terminal page whose
message_count - returnedexceedsunpublished_message_countplus the counted omissions, that is a new defect. Open it with the describe output.
The installed beta.76 dashboard shows a raw session count of 478 messages, but a read-only walk of its canonical transcript using the supported 64-message page size terminates after 290 unique messages (pages 64,64,64,64,34). Every page reports counts.message_count=478 and summary_node_count=0; every returned summary_nodes array is empty.
The first four pages are partial with eligible/examined=64, omitted=0, and lcm_temporal_read_incomplete. The final page is domain_state=ready with no continuation, but coverage is unknown (all counts null, no omission reasons).
This does not yet prove message loss: Current/Occurrence retrieval can differ from raw storage counts through canonical resolution. However, the user-facing transcript ends without accounting for the difference. Determine whether the 188-record difference is intentional canonical representation, incomplete projection, or pagination omission, and make coverage/count guidance truthful. No direct database queries or transcript-content disclosure were used.
Observed during #3210 investigation; this is separate from the proven 100-vs-64 context capacity mismatch. Preserve canonical redaction/content authority; do not backfill from raw rows or increase budgets to hide the discrepancy.
Current verification: Merged #3218 distinguishes stored-message counts from canonical transcript rows. Merged #3141 preserves 64-record continuation and reports measured coverage on terminal pages. These repairs do not establish why the original session had 188 fewer canonical rows, so the issue remains open. No raw-row backfill or operator-data reset was used.