Repository navigation
perf: reduce read latency and checkpoint contention - #473
Merged
Merged
Conversation
service_rollup and edge_rollup still carried compound ART primary keys
while the span, log and endpoint summaries dropped theirs. Those two are
the larger indexes, and the pinned engine kept probing them under the
checkpoint vacuum lock this change set out to avoid, so the contention fix
was only half applied.
Drop both keys under the same invariant that already protects the scan
summaries: refreshServiceRollup replaces each affected scope in one
serialized transaction. service_rollup inserts a single grouped aggregate;
edge_rollup unions three branches whose edge_type literals ('call',
'messaging', 'peer') are disjoint and each groups by the full key, so the
complete key stays unique without the index. Keep every key column NOT NULL.
Extend the rollup-key tests to assert both tables drop their index and keep
NOT NULL keys, to check their key uniqueness after every refresh, and to
prove the cache rebuild drops a previously-present primary key on reopen.
vishr
added a commit
to labstack/pitot-bench
that referenced
this pull request
Nov 15, 2026
## Summary Companion product changes: labstack/pitot#473 Read reports can hide scheduler misses and queue delay, and count drain completions toward the offered load. Account for every scheduled arrival and qualify reads using completion time from the scheduled arrival. ```text offered arrivals → scheduler misses / shed / dispatched → dispatched queue + HTTP operation → success during load / success during drain / failure / cancellation ``` Report intended and performed operation kinds, including fallback attribution. Add the proposed `interactive-http-draft/v1` per-kind p95/p99 profile to workload metadata and qualification. Preserve historical service-time fields; legacy and undersampled reports cannot qualify against the new profile. ## Evidence - **Before:** ticker losses and scheduler queue time were absent from read evidence; drain completions could increase reported goodput. **After:** tests reconcile every offered/dispatched outcome, delayed scheduling, fallback mix, queue and completion latency, phase cutoff/drain, profile matching, and the 1,000-successful-samples-per-kind gate. - Passed `go test ./...`, `go test -race ./...`, `go vet ./...`, and staged secret scanning. - No new production capacity result is claimed. HTTP completion remains separate from browser paint. ## Merge Danger **Door:** two-way New read qualification is intentionally stricter. Standard short sweeps may use the documented best-completion topology fallback because they cannot provide enough p99 samples. Published measurements are not rewritten. **Blast Radius:** reporting Changes read-load scheduling, mixed-topology eligibility, report metadata and qualification; ingest measurement is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Dashboard reads can stall behind checkpoint vacuum locks and repeat avoidable native setup work. Reduce that contention while keeping resource budgets and admission limits unchanged.
Companion runner changes: labstack/pitot-bench#13
Evidence
Local synthetic 90-second comparison on Apple M3 Max, four DuckDB threads, 4 GB native budget, eight read connections, fixed history and no stored-result reuse. Both runs offered 40 Executor requests/s alongside 20,480 accepted telemetry records/s, with accelerated five-second maintenance.
The after run still fails zero-loss offered-load acceptance because of two generator misses. Percentiles describe successful requests; these local runs establish neither stable production capacity nor a maximum read rate. The separate 30-second candidate run completed all 1,200 offered requests.
After: the native control proves an identical unindexed table returns all 96 expected rows while checkpoint and indexed control remain pending.
just check(repository-wide Go tests, 1,790 UI unit tests, script tests, lint, generated fixtures/docs, embedded assets, dependency audits and site checks).just test-race, format-5 replay regressions, cache-upgrade and cold-start guards, opt-in harness compilation, and staged secret scanning.See the investigation for methodology, further measurements and limitations.
Merge Danger
Door: two-way
The first start after upgrade rebuilds derived DuckDB caches. Rolling back also requires a disposable-cache rebuild; canonical Parquet telemetry, SQLite control state and anomaly history are preserved.
Blast Radius: reads
Changes read execution and analytical-cache maintenance. The three scan-only summaries now rely on serialized delete-and-grouped-insert uniqueness rather than SQL primary-key enforcement, with explicit lifecycle tests. Snapshot-bound coverage remains transactional; the engine pin, telemetry format, public contracts and admission limits are unchanged.