Repository navigation
Conversation
Record three structural bets surfaced by the read-only engine audit as Draft proposals, so the reasoning and trade-offs are reviewable before any build: - 0007: additive per-batch rollup aggregates, generalizing the existing read_trace_parts model to the summary rollups so maintenance cost tracks new rows instead of the touched event-time span. - 0008: derive service percentiles from the mergeable span_rollup histogram so alerts, overview and dashboards stop disagreeing, and drop the stored t-digest result. - 0009: append-only manifest catalog, WAL with a head block, and parallel time-ordered compaction, with the freshness-guarantee decision called out as the gating question. Each proposal names the related audit issues and marks itself measurement-gated and incremental; the cheap in-place fixes ship first regardless.
…0009 Add a dated research-validation section to each engine-redesign proposal, capturing a prior-art/best-in-class pass, an in-codebase correctness and crash-safety pass, and a performance/sequencing pass: - 0007: confirmed as ClickHouse AggregatingMergeTree (-State/-Merge); make the compaction fold mandatory in phase one, state the exactly-once and sum+count-not-avg invariants, and resolve the cross-gate snapshot design. - 0008: confirmed as the Prometheus/OTel exponential-histogram approach; the shared helper must apply the served-kind filter, replicate the served-vs-all choice, pin interpolation, and handle the zero bucket; state the 4.33% bound and add an alert-migration note. Lowest risk; ship first. - 0009: manifest/WAL/compaction each match best-in-class, but parallel compaction is not safe as drafted (single marker/attempt/recovery, overlapping input selection, and a one-merge memory budget all cause data loss or OOM); add manifest checkpointing and a WAL replay bound; incremental fixes ship first (BatchMetadata snapshot already landed in #488). Proposals stay Draft; no production code changed.
…dmission; index 0007-0011 0010 addresses the result-cache key fragmentation and sub-refresh TTL (issue #477) with a normalized key and a watermark-split extent cache. 0011 addresses partial, cost-unaware read admission (issue #480) with one admission layer and interactive/panel/background/agent classes. Both are drafts and still need the three-angle research pass before build. Also adds the missing index rows for 0007-0011.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Design proposals that came out of the read-only engine/storage audit (the same audit that produced issues #474–#487). These are for discussion, not implementation — each is
Status: Draftand the three design-heavy ones carry a research-validation section; the two new ones (0010, 0011) still need their research pass.Proposals
AggregatingMergeTreemodel; in-repo precedenttransferBatchCacheLocked). Research-validated; buildable after the cross-gate snapshot design is specified and the compaction fold is committed.service_rollup; derive percentiles at read from the mergeablespan_rollupexponential histogram so alerts, overview, and dashboards agree. Research-validated; lowest risk; ship first.Also adds the missing index rows for 0007–0011 in
docs/proposals/README.md.No code changes.