Repository navigation
feat(master): leader-elected planner folds demand events into a cache - #361
Merged
Merged
Conversation
P1.5 (docs/design/scheduler-p1-data-model.md §1.2, §3). Behind PLANNER_ENABLED (default off), one master wins P/leader under a lease and, every PLANNER_RECONCILE_SECS, folds every demand-events/ record into a per-table P/demand/<hex> cache using TableDemand::fold. Rows for tables with no events are dropped. The cache is a pure function of the stored events and is never a source of truth. Shadow only: this places nothing and removes nothing; every existing loop keeps running for every table. Outputs are the cache, the new GET /api/v1/scheduler/demand[?target=] view (leader record, per-table pending counts/bytes/age/shards, full record) and metrics (planner_is_leader, planner_tables, planner_pending_generations_total, planner_reconcile_seconds, planner_events_skipped_total, planner_reconcile_errors_total). Leadership: campaign txn on Version(P/leader)==0 writing the record and a bare P/leader-token under one lease; every cache write txn compares the token; every reconcile reads the token first even with nothing to write, so a deposed leader learns it on the next tick. Writes are chunked under etcd's 128-op limit. Loser campaigns return promptly and re-campaign after 2 s. Validation: unit tests for fold_events (groups by table, skips bad key, bad body and future version) and hex round-trip. etcd test: two masters campaign, exactly one leads (the test detects a dual leader; a mutation that ignores the campaign result fails it); the cache equals a direct fold of the events; the HTTP view reports the leader and 42 pending on the two-shard table; a forged token is refused with "leadership lost"; deleting a table's events drops its cache row on the next tick. Clippy -D warnings, fmt, typos clean. Refs lance-format#333. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
17 of 18 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #333 (P1.5). Spec §1.2/§3. The gate (#359) is merged.
What
Behind
PLANNER_ENABLED(default off): one master winsP/leaderunder a lease and everyPLANNER_RECONCILE_SECSfolds alldemand-events/into a per-tableP/demand/<hex>cache viaTableDemand::fold. Rows for tables with no events are dropped. The cache is a pure function of stored events — never a source of truth.Shadow only. Places nothing, removes nothing; every existing loop keeps running (review point 5).
Outputs: the cache;
GET /api/v1/scheduler/demand[?target=](leader record + per-table pending/bytes/age/shards + full record); metricsplanner_is_leader,planner_tables,planner_pending_generations_total,planner_reconcile_seconds,planner_events_skipped_total,planner_reconcile_errors_total.Leadership
Campaign txn on
Version(P/leader)==0writes the record and a bareP/leader-tokenunder one lease. Every cache-write txn compares the token; every reconcile reads it first even with nothing to write, so a deposed leader notices on the next tick. Writes chunked under etcd's 128-op limit. Losers return promptly and re-campaign after 2 s.Validation
fold_eventsgroups by table and skips bad key / bad body / future version (counted); hex round-trip.finished=[false, false]); cache == direct fold; HTTP view shows leader and 42 pending on a two-shard table; forged token →leadership lost; deleting a table's events drops its row next tick.-D warnings, fmt, typos clean.🤖 Generated with Claude Code