Repository navigation
feat(master): score merge demand and record shadow placements - #363
Merged
Merged
Conversation
P1.7, the last shadow-phase piece (design §4.2, spec §2.3).
scoring.rs: pure functions. merge_score = max(pending/min_gens,
bytes/min_bytes) x age_boost, with age_boost = 1 + wait/max_age.
Classes: Critical at pending >= critical_generations (256) or wait >=
max_turn_wait_secs (promoted for age); Normal at score >= 1; Tail
below. planner_order: class ascending; class 1 sorts oldest wait first,
never by score (review round 2 - a promoted tail cannot lose to a hot
table's critical merge); score descending elsewhere; target name as the
final tiebreak so the order is total. Compaction/index scoring, class 2
and max_consecutive_turns wait for P2's preparation integration.
planner.rs shadow pass, every reconcile: score, order, skip tails (the
real sweeps do too), place each unit on the fittest executor by
headroom (slots and bytes from durable reservations), and write Shadow
assignments replaced wholesale each pass under the leader token. Then
compare with reality: planner-would-place vs a real MergeWal task
active per table; each mismatch increments
scheduler_shadow_disagreements_total{kind} and the per-pass gauge.
Gauges for score per target, units per class, placements, unplaceable.
Spec test 5 as a source-level test: no file outside planner.rs,
executors.rs and the operator route may reference assignments; a stray
read fails CI, not production.
Validation: scoring unit tests cover thresholds, age lift (score doubles
at max_age), promotion for age, class 1 oldest-first beating a 100x
score (spec test 6), class before score, and a total stable order. The
planner etcd test now also asserts: exactly one Shadow assignment for
the 42-pending table against the live heartbeat, none for the empty
table, headroom shows the reservation, and after a real MergeWal task
is enqueued the next pass replaces rather than accumulates the row.
Clippy -D warnings, fmt, typos clean.
Refs lance-format#333.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
17 of 18 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #333 (P1.7 — last shadow-phase piece). Design §4.2, spec §2.3.
What
scoring.rs(pure):merge_score = max(pending/min_gens, bytes/min_bytes) × (1 + wait/max_age). Classes: Critical (≥256 pending, or promoted for age ≥max_turn_wait_secs), Normal (score ≥ 1), Tail.planner_order: class asc; class 1 oldest-first, never by score (review round 2); score desc elsewhere; name tiebreak → total order.Shadowassignments (replaced wholesale per pass, leader-token guarded). Then compare with reality (realMergeWaltask active?) →scheduler_shadow_disagreements_total{kind}+ per-pass gauge. This is the number that must approach zero before P2.planner_merge_score{target},planner_units{class},planner_shadow_placements,planner_shadow_unplaceable.planner.rs/executors.rs/operator route references assignments.max_consecutive_turns.Validation
max_age; promotion for age; class 1 oldest-first beats a 100× score (spec test 6); class before score; total stable order (20 items, forward == reverse).-D warnings, fmt, typos clean.🤖 Generated with Claude Code