Skip to content

feat(master): score merge demand and record shadow placements - #363

Merged
beinan merged 1 commit into
lance-format:mainfrom
beinan:feat/planner-shadow-placement
Oct 8, 2026
Merged

beinan merged 1 commit into
lance-format:mainfrom
beinan:feat/planner-shadow-placement

Conversation

@beinan

@beinan beinan commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Part of #333 (P1.7 — last shadow-phase piece). Design §4.2, spec §2.3.

What

  • scoring.rs (pure): merge_score = max(pending/min_gens, bytes/min_bytes) × (1 + wait/max_age). Classes: Critical (≥256 pending, or promoted for age ≥ max_turn_wait_secs), Normal (score ≥ 1), Tail. planner_order: class asc; class 1 oldest-first, never by score (review round 2); score desc elsewhere; name tiebreak → total order.
  • Shadow pass in the planner, every reconcile: score → order → skip tails (real sweeps do too) → place on fittest executor by headroom → write Shadow assignments (replaced wholesale per pass, leader-token guarded). Then compare with reality (real MergeWal task active?) → scheduler_shadow_disagreements_total{kind} + per-pass gauge. This is the number that must approach zero before P2.
  • Gauges: planner_merge_score{target}, planner_units{class}, planner_shadow_placements, planner_shadow_unplaceable.
  • Spec test 5 as a source-level test: no file outside planner.rs/executors.rs/operator route references assignments.
  • Deferred to P2: compaction/index scoring, class 2 commit-ready, max_consecutive_turns.

Validation

  • Scoring: thresholds; age doubles score at max_age; promotion for age; class 1 oldest-first beats a 100× score (spec test 6); class before score; total stable order (20 items, forward == reverse).
  • Planner etcd test extended: exactly one Shadow assignment for the 42-pending table on the live heartbeat, none for the empty table, headroom shows it, enqueuing a real task → next pass replaces not accumulates.
  • Isolation test passes; clippy -D warnings, fmt, typos clean.

🤖 Generated with Claude Code

P1.7, the last shadow-phase piece (design §4.2, spec §2.3).

scoring.rs: pure functions. merge_score = max(pending/min_gens,
bytes/min_bytes) x age_boost, with age_boost = 1 + wait/max_age.
Classes: Critical at pending >= critical_generations (256) or wait >=
max_turn_wait_secs (promoted for age); Normal at score >= 1; Tail
below. planner_order: class ascending; class 1 sorts oldest wait first,
never by score (review round 2 - a promoted tail cannot lose to a hot
table's critical merge); score descending elsewhere; target name as the
final tiebreak so the order is total. Compaction/index scoring, class 2
and max_consecutive_turns wait for P2's preparation integration.

planner.rs shadow pass, every reconcile: score, order, skip tails (the
real sweeps do too), place each unit on the fittest executor by
headroom (slots and bytes from durable reservations), and write Shadow
assignments replaced wholesale each pass under the leader token. Then
compare with reality: planner-would-place vs a real MergeWal task
active per table; each mismatch increments
scheduler_shadow_disagreements_total{kind} and the per-pass gauge.
Gauges for score per target, units per class, placements, unplaceable.

Spec test 5 as a source-level test: no file outside planner.rs,
executors.rs and the operator route may reference assignments; a stray
read fails CI, not production.

Validation: scoring unit tests cover thresholds, age lift (score doubles
at max_age), promotion for age, class 1 oldest-first beating a 100x
score (spec test 6), class before score, and a total stable order. The
planner etcd test now also asserts: exactly one Shadow assignment for
the 42-pending table against the live heartbeat, none for the empty
table, headroom shows the reservation, and after a real MergeWal task
is enqueued the next pass replaces rather than accumulates the row.
Clippy -D warnings, fmt, typos clean.

Refs lance-format#333.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@beinan
beinan merged commit fe28cfe into lance-format:main Oct 8, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant