Repository navigation
feat(master): executor heartbeats and headroom from durable reservations - #362
Merged
Merged
Conversation
P1.6 (docs/design/scheduler-p1-data-model.md §2). Behind PLANNER_ENABLED, every master publishes a leased P/executors/<id> heartbeat: kind, slots per task kind mirrored from the real dispatch pools (general, compact, merge plus the resident lane), byte budget from the local append budget, a diagnostic bytes_free_sample, and draining. Draining stops the heartbeat and revokes the lease so the key disappears promptly. Headroom is a pure fold of heartbeats and P/assignments/ records: slots_total - sum(reserved_slots) and bytes_total - sum(reserved_bytes) over every assignment that is not Released, including ones not yet bound, and never the sampled free bytes (review point 4). An assignment whose executor has no heartbeat still reserves - reported as draining with zero totals so a leader can see and release it; capacity cannot be double booked across a failover. The planner loads headroom on every reconcile (planner_executors, planner_reserved_bytes_total) and GET /api/v1/scheduler/executors exposes it. No assignments are written yet; the Shadow state exists for the next PR. Assignment and Heartbeat carry v=1. TaskKind gains PartialOrd/Ord/Hash so it can key the slot maps. Validation: unit tests for spec tests 3 and 4 - an executor whose sample says 2 GiB free but with 2 GiB reserved (one Assigned, one Shadow) gets no placement while a Released assignment does not count; assignments outlive a missing heartbeat; draining takes nothing and a zero byte budget defers to slots; the fold is order independent. etcd test: heartbeat appears under a lease with the configured slot counts; a shadow assignment reduces merge headroom from 3 to 2; begin_drain ends the loop, the key is gone, the assignment still reserves. Clippy -D warnings, fmt, typos clean. Refs lance-format#333. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
17 of 18 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #333 (P1.6). Spec §2.
What
Behind
PLANNER_ENABLED: every master publishes a leasedP/executors/<id>heartbeat — kind, slots per task kind mirrored from the real dispatch pools (general / compact / merge + resident lane), byte budget from the local append budget, a diagnosticbytes_free_sample,draining. Drain stops the heartbeat and revokes the lease.Headroom is a pure fold of heartbeats +
P/assignments/records:slots_total − Σ reserved_slots,bytes_total − Σ reserved_bytesover every non-Releasedassignment including unbound ones, never the sample (review point 4). An assignment whose executor has no heartbeat still reserves (shown as draining, zero totals) so capacity cannot be double-booked across a failover.Planner loads headroom on every reconcile;
GET /api/v1/scheduler/executorsexposes it. No assignments are written yet —Shadowstate is defined for the next PR.TaskKindgainsPartialOrd/Ord/Hashfor the slot maps.Validation
fitsfalse; Released doesn't count.begin_drainends loop, key gone, assignment still reserves.-D warnings, fmt, typos clean; api crate tests pass.🤖 Generated with Claude Code