Repository navigation
feat(master): stats scan publishes scan-sourced demand for every shard - #356
Merged
Merged
Conversation
17 of 18 tasks
beinan
force-pushed
the
feat/scanner-demand-events
branch
2 times, most recently
from
October 8, 2026 22:14
d8eab82 to
40770e6
Compare
P1.4 (docs/design/scheduler-p1-data-model.md §1.3). The scan is the
reconciliation source for demand: it covers shards whose writer never
ran the server flush path (retired writers, pre-upgrade shards) and
corrects any writer event that was dropped.
- core: all_shard_watermarks() reads every shard's latest manifest for
(shard_id, writer_epoch, max flushed generation, pending count).
Unlike wal_shard_snapshots it does not enforce the read cap: the
scheduler must see an over-cap shard, not be refused by it.
- master: demand_publish::spawn_scan_demand runs after each successful
observation, detached and bounded (15 s timeout, 4 slots, drop and
count), publishing one scan-sourced event per shard through the same
per-dimension join as writers and executors, so a scan can only raise
a watermark and never lowers a merged mark it does not know about.
Metric master_scan_demand_events_total{result}. The scan's existing
enqueue and retirement behaviour is untouched.
Validation: etcd test with two writer shards and no server flush path:
one scan event per shard, fold gives six pending; after an executor
reports merged=2 on one shard, a second scan leaves merged intact and
does not rewrite the record. 15 scanner unit tests pass; clippy -D
warnings, fmt, typos clean.
Refs lance-format#333. Stacked on lance-format#352.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
beinan
force-pushed
the
feat/scanner-demand-events
branch
from
October 8, 2026 22:30
40770e6 to
be91a24
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #333 (P1.4). Spec §1.3, source
scan. Stacked on #359 (rebased from the superseded #352); merge after #357 and #359.What
all_shard_watermarks()— every shard's(shard_id, writer_epoch, max flushed generation, pending)from its latest manifest. Does not enforce the read cap (the scheduler must see an over-cap shard, not be refused).demand_publish::spawn_scan_demandafter each successful observation. Detached and bounded (15 s, 4 slots, drop-and-count), onescan-sourced event per shard viapublish_demand_event's per-dimension join, so a scan can only raise and never lowers amergedit does not know. Metricmaster_scan_demand_events_total{result}. Existing enqueue/retire behaviour untouched (review point 5 / P1 rule).Validation
scan_publishes_every_shard_and_never_lowers_a_watermark(etcd): two writer shards, no server flush path → one event per shard, fold = 6 pending; executor reportsmerged=2on a shard; a second scan leavesmergedintact and does not bumpmod_revision.-D warnings, fmt, typos clean.🤖 Generated with Claude Code