Repository navigation
feat(io): a segment-keyed IO protocol for planner-driven scans - #10568
Draft
joseph-isaacs wants to merge 1 commit into
Draft
joseph-isaacs wants to merge 1 commit into
joseph-isaacs wants to merge 1 commit into
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
vortex_io::segments, an IO protocol for planner-driven scans that is keyed by segment source and source request. It sits next to the byte-rangevortex_io::requestprotocol, which the V2 executor ondevelopstill uses. This PR is additive: it changes nothing else, so it compiles ondevelopby itself.#10527 is stacked on this PR. It switches the planning traits and the V2 executor to this protocol, adds
FileSegmentIoto serve it, and removesrequest.Changes
IoKey { source: SegmentSourceId, request: SourceRequest }, whereSourceRequest ::= Size | Segment(id). It also carries anIoIntent:Fetch,PrefetchorAnnounce. One service can serve many segment sources, and a source can answer more than segment reads.SourceResult: eitherSize(u64)orBytes(BufferHandle). The checked accessors aresize()andinto_bytes().SegmentIoSession::submitregisters a whole batch before any read starts, and returns oneIoHandleper request.SegmentIoSession::poll_readyreports each fetch's owner once the fetch finishes, including fetches that finished at submit, and drives the session's prefetches.FetchTrackerimplements this for sessions.SegmentIoService::session()opens one session per scope.RegisteredSourceis a source's registration with a service.Tests cover handles, results checked against what was asked, and
FetchTracker.Checks run:
cargo clippy -p vortex-io --all-targets --all-features -- -D warningscargo +nightly-2026-09-10 fmt -p vortex-ioRUSTDOCFLAGS="-D warnings --cfg docsrs" cargo doc --no-deps -p vortex-io --all-featurescargo test -p vortex-io --lib segments(4 tests)typosAPI Changes
New:
vortex_io::segments.🤖 Generated with Claude Code
https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9