Skip to content

feat(io): a segment-keyed IO protocol for planner-driven scans - #10568

Draft
joseph-isaacs wants to merge 1 commit into
developfrom
ji/segment-io-protocol
Draft

joseph-isaacs wants to merge 1 commit into
developfrom
ji/segment-io-protocol

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds vortex_io::segments, an IO protocol for planner-driven scans that is keyed by segment source and source request. It sits next to the byte-range vortex_io::request protocol, which the V2 executor on develop still uses. This PR is additive: it changes nothing else, so it compiles on develop by itself.

#10527 is stacked on this PR. It switches the planning traits and the V2 executor to this protocol, adds FileSegmentIo to serve it, and removes request.

Changes

  • A request names IoKey { source: SegmentSourceId, request: SourceRequest }, where SourceRequest ::= Size | Segment(id). It also carries an IoIntent: Fetch, Prefetch or Announce. One service can serve many segment sources, and a source can answer more than segment reads.
  • Reads resolve to a SourceResult: either Size(u64) or Bytes(BufferHandle). The checked accessors are size() and into_bytes().
  • SegmentIoSession::submit registers a whole batch before any read starts, and returns one IoHandle per request.
    • The session holds fetches and prefetches until they finish.
    • An announcement is withdrawn by dropping its handles.
  • SegmentIoSession::poll_ready reports each fetch's owner once the fetch finishes, including fetches that finished at submit, and drives the session's prefetches. FetchTracker implements this for sessions.
  • SegmentIoService::session() opens one session per scope. RegisteredSource is a source's registration with a service.

Tests cover handles, results checked against what was asked, and FetchTracker.

Checks run:

  • cargo clippy -p vortex-io --all-targets --all-features -- -D warnings
  • cargo +nightly-2026-09-10 fmt -p vortex-io
  • RUSTDOCFLAGS="-D warnings --cfg docsrs" cargo doc --no-deps -p vortex-io --all-features
  • cargo test -p vortex-io --lib segments (4 tests)
  • typos

API Changes

New: vortex_io::segments.

🤖 Generated with Claude Code

https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants