Skip to content

fix(io): return an error for short reads - #8164

Merged
westonpace merged 2 commits into
mainfrom
gatekeeper/fix-5674-1
Aug 4, 2026
Merged

fix(io): return an error for short reads#8164
westonpace merged 2 commits into
mainfrom
gatekeeper/fix-5674-1

Conversation

@lance-gatekeeper

Copy link
Copy Markdown
Contributor

Summary

  • validate physical read lengths before reconstructing coalesced requests
  • return an I/O error with the file, byte range, actual length, and expected length
  • add a regression test for an empty short read

Root cause

FileScheduler assumed every reader result exactly matched its requested physical byte range. When a reader returned fewer bytes, request reconstruction called Bytes::slice with the expected end offset and panicked before reporting which read was incomplete.

Validation

  • cargo fmt --all
  • cargo test -p lance-io test_short_read_returns_io_error
  • cargo clippy --all --tests --benches -- -D warnings
  • cargo test -p lance-io (190 tests passed; 8 unrelated io_uring tests could not initialize in this container because IoUring::new returned EPERM)

Fixes #5674

@github-actions github-actions Bot added A-encoding Encoding, IO, file reader/writer bug Something isn't working labels Aug 3, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve. The shared post-read validation enforces the exact physical-range contract before coalesced and split reconstruction, converting short-read panics into contextual I/O errors while covering both scheduler backends and custom readers.

Comment thread rust/lance-io/src/scheduler.rs Outdated
priority,
self.bypass_backpressure,
);
let file_path = self.reader.path().clone();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be premature optimization but it would be nice if we could avoid the path clone on every IOP. If we move the validation lower we might be able to avoid this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 177c716: moved short-read validation into the physical scheduler paths, avoiding the per-I/O path clone while preserving eager lite submission and contextual I/O errors.

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The revision enforces exact physical-read lengths inside both scheduler backends, eliminating the eager path clone while preserving typed errors and the existing coalescing, ordering, and backpressure contracts.

@Xuanwo
Xuanwo requested a review from lance-community August 4, 2026 08:18
@lance-gatekeeper
lance-gatekeeper Bot removed the request for review from lance-community August 4, 2026 08:18

@Xuanwo Xuanwo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice change, thank you!

@lance-gatekeeper
lance-gatekeeper Bot removed the request for review from lance-community August 4, 2026 08:32
@westonpace
westonpace merged commit fd162ea into main Aug 4, 2026
40 checks passed
@westonpace
westonpace deleted the gatekeeper/fix-5674-1 branch August 4, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-encoding Encoding, IO, file reader/writer bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

thread 'tokio-runtime-worker' panicked at 'range end out of bounds: 8 <= 0'

2 participants