FSST: true per-row lazy/random-access decode - #392
Merged
Merged
Conversation
FsstEncodingDecoder.decode() no longer decompresses the whole column eagerly. FSST's per-row code range is independent of every other row (unlike Bitpacked/Pco/Zstd, which need a decoded window), so it returns a LazyFsstVarBinArray that decompresses row i only when row i is actually read; getByteLength/forEachByteLength answer straight from the uncompressed-lengths child with no decompression at all. See ADR 0026. Per-row bounds validation (malformed lengths/code offsets) moves from decode-time to access-time, mirroring VarBinArrays' existing "offsets are not scanned at decode time" convention, and is strictly more precise than the old aggregate check it replaces. Adversarial edge-case tests against the new accessors found two raw JDK exceptions escaping getBytes() in violation of the security contract — a truncated escape code with no literal byte after it, and a code byte naming an untrained symbol index — both now caught around the decompress() call and rewrapped as VortexException. Also adds coverage for zero rows, an empty row amid non-empty ones, independent access to a later row when an earlier row is structurally invalid, idempotent repeated access, the exact claimed-length-equals-bound case, a negative code offset, and the Integer.MAX_VALUE-7 code-range overflow guard. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P4ijFsGW1MHEcGiu26vNzi
dfa1
force-pushed
the
fsst-per-row-lazy-decode
branch
from
September 13, 2026 18:20
d369a69 to
fcfe48f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FsstEncodingDecoder.decode()no longer decompresses the whole column eagerly — it returns a newLazyFsstVarBinArraythat decompresses rowi's code range only when rowiis actually read.getByteLength/forEachByteLengthanswer straight from the uncompressed-lengths child, with no decompression at all — the exact waste this closes for filter/projection/length-only workloads.VarBinArrays.checkedLength"don't scan untrusted offsets eagerly" convention, and is strictly more precise than the old aggregate check.Test plan
./mvnw -pl reader -am verify(javadoc + unit tests)FsstEncodingDecoderTestguards updated to trigger via row accessors; newLazinessnested class provesdecode()succeeds on a column with one corrupt row until that row is read, and thatforEachByteLengthnever touches the code-offsets/compressed buffersRustWritesJavaReadsIntegrationTest,RustJavaReaderComparisonIntegrationTest,DictValuesPoolCompressionIntegrationTest(FSST nested undervortex.dict) all pass unchanged against real Rust-written files🤖 Generated with Claude Code
https://claude.ai/code/session_01P4ijFsGW1MHEcGiu26vNzi