Handle UTF-8 JSONL tail windows losslessly - #109
Draft
theamazinghenk wants to merge 1 commit into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Problem and root cause
JsonlAppendStore::next_offsetseeks tolen - 4096for larger streams and then callsread_to_string. That seek position is an arbitrary byte offset, not necessarily a UTF-8 character boundary. If it lands inside a multi-byte character, the read fails withstream did not contain valid UTF-8, so the append is rejected and the stream remains stuck at the same failing window.The code now reads bytes and uses
String::from_utf8_lossy. This is safe for the tail scan because, whenever the window starts after byte zero, the first potentially partial line is already discarded bycomplete_from. A window beginning at byte zero starts on a valid character boundary.Impact
JSONL streams containing non-ASCII text no longer lose an append or become permanently stuck merely because the 4 KiB tail window starts midway through a character. Existing stuck streams recover on their next append without migration.
Validation
Validated against current upstream
mainatc6a5f24bd015fc2acfa423b161cedfdbc47bae12.Validation("append store read error: stream did not contain valid UTF-8")cargo fmt --checkcargo clippy --all-targets -- -D warningscargo build --all-targetscargo test(including doctests)cargo run --example basic_graphNo public API changes and no new dependencies.