Fix progress bar stalling at ~50% for multi-byte encodings (e.g. utf-16-le)#812
Open
ikatyal2110 wants to merge 1 commit into
Open
Fix progress bar stalling at ~50% for multi-byte encodings (e.g. utf-16-le)#812ikatyal2110 wants to merge 1 commit into
ikatyal2110 wants to merge 1 commit into
Conversation
…6-le UpdateWrapper.__iter__ was using len(line) (character count) to track progress, but the progress bar is initialised with os.path.getsize() (byte count). For utf-16-le each character is 2 bytes, so the bar only ever reached ~50% before the insert completed. Use TextIOWrapper.buffer.tell() when available to report the true number of bytes consumed, falling back to len(line) for non-TextIOWrapper objects. Fixes simonw#439
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When inserting a CSV file with a multi-byte encoding like
utf-16-le, the progress bar would stall at around 50% and then jump to done at the end. This happened becauseUpdateWrapper.__iter__tracked progress usinglen(line)(character count), but the bar is initialised withos.path.getsize()(byte count) — for utf-16-le each character is 2 bytes, so the reported progress was always half the real value.The fix uses
TextIOWrapper.buffer.tell()to read the underlying binary stream position (actual bytes consumed) when available, falling back tolen(line)for non-TextIOWrapperobjects so existing behaviour is unchanged.Fixes #439
Generated by Claude Code
📚 Documentation preview 📚: https://sqlite-utils--812.org.readthedocs.build/en/812/