Skip to content

Commit 29f7951

Browse files
committed
gh-160309: Fix StreamReader.read() silently dropping incomplete final character
When a byte stream ends with an incomplete multibyte sequence (e.g. a truncated UTF-8 character), StreamReader.read() would silently discard those bytes: the read loop exited on empty newdata while bytebuffer still held the fragment, and bytebuffer was never decoded. Fix by detecting leftover bytes in bytebuffer when the underlying stream is exhausted and decoding them immediately. The codec's stateless decode function (e.g. utf_8_decode with final=True) then applies the error handler: 'strict' raises UnicodeDecodeError, 'replace' emits the replacement character, and 'ignore' drops the bytes - matching the behaviour of bytes.decode() and io.TextIOWrapper.
1 parent ffe1315 commit 29f7951

1 file changed

Lines changed: 9 additions & 0 deletions

File tree

‎Lib/codecs.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -525,6 +525,15 @@ def read(self, size=-1, chars=-1, firstline=False):
525525
self.charbuffer += newchars
526526
# there was no data available
527527
if not newdata:
528+
# The stream is exhausted. If there are leftover bytes in
529+
# bytebuffer, they form an incomplete character. Decode them
530+
# now so that the error handler is applied (strict raises,
531+
# replace emits the replacement character, ignore drops them).
532+
if self.bytebuffer:
533+
newchars, decodedbytes = self.decode(
534+
self.bytebuffer, self.errors)
535+
self.bytebuffer = self.bytebuffer[decodedbytes:]
536+
self.charbuffer += newchars
528537
break
529538
if chars < 0:
530539
# Return everything we've got

0 commit comments

Comments
 (0)