Repository navigation
gh-160309: Fix StreamReader.read() silently dropping an incomplete final character - #160642
saiyam-jain-dev wants to merge 2 commits into
Conversation
|
All commit authors signed the Contributor License Agreement. |
|
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
BHUVANSH855
left a comment
BHUVANSH855
left a comment
There was a problem hiding this comment.
CLA check is failing, sign it we can't review without that.
29f7951 to
db97211
Compare
|
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
|
The commit author email has been updated to match the signed CLA, and the CLA check is now passing (green ✅). I have also ensured inal=True semantics are applied during EOF decode and added regression test coverage in est_codecs.py. |
… final character When a byte stream ends with an incomplete multibyte sequence (e.g. a truncated UTF-8 character), StreamReader.read() would silently discard those bytes: the read loop exited on empty newdata while bytebuffer still held the fragment, and bytebuffer was never decoded. Fix by detecting leftover bytes in bytebuffer when the underlying stream is exhausted and decoding them immediately. The codec's stateless decode function (e.g. utf_8_decode with final=True) then applies the error handler: 'strict' raises UnicodeDecodeError, 'replace' emits the replacement character, and 'ignore' drops the bytes - matching the behaviour of bytes.decode() and io.TextIOWrapper.
db97211 to
e6fb522
Compare
|
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
Summary
Fixes #160309
codecs.StreamReader.read() silently drops bytes when the stream ends with an incomplete multibyte sequence. The read loop exits on empty
ewdata while �ytebuffer still holds the trailing bytes, and those bytes are never decoded.
Bug demonstration (before fix)
`python
Every other decoding path raises or applies the error handler:
`python
Fix
When the underlying stream is exhausted and �ytebuffer is non-empty, decode the leftover bytes immediately before breaking out of the loop. The codec's stateless decode function applies inal=True semantics, so:
This matches the behaviour of �ytes.decode() and io.TextIOWrapper.
codecs.StreamReader.read()silently drops an incomplete character at the end of the stream #160309