Repository navigation
ext/mbstring: Don't emit surrogates when encoding to UTF-8 - #23959
Closed
nicolas-grekas wants to merge 1 commit into
Closed
nicolas-grekas wants to merge 1 commit into
nicolas-grekas wants to merge 1 commit into
Conversation
nicolas-grekas
requested review from
alexdowad and
youkidearitai
as code owners
September 28, 2026 08:08
Contributor
|
Thanks for the great catch! This looks right to me. |
youkidearitai
approved these changes
Sep 28, 2026
youkidearitai
left a comment
youkidearitai
left a comment
Contributor
There was a problem hiding this comment.
LGTM, Thanks!
Please add also NEWS entry.
The UTF-8 encoder turned surrogate codepoints into 3-byte sequences like "\xED\xA0\x80", which are not valid UTF-8. Such codepoints come from UCS-2, UCS-4 or HTML-ENTITIES input, or from mb_decode_numericentity(). Since mbstring marks the UTF-8 strings it returns as valid UTF-8, functions that trust that flag, like mb_check_encoding(), mb_scrub() or preg_match(), accepted them too. Surrogates are now replaced by the substitute character, as codepoints above U+10FFFF already are. The internal mode used by mb_strpos() and similar functions still encodes them, so that different surrogates don't match each other.
nicolas-grekas
force-pushed
the
mbstring-utf8-surrogates
branch
from
September 28, 2026 08:33
550f4c4 to
89246d2
Compare
Contributor
Author
done thanks for the suggestion |
Contributor
|
@nicolas-grekas Do you want to go ahead and merge? Or should I merge this for you? |
Contributor
Author
|
Please merge, I'm not sure I can. |
Contributor
|
OK, pulled Thank you very much for the good fix, @nicolas-grekas! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mb_convert_encoding("\0\0\xD8\0", 'UTF-8', 'UCS-4')returns"\xED\xA0\x80", which isn't valid UTF-8. Since 4427b2e (PHP 8.3), mbstring also flags its UTF-8 output as valid, somb_check_encoding(),mb_scrub(),preg_match('//u')and others accept that string while they reject the same bytes written as a literal. The same happens with UCS-2 and HTML-ENTITIES input,mb_convert_variables(),mb_decode_numericentity()andmb_decode_mimeheader().The UTF-8 encoder now replaces surrogates with the substitute character, as it already does for codepoints above U+10FFFF. The internal mode used by
mb_strpos()and friends keeps encoding them, so that different surrogates in a UCS-2 string still don't match each other.Found while working on #23957.