Repository navigation
Fix API level 500 string and UTF-8 faults (#405, #406, #407, #408, #412) - #439
Merged
Merged
Conversation
encode(String) returned 0 for the empty string, which made a null array (libhxcpp #406). It returns an empty one. encode(char32_t, View) chose a byte count by range alone, so a surrogate went out as three bytes and anything past U+10FFFF as four, where the UTF-16 encoder refused both (libhxcpp #407). Both encoders, and both getByteCount for a code point, throw for one that is no scalar value, before writing anything. A String's lone surrogate still becomes U+FFFD before it gets there. The array encode(String) allocated three bytes for each unit of a string that was not all ASCII and resize only shortened it, so Haxe 5's Bytes.ofString kept 3 MB for a megabyte of ASCII and one other character (libhxcpp #405). It counts the bytes first and allocates them exactly. The fixtures do the level 500 calls in native code that passes below that level, as Haxe 4 and 5.0.0-preview.1 build against the 430 runtime, and Utf8EncodeCapacity checks Haxe 4's Bytes.ofString too.
utf8_str into a View<char> checked that the bytes fit but not the terminator, so a view with room for the bytes alone took them and then threw, and an empty view threw. It returns false before writing. For a UTF-16 string it also scanned to the first NUL, a character in a Haxe string, so the rest was dropped and the length was short. It converts the string's whole length (libhxcpp #408). create from a View<char16_t> scanned with Char16Advance, which completes a high surrogate with the next unit wherever it is, so a pair split by the end of the view took the unit past it (libhxcpp #412). The scan stops at the view's end or a terminator, and a pair the view does not complete throws Invalid UTF16, as a broken pair inside it did.
Haxe 5's Bytes.toString and getString decode with cpp.encoding.Utf8.decode, which threw for malformed input, where Haxe 4's go through String::create and give U+FFFD for each malformed sequence (48fe31a). The nightly workflow's Utf8Malformed failed on it. decode now gives the same text as Haxe 4: valid input still takes the strict decoder that counts the UTF-16 units first, and input it refuses goes to _hx_utf8_to_utf16, the decoder String::create uses. Without smart strings, where a String keeps UTF-8, valid bytes are still copied as they are and malformed ones are rewritten by the same decoder, in _hx_utf8_replace_malformed. Utf8.codepoint, which reads one character, still throws. The native suite's test_decode_malformed_string expected the throw. It checks the replacement text now, which is the same in both string modes.
dimensionscape
force-pushed
the
fix/utf8-api500
branch
from
November 13, 2026 06:09
222cb3f to
5511123
Compare
# Conflicts: # Changes.md
28 of 34 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes five faults in the API level 500 string and encoding code, each with a regression test, and makes Haxe 5's UTF-8 decoding match Haxe 4's for malformed input. The fixes were first written on the
cppia-bugs-1106branch and are picked from there onto current main.cpp.encoding.Utf8.encodereturned a null array for the empty string. It returns an empty one. Fixes Return an empty array from Utf8.encode for an empty string #406.Utf8.encode(codepoint, view)wrote bytes for a surrogate or a value past U+10FFFF, which the UTF-16 encoder refused. Both encoders, and bothgetByteCountfor a code point, now throw for anything that is not a Unicode scalar value, before writing. Fixes Validate Unicode scalar values before native UTF-8 encoding #407.Utf8.encode(String)allocated three bytes per unit for a string that was not all ASCII and only shortened the array afterwards, so Haxe 5'sBytes.ofStringkept 3 MB for a megabyte of ASCII with one other character. It counts the bytes first and allocates exactly that. Fixes Avoid retaining three times the encoded payload in Haxe 5 Bytes.ofString #405.String::utf8_strinto aView<char>checked for room for the bytes but not the terminator, so it wrote and then threw, and for a UTF-16 string it stopped at the first NUL. It returns false before writing anything, and it converts the string's whole length. Fixes Honor string lengths and terminator capacity in UTF-8 buffer conversion #408.String::createfrom aView<char16_t>completed a high surrogate at the end of the view with the unit after it. It stays inside the view, and an incomplete pair throwsInvalid UTF16. Fixes Keep UTF-16 string construction inside the supplied native view #412.Utf8.decode, which Haxe 5'sBytes.toStringandgetStringuse, threw on malformed input, where Haxe 4 gives U+FFFD for each malformed sequence. It gives the same text as Haxe 4 now.Utf8.codepoint, which reads one character, still throws. The nightly workflow'sUtf8Malformedfailed on it. The native suite'stest_decode_malformed_stringexpected the throw and checks the replacement text now.The tests make the level 500 calls from native code that compiles at every level, so Haxe 4.3.7 and 5.0.0-preview.1 build them as no-ops and only the Haxe nightly runs the checks. I ran all five, and
Utf8Malformed, here with the Haxe nightly (5.0.0-preview.1+3cfeac4, API level 500, MSVC x64). They fail on main and pass with these commits. With Haxe 4.3.7 they build and pass.