Skip to content

Fix API level 500 string and UTF-8 faults (#405, #406, #407, #408, #412) - #439

Merged
dimensionscape merged 4 commits into
mainfrom
fix/utf8-api500
Nov 13, 2026
Merged

dimensionscape merged 4 commits into
mainfrom
fix/utf8-api500

Conversation

@dimensionscape

@dimensionscape dimensionscape commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Fixes five faults in the API level 500 string and encoding code, each with a regression test, and makes Haxe 5's UTF-8 decoding match Haxe 4's for malformed input. The fixes were first written on the cppia-bugs-1106 branch and are picked from there onto current main.

  • cpp.encoding.Utf8.encode returned a null array for the empty string. It returns an empty one. Fixes Return an empty array from Utf8.encode for an empty string #406.
  • Utf8.encode(codepoint, view) wrote bytes for a surrogate or a value past U+10FFFF, which the UTF-16 encoder refused. Both encoders, and both getByteCount for a code point, now throw for anything that is not a Unicode scalar value, before writing. Fixes Validate Unicode scalar values before native UTF-8 encoding #407.
  • Utf8.encode(String) allocated three bytes per unit for a string that was not all ASCII and only shortened the array afterwards, so Haxe 5's Bytes.ofString kept 3 MB for a megabyte of ASCII with one other character. It counts the bytes first and allocates exactly that. Fixes Avoid retaining three times the encoded payload in Haxe 5 Bytes.ofString #405.
  • String::utf8_str into a View<char> checked for room for the bytes but not the terminator, so it wrote and then threw, and for a UTF-16 string it stopped at the first NUL. It returns false before writing anything, and it converts the string's whole length. Fixes Honor string lengths and terminator capacity in UTF-8 buffer conversion #408.
  • String::create from a View<char16_t> completed a high surrogate at the end of the view with the unit after it. It stays inside the view, and an incomplete pair throws Invalid UTF16. Fixes Keep UTF-16 string construction inside the supplied native view #412.
  • Utf8.decode, which Haxe 5's Bytes.toString and getString use, threw on malformed input, where Haxe 4 gives U+FFFD for each malformed sequence. It gives the same text as Haxe 4 now. Utf8.codepoint, which reads one character, still throws. The nightly workflow's Utf8Malformed failed on it. The native suite's test_decode_malformed_string expected the throw and checks the replacement text now.

The tests make the level 500 calls from native code that compiles at every level, so Haxe 4.3.7 and 5.0.0-preview.1 build them as no-ops and only the Haxe nightly runs the checks. I ran all five, and Utf8Malformed, here with the Haxe nightly (5.0.0-preview.1+3cfeac4, API level 500, MSVC x64). They fail on main and pass with these commits. With Haxe 4.3.7 they build and pass.

@dimensionscape dimensionscape changed the title Fix five API level 500 string and UTF-8 faults (#405, #406, #407, #408, #412) Fix API level 500 string and UTF-8 faults (#405, #406, #407, #408, #412) Nov 13, 2026
encode(String) returned 0 for the empty string, which made a null array
(libhxcpp #406). It returns an empty one.

encode(char32_t, View) chose a byte count by range alone, so a surrogate
went out as three bytes and anything past U+10FFFF as four, where the
UTF-16 encoder refused both (libhxcpp #407). Both encoders, and both
getByteCount for a code point, throw for one that is no scalar value,
before writing anything. A String's lone surrogate still becomes U+FFFD
before it gets there.

The array encode(String) allocated three bytes for each unit of a string
that was not all ASCII and resize only shortened it, so Haxe 5's
Bytes.ofString kept 3 MB for a megabyte of ASCII and one other character
(libhxcpp #405). It counts the bytes first and allocates them exactly.

The fixtures do the level 500 calls in native code that passes below that
level, as Haxe 4 and 5.0.0-preview.1 build against the 430 runtime, and
Utf8EncodeCapacity checks Haxe 4's Bytes.ofString too.
utf8_str into a View<char> checked that the bytes fit but not the
terminator, so a view with room for the bytes alone took them and then
threw, and an empty view threw. It returns false before writing. For a
UTF-16 string it also scanned to the first NUL, a character in a Haxe
string, so the rest was dropped and the length was short. It converts the
string's whole length (libhxcpp #408).

create from a View<char16_t> scanned with Char16Advance, which completes
a high surrogate with the next unit wherever it is, so a pair split by the
end of the view took the unit past it (libhxcpp #412). The scan stops at
the view's end or a terminator, and a pair the view does not complete
throws Invalid UTF16, as a broken pair inside it did.
Haxe 5's Bytes.toString and getString decode with
cpp.encoding.Utf8.decode, which threw for malformed input, where Haxe 4's
go through String::create and give U+FFFD for each malformed sequence
(48fe31a). The nightly workflow's Utf8Malformed failed on it.

decode now gives the same text as Haxe 4: valid input still takes the
strict decoder that counts the UTF-16 units first, and input it refuses
goes to _hx_utf8_to_utf16, the decoder String::create uses. Without smart
strings, where a String keeps UTF-8, valid bytes are still copied as they
are and malformed ones are rewritten by the same decoder, in
_hx_utf8_replace_malformed. Utf8.codepoint, which reads one character,
still throws.

The native suite's test_decode_malformed_string expected the throw. It
checks the replacement text now, which is the same in both string modes.
# Conflicts:
#	Changes.md
@dimensionscape
dimensionscape merged commit 919b735 into main Nov 13, 2026
37 checks passed
@dimensionscape
dimensionscape deleted the fix/utf8-api500 branch November 13, 2026 15:30
@dimensionscape dimensionscape mentioned this pull request Nov 13, 2026
28 of 34 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant