Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions Changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@
libhxcpp 6.0.0 (unreleased)
------------------------------------------------------------

* cpp.encoding.Utf8.decode replaces each malformed UTF-8 sequence with U+FFFD instead of throwing, with smart strings or without. Haxe 5's Bytes.toString and getString decode with it, so they threw for bytes that Haxe 4's, which go through String::create, decode with U+FFFD in the same places. Valid input still takes the decoder that counts first; only malformed input takes the replacing one. Utf8.codepoint still throws
* Fixed String::utf8_str into a view, at API level 500, writing the bytes and then throwing when the view had room for them but not the terminator. It returns false before writing anything. It also stopped a UTF-16 string at its first NUL, which is a character, and reported that length (libhxcpp #408)
* Fixed String::create from a UTF-16 view, at API level 500, completing a high surrogate at the end of the view with the unit after it. The view bounds it: a high surrogate the view does not complete throws Invalid UTF16, as a broken pair inside it did (libhxcpp #412)
* Fixed cpp.encoding.Utf8.encode returning a null array for the empty string, at API level 500 (libhxcpp #406)
* Fixed cpp.encoding.Utf8.encode writing bytes for a code point that is no Unicode scalar value, a surrogate or one past U+10FFFF, which UTF-8 cannot hold and the UTF-16 encoder refused. Both encoders, and both getByteCount, throw for one before writing anything, at API level 500 (libhxcpp #407)
* Fixed Haxe 5's Bytes.ofString keeping an array of three bytes for each unit of a string that was not all ASCII: cpp.encoding.Utf8.encode allocated for the worst case and only shortened it, so a megabyte of ASCII with one other character held 3 MB. It counts the bytes and allocates them exactly (libhxcpp #405)
* Added timedWait to hx::thread::ConditionVariable, which Haxe 5's sys.thread.Condition.timedWait calls: with the Haxe nightly, every program that used Condition failed to compile. It returns true once a signal or broadcast came and false when the time ran out, and it waits out the wakeups MSVC's timed waits give just before the deadline with nothing signalled
* Fixed an exception from blit, setSize or resize under the cppia JIT ending the process instead of reaching the script's catch, on typed arrays and on Array<Dynamic>: their native calls had no exception bridge. The script catches it, as it does interpreted and in compiled code (libhxcpp #404)
* Fixed resize() and cpp.NativeArray.setSize() on an Array<Dynamic> under the cppia JIT evaluating the array expression twice, so a call in it ran twice and the array it returned the second time was the one resized. The JIT evaluates it once (libhxcpp #411)
Expand Down
60 changes: 41 additions & 19 deletions src/String.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -965,21 +965,27 @@ String String::create(const::cpp::marshal::View<char>& buffer)

String String::create(const cpp::marshal::View<char16_t>& buffer)
{
auto start = reinterpret_cast<const char16_t*>(buffer.ptr.ptr);
auto end = start + buffer.length;
auto extra = 0;
auto units = reinterpret_cast<const char16_t*>(buffer.ptr.ptr);
auto length = int64_t{ 0 };

while (start < end) {
if (Char16Advance(start) == false)
// Up to a terminator, inside the view: a high surrogate is completed only
// by a low one the view holds, and Char16Advance would read past its end
while (length < buffer.length && units[length] != 0)
{
auto unit = units[length++];

if (IsUtf16Surrogate(unit))
{
// set extra to 1 so we don't include the null terminating character in the calculated length.
extra = 1;
if (IsUtf16LowSurrogate(unit) || length >= buffer.length || !IsUtf16LowSurrogate(units[length]))
{
hx::Throw(HX_CSTRING("Invalid UTF16"));
}

break;
length++;
}
}

return String::create(buffer.ptr.ptr, buffer.length - (end - start) - extra);
return String::create(buffer.ptr.ptr, length);
}

#endif
Expand Down Expand Up @@ -2592,6 +2598,27 @@ String _hx_utf8_to_utf16(const unsigned char *ptr, int inUtf8Len, bool addHash)

return String(str, char16Count);
}
#elif (HXCPP_API_LEVEL>=500)
// For cpp.encoding.Utf8.decode without smart strings, where a String keeps
// UTF-8: the bytes, with each malformed sequence replaced by U+FFFD as
// _hx_utf8_to_utf16 replaces it, so both give the same text
String _hx_utf8_replace_malformed(const unsigned char *ptr, int inUtf8Len)
{
int bytes = 0;
const unsigned char *u = ptr;
const unsigned char *end = ptr + inUtf8Len;
while(u<end)
bytes += UTF8Bytes(DecodeAdvanceUTF8(u,end));

char *str = hx::NewString(bytes);
char *o = str;
u = ptr;
while(u<end)
UTF8EncodeAdvance(o,DecodeAdvanceUTF8(u,end));
str[bytes] = 0;

return String(str, bytes);
}
#endif


Expand Down Expand Up @@ -2886,16 +2913,10 @@ bool String::utf8_str(::cpp::marshal::View<char> buffer, int* outByteLength) con
#ifdef HX_SMART_STRINGS
if (isUTF16Encoded())
{
// All of it: a NUL in the string is a character, not its end
auto cursor = __w;

while (Char16Advance(cursor,false)) {}

auto calculated = cursor - __w - 1;

cursor = __w;

auto end = cursor + calculated;
auto chars = 0;
auto end = __w + length;
auto chars = 0;

while (cursor < end)
{
Expand Down Expand Up @@ -2926,7 +2947,8 @@ bool String::utf8_str(::cpp::marshal::View<char> buffer, int* outByteLength) con
}
#endif

if (buffer.length < length)
// The terminator too, before anything is written
if (buffer.length < length + 1)
{
return false;
}
Expand Down
109 changes: 78 additions & 31 deletions src/cpp/encoding/Encodings.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@

using namespace cpp::marshal;

#if !defined(HX_SMART_STRINGS)
// In String.cpp, beside _hx_utf8_to_utf16
String _hx_utf8_replace_malformed(const unsigned char *ptr, int inUtf8Len);
#endif

bool cpp::encoding::Ascii::isEncoded(const String& string)
{
if (null() == string)
Expand Down Expand Up @@ -142,6 +147,17 @@ namespace
return 0xFFFD;
}
#endif

// A code point that is not a Unicode scalar value, a surrogate or past
// U+10FFFF, has no UTF-8 or UTF-16 form. The UTF-16 encoder refused them
// and the UTF-8 one wrote invalid bytes
void checkScalar(char32_t codepoint)
{
if ((codepoint >= 0xD800 && codepoint < 0xE000) || codepoint > 0x10FFFF)
{
hx::Throw(HX_CSTRING("Invalid codepoint"));
}
}
}

int cpp::encoding::Utf8::getByteCount(const null&)
Expand All @@ -152,6 +168,8 @@ int cpp::encoding::Utf8::getByteCount(const null&)

int cpp::encoding::Utf8::getByteCount(char32_t codepoint)
{
checkScalar(codepoint);

if (codepoint <= 0x7F)
{
return 1;
Expand Down Expand Up @@ -281,7 +299,8 @@ Array<uint8_t> cpp::encoding::Utf8::encode(const String& string)

if (0 == string.length)
{
return 0;
// An empty array: 0 would be a null one
return Array<uint8_t>(0, 0);
}

if (string.isAsciiEncoded())
Expand All @@ -297,8 +316,17 @@ Array<uint8_t> cpp::encoding::Utf8::encode(const String& string)
}

#if defined(HX_SMART_STRINGS)
// estimate the utf8 length with an upper bound
Array<uint8_t> out(string.length * 3, 0);
// Counted first: an array of three bytes a unit, which resize only made
// shorter, kept that capacity as long as the bytes lived, three times
// what a mostly ASCII string needs (libhxcpp #405)
auto bytes = getByteCount(string);

if (bytes > 0x7fffffff)
{
return hx::Throw(HX_CSTRING("String too long"));
}

Array<uint8_t> out(static_cast<int>(bytes), 0);
View<uint8_t> buffer(out->Pointer(), out->length);

auto units = string.raw_wptr();
Expand All @@ -310,8 +338,6 @@ Array<uint8_t> cpp::encoding::Utf8::encode(const String& string)
k += encode(stringCodepoint(units, string.length, i), buffer.slice(k));
}

out->resize(k);

return out;
#else
return hx::Throw(HX_CSTRING("Unexpected encoding error"));
Expand All @@ -320,6 +346,9 @@ Array<uint8_t> cpp::encoding::Utf8::encode(const String& string)

int cpp::encoding::Utf8::encode(char32_t codepoint, const cpp::marshal::View<uint8_t>& buffer)
{
// Before anything is written
checkScalar(codepoint);

if (codepoint <= 0x7F)
{
buffer[0] = static_cast<uint8_t>(codepoint);
Expand Down Expand Up @@ -383,42 +412,58 @@ String cpp::encoding::Utf8::decode(const cpp::marshal::View<uint8_t>& buffer)
{
return Ascii::decode(buffer);
}
#if defined(HX_SMART_STRINGS)
auto bytes = utf16Units(buffer) * sizeof(char16_t);
auto exact = static_cast<char16_t*>(::hx::InternalNew(bytes + sizeof(char16_t), false));
auto output = View<uint8_t>(reinterpret_cast<uint8_t*>(exact), bytes);
int64_t k = 0;
int64_t i = 0;
if (buffer.length > 0x7fffffff)
{
return hx::Throw(HX_CSTRING("String too long"));
}

while (i < buffer.length)
// Malformed input decodes as Haxe 4's String::create and Bytes.toString
// decode it, each malformed sequence as one U+FFFD, since Haxe 5's
// Bytes.toString and getString come here. The strict decoder below
// throws for one, and is kept for valid input, which it decodes faster
auto ptr = static_cast<const unsigned char*>(buffer.ptr.ptr);
auto len = static_cast<int>(buffer.length);
#if defined(HX_SMART_STRINGS)
try
{
auto p = codepoint(buffer.slice(i));
auto bytes = utf16Units(buffer) * sizeof(char16_t);
auto exact = static_cast<char16_t*>(::hx::InternalNew(bytes + sizeof(char16_t), false));
auto output = View<uint8_t>(reinterpret_cast<uint8_t*>(exact), bytes);
int64_t k = 0;
int64_t i = 0;

i += getByteCount(p);
k += Utf16::encode(p, output.slice(k));
}
while (i < buffer.length)
{
auto p = codepoint(buffer.slice(i));

i += getByteCount(p);
k += Utf16::encode(p, output.slice(k));
}

auto chars = static_cast<int>(k / sizeof(char16_t));
auto chars = static_cast<int>(k / sizeof(char16_t));

((unsigned int*)exact)[-1] |= HX_GC_STRING_CHAR16_T;
exact[chars] = 0;
((unsigned int*)exact)[-1] |= HX_GC_STRING_CHAR16_T;
exact[chars] = 0;

return String(exact, chars);
return String(exact, chars);
}
catch (Dynamic&)
{
return _hx_utf8_to_utf16(ptr, len, false);
}
#else
// The bytes are kept as they are, but malformed input still throws, as
// it does with smart strings. codepoint takes an encoded surrogate
// (ED A0 80 to ED BF BF), which Utf16::encode rejects there, so it is
// rejected here too
for (int64_t i = 0; i < buffer.length;)
// The bytes are kept as they are when they are valid. codepoint takes an
// encoded surrogate (ED A0 80 to ED BF BF), which getByteCount refuses
try
{
auto p = codepoint(buffer.slice(i));

if (0xD800 <= p && p < 0xE000)
for (int64_t i = 0; i < buffer.length;)
{
hx::Throw(HX_CSTRING("Failed to read codepoint"));
i += getByteCount(codepoint(buffer.slice(i)));
}

i += getByteCount(p);
}
catch (Dynamic&)
{
return _hx_utf8_replace_malformed(ptr, len);
}

// +1 for the terminator - the allocation is not zeroed and hxcpp
Expand Down Expand Up @@ -557,6 +602,8 @@ int cpp::encoding::Utf16::getByteCount(const null&)

int cpp::encoding::Utf16::getByteCount(char32_t codepoint)
{
checkScalar(codepoint);

return codepoint <= 0xFFFF ? 2 : 4;
}

Expand Down
14 changes: 12 additions & 2 deletions test/native/tests/encoding/TestUtf8.hx
Original file line number Diff line number Diff line change
Expand Up @@ -157,8 +157,18 @@ class TestUtf8 extends Test {
}

public function test_decode_malformed_string() {
for (hex in ['61626364656667f09d84', '6162636465666768c3', 'e3833061', '616263646566676880', 'eda080', 'c0af']) {
Assert.raises(() -> Utf8.decode(Bytes.ofHex(hex).asView()));
// Each malformed sequence becomes one U+FFFD, as Haxe 4's Bytes.toString
// gives, since Haxe 5's decodes here
var expected = [
'61626364656667f09d84' => 'abcdefg\u{FFFD}',
'6162636465666768c3' => 'abcdefgh\u{FFFD}',
'e3833061' => '\u{FFFD}0a',
'616263646566676880' => 'abcdefgh\u{FFFD}',
'eda080' => '\u{FFFD}\u{FFFD}\u{FFFD}',
'c0af' => '\u{FFFD}\u{FFFD}'
];
for (hex => text in expected) {
Assert.equals(text, Utf8.decode(Bytes.ofHex(hex).asView()));
}
}
}
43 changes: 43 additions & 0 deletions test/regression/StringUtf16ViewBounds/Main.hx
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
// String::create from a UTF-16 view completed a high surrogate at the end of
// the view with the unit past it (openfl/libhxcpp#412). It is API level 500,
// so the checks are native code that passes below it.
@:cppFileCode('
#if (HXCPP_API_LEVEL>=500)
#include <cpp/marshal/View.hpp>
static bool boundedUtf16Prefix() {
char16_t data[3] = {0xD83D, 0xDE00, 0};
try {
String value = String::create(cpp::marshal::View<char16_t>(data, 1));
return value.length <= 1;
} catch (...) { return true; }
}
static bool completeUtf16Pair() {
char16_t data[3] = {0xD83D, 0xDE00, 0};
String value = String::create(cpp::marshal::View<char16_t>(data, 2));
// fromCharCode gives the string either way, smart strings or not
return value == String::fromCharCode(0x1F600);
}
static bool terminatedUtf16View() {
char16_t data[4] = {0x0100, 0, 65, 0};
String value = String::create(cpp::marshal::View<char16_t>(data, 3));
return value == String::fromCharCode(0x0100);
}
#else
static bool boundedUtf16Prefix() { return true; }
static bool completeUtf16Pair() { return true; }
static bool terminatedUtf16View() { return true; }
#endif
')
class Main {
static function main() {
var bounded = true;
var pair = true;
var terminated = true;
bounded = untyped __cpp__("boundedUtf16Prefix()");
pair = untyped __cpp__("completeUtf16Pair()");
terminated = untyped __cpp__("terminatedUtf16View()");
Sys.println("UTF16 construction stays inside the supplied view: " + bounded);
Sys.println("a complete surrogate pair is preserved: " + pair);
Sys.println("a NUL terminator ends native string construction: " + terminated);
}
}
2 changes: 2 additions & 0 deletions test/regression/StringUtf16ViewBounds/build.hxml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
-m Main
--cpp bin
3 changes: 3 additions & 0 deletions test/regression/StringUtf16ViewBounds/stdout.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
UTF16 construction stays inside the supplied view: true
a complete surrogate pair is preserved: true
a NUL terminator ends native string construction: true
Loading
Loading