From 0981142e49ae660c73bfe52b6be141fb26f74309 Mon Sep 17 00:00:00 2001 From: Chris Speciale Date: Thu, 19 Nov 2026 08:14:28 -0500 Subject: [PATCH 1/7] Keep the caller's GC-free zone in Sys.sleep, command and getch Sys.sleep, Sys.command and Sys.getch entered and left a zone with plain EnterGCFreeZone and ExitGCFreeZone. Called in a zone the caller had entered, as with cpp.vm.Gc.enterGCFreeZone around a blocking call, which docs/ThreadsAndStacks.md suggests, their exit ended the caller's zone, and the caller's own exit then stopped the process with "GCFree Zone mismatch". Mutex, Lock and Semaphore were made safe for this in 603fd03c and c3a9ee48. These three allocate nothing between entering and leaving, so they use AutoGCFreeZone, which keeps a zone the caller entered. CFFI's gc_safe_point was a copy of the old __hxcpp_gc_safe_point, calling PauseForCollect even in a zone, so an extension calling it between gc_enter_blocking and gc_exit_blocking ended the process or waited for a release that never came. It calls __hxcpp_gc_safe_point now. Test: GcFreeZoneLeafNatives. Before, the process stopped at the first Sys.sleep; now Sys.sleep and Sys.command work in a zone, and a thread calling gc_safe_point in one lets 100 collections through, 3 of 3 runs on Windows x64. Fixes #457 --- Changes.md | 1 + src/hx/CFFI.cpp | 6 +- src/hx/libs/std/Sys.cpp | 24 ++++--- test/regression/GcFreeZoneLeafNatives/Main.hx | 63 +++++++++++++++++++ .../GcFreeZoneLeafNatives/build.hxml | 2 + .../GcFreeZoneLeafNatives/stdout.txt | 3 + test/regression/emscripten-skip.txt | 1 + 7 files changed, 88 insertions(+), 12 deletions(-) create mode 100644 test/regression/GcFreeZoneLeafNatives/Main.hx create mode 100644 test/regression/GcFreeZoneLeafNatives/build.hxml create mode 100644 test/regression/GcFreeZoneLeafNatives/stdout.txt diff --git a/Changes.md b/Changes.md index 54466e163..08120b9f1 100644 --- a/Changes.md +++ b/Changes.md @@ -2,6 +2,7 @@ libhxcpp 6.0.0 (unreleased) ------------------------------------------------------------ +* Fixed Sys.sleep, Sys.command and Sys.getch ending the process with "GCFree Zone mismatch" when called inside a GC-free zone the caller had entered, as with cpp.vm.Gc.enterGCFreeZone around a blocking call: they left that zone themselves, so the caller's own exit failed. They keep the caller's zone, as Mutex, Lock and Semaphore do. CFFI's gc_safe_point, between gc_enter_blocking and gc_exit_blocking, no longer pauses as if outside the zone, which ended the process with "Bad Allocation while collecting" or waited for ever (libhxcpp #457) * Fixed a Haxe 4 haxe.atomic.AtomicInt kept in a field reading freed memory after a collection: Haxe 4's AtomicInt on C++ is a cpp.Pointer into a one-element array nothing else holds, and the collector ignored cpp.Pointer fields, so it freed the array (HaxeFoundation/hxcpp#1225; Haxe 5 changed AtomicInt instead, in HaxeFoundation/haxe#12236). A cpp.Pointer field, or a pointer boxed in a Dynamic, now keeps what it points into in the GC heap, as a pointer on the stack does, and pins it under HXCPP_GC_MOVING. Of 10,000 atomics in fields, 9,999 read freed memory after a collection and none do now * Fixed HXCPP_GC_INCREMENTAL failing to compile ("Incremental GC requires regenerated C++") with libhxcpp installed from a release, 6.0.0-rc.2 among them: the release workflow rewrote extraParams.hxml with the version and dropped its other line, the macro that turns on the generated write barriers the incremental collector needs. Only the version lines are rewritten now * Fixed a change of link flags or libraries alone leaving the program as it was: the build tool linked again only when an object or library file had changed, so -D HXCPP_LINK_STACK_SIZE, HXCPP_LINK_NO_MEMORY_GROWTH, HXCPP_LINK_NO_EMBIND or HXCPP_LINK_NODERAWFS on an Emscripten build, which change nothing that is compiled, or a flag or library added to a target in Build.xml, did nothing until the objects changed. The flags, libraries and library paths are now part of what a link is checked against, so each program links once more after updating diff --git a/src/hx/CFFI.cpp b/src/hx/CFFI.cpp index 18584bcd0..9dcbbaeaf 100644 --- a/src/hx/CFFI.cpp +++ b/src/hx/CFFI.cpp @@ -1083,10 +1083,12 @@ bool gc_try_unblocking() } +// The runtime's own safe point, which does nothing in a GC-free zone, as +// between gc_enter_blocking and gc_exit_blocking: pausing there was "Bad +// Allocation while collecting", or waited for a release that never came void gc_safe_point() { - if (hx::gPauseForCollect) - hx::PauseForCollect(); + __hxcpp_gc_safe_point(); } gcroot create_root(value) { return 0; } diff --git a/src/hx/libs/std/Sys.cpp b/src/hx/libs/std/Sys.cpp index 392642266..be948e866 100644 --- a/src/hx/libs/std/Sys.cpp +++ b/src/hx/libs/std/Sys.cpp @@ -189,6 +189,9 @@ void _hx_std_put_env( String e, String v ) Sleep a given number of seconds **/ +// In a guard, which keeps a zone the caller entered, as with +// cpp.vm.Gc.enterGCFreeZone around a blocking call: a plain exit here ended it, +// and the caller's own exit then stopped the process (libhxcpp #457) void _hx_std_sys_sleep( double f ) { // A negative duration sleeps for zero - the double->DWORD conversion on @@ -198,7 +201,7 @@ void _hx_std_sys_sleep( double f ) // and spin-wait loops rely on that to let other threads run. if (!(f > 0)) f = 0; - hx::EnterGCFreeZone(); + hx::AutoGCFreeZone zone; #if defined(NEKO_WINDOWS) // Sleep takes a DWORD of milliseconds, and 0xffffffff is INFINITE. A longer // sleep, infinity included, converted to a short one or none, so it goes in @@ -227,7 +230,6 @@ void _hx_std_sys_sleep( double f ) { if( errno != EINTR ) { - hx::ExitGCFreeZone(); return; } t = tmp; @@ -235,7 +237,6 @@ void _hx_std_sys_sleep( double f ) } while (f > 0); #endif - hx::ExitGCFreeZone(); } /** @@ -392,6 +393,9 @@ bool _hx_std_sys_is64() sys_command : string -> int Run the shell command and return exit code **/ +// In a guard, which keeps a zone the caller entered, as with +// cpp.vm.Gc.enterGCFreeZone around a blocking call: a plain exit here ended it, +// and the caller's own exit then stopped the process (libhxcpp #457) int _hx_std_sys_command( String cmd ) { #if defined(HX_WINRT) || defined(__EMSCRIPTEN__) || defined(EPPC) || defined(IPHONE) || defined(APPLETV) || defined(HX_APPLEWATCH) @@ -402,7 +406,7 @@ int _hx_std_sys_command( String cmd ) #ifdef NEKO_WINDOWS int result = 0; - hx::EnterGCFreeZone(); + hx::AutoGCFreeZone zone; #ifdef HX_SMART_STRINGS if (!cmd.isUTF16Encoded()) @@ -415,10 +419,9 @@ int _hx_std_sys_command( String cmd ) } #else hx::strbuf buf; - hx::EnterGCFreeZone(); + hx::AutoGCFreeZone zone; int result = system(cmd.utf8_str(&buf)); #endif - hx::ExitGCFreeZone(); #if !defined(NEKO_WINDOWS) result = WEXITSTATUS(result) | (WTERMSIG(result) << 8); @@ -1052,14 +1055,16 @@ Array _hx_std_sys_env() sys_getch : bool -> int Read a character from stdin with or without echo **/ +// In a guard, which keeps a zone the caller entered, as with +// cpp.vm.Gc.enterGCFreeZone around a blocking call: a plain exit here ended it, +// and the caller's own exit then stopped the process (libhxcpp #457) int _hx_std_sys_getch( bool b ) { #if defined(HX_WINRT) || defined(__EMSCRIPTEN__) || defined(EPPC) return 0; #elif defined(NEKO_WINDOWS) - hx::EnterGCFreeZone(); + hx::AutoGCFreeZone zone; int result = b?getche():getch(); - hx::ExitGCFreeZone(); return result; #else @@ -1068,7 +1073,7 @@ int _hx_std_sys_getch( bool b ) // terminal on initscr() int c; struct termios term, old; - hx::EnterGCFreeZone(); + hx::AutoGCFreeZone zone; tcgetattr(fileno(stdin), &old); term = old; cfmakeraw(&term); @@ -1076,7 +1081,6 @@ int _hx_std_sys_getch( bool b ) c = getchar(); tcsetattr(fileno(stdin), 0, &old); if( b ) fputc(c,stdout); - hx::ExitGCFreeZone(); return c; # endif } diff --git a/test/regression/GcFreeZoneLeafNatives/Main.hx b/test/regression/GcFreeZoneLeafNatives/Main.hx new file mode 100644 index 000000000..1c3a1ad6f --- /dev/null +++ b/test/regression/GcFreeZoneLeafNatives/Main.hx @@ -0,0 +1,63 @@ +import cpp.vm.Gc; +import haxe.atomic.AtomicInt; +import sys.thread.Thread; + +// libhxcpp #457. +// - Sys.sleep, Sys.command and Sys.getch entered and left a GC-free zone with +// plain calls, so called in a zone the caller had entered, as +// docs/ThreadsAndStacks.md suggests around a blocking call, their exit ended +// the caller's zone, and the caller's own exit stopped the process with +// "GCFree Zone mismatch". +// - CFFI's gc_safe_point paused for a collection even in a zone, between +// gc_enter_blocking and gc_exit_blocking, which ended the process with "Bad +// Allocation while collecting" or waited for ever. +@:cppFileCode(' +extern "C" void gc_enter_blocking(); +extern "C" void gc_exit_blocking(); +extern "C" void gc_safe_point(); +') +class Main { + static var count = new AtomicInt(0); + static var stop = new AtomicInt(0); + + static function main() { + Gc.enterGCFreeZone(); + Sys.sleep(0.01); + Gc.exitGCFreeZone(); + Sys.println("Sys.sleep in a zone: ok"); + + Gc.enterGCFreeZone(); + var code = Sys.command("exit 3"); + Gc.exitGCFreeZone(); + Sys.println("Sys.command in a zone: " + code); + + var done = new sys.thread.Lock(); + Thread.create(() -> { + while (stop.load() == 0) { + untyped __cpp__("gc_enter_blocking()"); + untyped __cpp__("gc_safe_point()"); + untyped __cpp__("gc_exit_blocking()"); + count.add(1); + } + done.release(); + }); + var stalled = false; + for (round in 0...100) { + var junk = [for (i in 0...2000) [i]]; + Gc.run(false); + var before = count.load(); + Sys.sleep(0.005); + if (count.load() == before) { + Sys.sleep(2.0); + if (count.load() == before) { + stalled = true; + break; + } + } + } + stop.store(1); + if (!stalled) + done.wait(); + Sys.println("gc_safe_point in a zone, through 100 collections: " + (stalled ? "stalled" : "ok")); + } +} diff --git a/test/regression/GcFreeZoneLeafNatives/build.hxml b/test/regression/GcFreeZoneLeafNatives/build.hxml new file mode 100644 index 000000000..b484e67e7 --- /dev/null +++ b/test/regression/GcFreeZoneLeafNatives/build.hxml @@ -0,0 +1,2 @@ +-m Main +--cpp bin diff --git a/test/regression/GcFreeZoneLeafNatives/stdout.txt b/test/regression/GcFreeZoneLeafNatives/stdout.txt new file mode 100644 index 000000000..056f5fab1 --- /dev/null +++ b/test/regression/GcFreeZoneLeafNatives/stdout.txt @@ -0,0 +1,3 @@ +Sys.sleep in a zone: ok +Sys.command in a zone: 3 +gc_safe_point in a zone, through 100 collections: ok diff --git a/test/regression/emscripten-skip.txt b/test/regression/emscripten-skip.txt index 6342a8d49..a7846a887 100644 --- a/test/regression/emscripten-skip.txt +++ b/test/regression/emscripten-skip.txt @@ -33,6 +33,7 @@ FinalizerExceptionState: threads (std::thread, which needs a pthreads build) FinalizersAcrossFreeZones: threads GcConcurrentCensus: threads (std::thread, which needs a pthreads build) GcFinalizerPromotion: threads (std::thread, which needs a pthreads build) +GcFreeZoneLeafNatives: threads and processes GcFreeZoneNestedGuard: single-threaded Emscripten builds have no GC-free zones, so a guard enters none GcFreeZoneThreadWaits: threads GcHeapGrows: expects the desktop 20 MB working target; other platforms, Emscripten included, use 8 MB From 21b8c15a56b32e0298c73c761be048dee15e37f2 Mon Sep 17 00:00:00 2001 From: Chris Speciale Date: Thu, 19 Nov 2026 08:15:44 -0500 Subject: [PATCH 2/7] Look up the Windows thread description functions when used ThreadImpl.win32.cpp called GetThreadDescription and SetThreadDescription directly, and it is always linked, since Thread_obj::current() runs in InitAlloc. So every Windows program imported both from KERNEL32.dll at load time, and they only exist from Windows 10 1607: on Windows 8.1, Server 2012 R2 and early Windows 10, the program did not start, HXCPP_WINXP_COMPAT or not. They are looked up once with GetProcAddress. Where they are missing, a thread's name is null and setting it does nothing. UWP builds, which need Windows 10 and cannot use GetModuleHandle, call them directly. Test: WindowsThreadNameImports reads the program's own import table: KERNEL32.dll's entries named both before and neither now, on Windows x64. Fixes #456 --- Changes.md | 1 + src/hx/thread/ThreadImpl.win32.cpp | 44 +++++++++++++++++- .../WindowsThreadNameImports/Main.hx | 46 +++++++++++++++++++ .../WindowsThreadNameImports/build.hxml | 2 + .../WindowsThreadNameImports/stdout.txt | 2 + 5 files changed, 93 insertions(+), 2 deletions(-) create mode 100644 test/regression/WindowsThreadNameImports/Main.hx create mode 100644 test/regression/WindowsThreadNameImports/build.hxml create mode 100644 test/regression/WindowsThreadNameImports/stdout.txt diff --git a/Changes.md b/Changes.md index 08120b9f1..abbcd6a52 100644 --- a/Changes.md +++ b/Changes.md @@ -2,6 +2,7 @@ libhxcpp 6.0.0 (unreleased) ------------------------------------------------------------ +* Fixed Windows programs not starting on Windows 8.1, Server 2012 R2 and Windows 10 before 1607 with a missing entry point, HXCPP_WINXP_COMPAT or not: every program imported GetThreadDescription and SetThreadDescription, which those lack. They are looked up when used now, and where they are missing a thread's name is null and setting it does nothing (libhxcpp #456) * Fixed Sys.sleep, Sys.command and Sys.getch ending the process with "GCFree Zone mismatch" when called inside a GC-free zone the caller had entered, as with cpp.vm.Gc.enterGCFreeZone around a blocking call: they left that zone themselves, so the caller's own exit failed. They keep the caller's zone, as Mutex, Lock and Semaphore do. CFFI's gc_safe_point, between gc_enter_blocking and gc_exit_blocking, no longer pauses as if outside the zone, which ended the process with "Bad Allocation while collecting" or waited for ever (libhxcpp #457) * Fixed a Haxe 4 haxe.atomic.AtomicInt kept in a field reading freed memory after a collection: Haxe 4's AtomicInt on C++ is a cpp.Pointer into a one-element array nothing else holds, and the collector ignored cpp.Pointer fields, so it freed the array (HaxeFoundation/hxcpp#1225; Haxe 5 changed AtomicInt instead, in HaxeFoundation/haxe#12236). A cpp.Pointer field, or a pointer boxed in a Dynamic, now keeps what it points into in the GC heap, as a pointer on the stack does, and pins it under HXCPP_GC_MOVING. Of 10,000 atomics in fields, 9,999 read freed memory after a collection and none do now * Fixed HXCPP_GC_INCREMENTAL failing to compile ("Incremental GC requires regenerated C++") with libhxcpp installed from a release, 6.0.0-rc.2 among them: the release workflow rewrote extraParams.hxml with the version and dropped its other line, the macro that turns on the generated write barriers the incremental collector needs. Only the version lines are rewritten now diff --git a/src/hx/thread/ThreadImpl.win32.cpp b/src/hx/thread/ThreadImpl.win32.cpp index 5ea74d6b0..d777ef6e3 100644 --- a/src/hx/thread/ThreadImpl.win32.cpp +++ b/src/hx/thread/ThreadImpl.win32.cpp @@ -6,12 +6,48 @@ #include "ThreadImpl.hpp" #include +namespace +{ + // Windows 10 1607 and later have these. Imported directly, they stopped a + // program loading on anything older, Windows 8.1 among them, whatever it + // did with threads (libhxcpp #456), so they are looked up once instead + typedef HRESULT (WINAPI *GetThreadDescriptionFn)(HANDLE, PWSTR *); + typedef HRESULT (WINAPI *SetThreadDescriptionFn)(HANDLE, PCWSTR); + + #ifdef HX_WINRT + GetThreadDescriptionFn getThreadDescription() { return GetThreadDescription; } + SetThreadDescriptionFn setThreadDescription() { return SetThreadDescription; } + #else + template F kernelFunction(const char *inName) + { + HMODULE kernel = GetModuleHandleW(L"kernel32.dll"); + return kernel ? reinterpret_cast(reinterpret_cast(GetProcAddress(kernel, inName))) : nullptr; + } + + GetThreadDescriptionFn getThreadDescription() + { + static GetThreadDescriptionFn fn = kernelFunction("GetThreadDescription"); + return fn; + } + + SetThreadDescriptionFn setThreadDescription() + { + static SetThreadDescriptionFn fn = kernelFunction("SetThreadDescription"); + return fn; + } + #endif +} + String hx::thread::ThreadImpl_obj::getName() { + auto get = getThreadDescription(); + if (!get) + return null(); + hx::EnterGCFreeZone(); auto buffer = PWSTR{ nullptr }; - auto result = GetThreadDescription(native->handle, &buffer); + auto result = get(native->handle, &buffer); if (FAILED(result)) { _com_error error(result); @@ -33,10 +69,14 @@ String hx::thread::ThreadImpl_obj::getName() void hx::thread::ThreadImpl_obj::setName(const String& name) { + auto set = setThreadDescription(); + if (!set) + return; + hx::strbuf buffer; hx::EnterGCFreeZone(); - auto result = SetThreadDescription(native->handle, name.wchar_str(&buffer)); + auto result = set(native->handle, name.wchar_str(&buffer)); if (FAILED(result)) { _com_error error(result); diff --git a/test/regression/WindowsThreadNameImports/Main.hx b/test/regression/WindowsThreadNameImports/Main.hx new file mode 100644 index 000000000..531742dda --- /dev/null +++ b/test/regression/WindowsThreadNameImports/Main.hx @@ -0,0 +1,46 @@ +// libhxcpp #456. Every Windows program imported GetThreadDescription and +// SetThreadDescription from KERNEL32.dll at load time, which only Windows 10 +// 1607 and later have, so on Windows 8.1 and older it did not start at all. +// They are looked up when used now. This reads the program's own import table. +@:cppFileCode(' +#ifdef HX_WINDOWS +#include +#include +static bool importsFromKernel32(const char *inName) +{ + unsigned char *base = (unsigned char *)GetModuleHandleW(nullptr); + IMAGE_DOS_HEADER *dos = (IMAGE_DOS_HEADER *)base; + IMAGE_NT_HEADERS *nt = (IMAGE_NT_HEADERS *)(base + dos->e_lfanew); + IMAGE_DATA_DIRECTORY dir = nt->OptionalHeader.DataDirectory[IMAGE_DIRECTORY_ENTRY_IMPORT]; + if (!dir.VirtualAddress) + return false; + for(IMAGE_IMPORT_DESCRIPTOR *d = (IMAGE_IMPORT_DESCRIPTOR *)(base + dir.VirtualAddress); d->Name; d++) + { + if (_stricmp((const char *)(base + d->Name), "KERNEL32.dll")) + continue; + IMAGE_THUNK_DATA *thunk = (IMAGE_THUNK_DATA *)(base + (d->OriginalFirstThunk ? d->OriginalFirstThunk : d->FirstThunk)); + for(; thunk->u1.AddressOfData; thunk++) + { + if (IMAGE_SNAP_BY_ORDINAL(thunk->u1.Ordinal)) + continue; + IMAGE_IMPORT_BY_NAME *byName = (IMAGE_IMPORT_BY_NAME *)(base + thunk->u1.AddressOfData); + if (!strcmp((const char *)byName->Name, inName)) + return true; + } + } + return false; +} +#else +static bool importsFromKernel32(const char *) { return false; } +#endif +') +class Main { + static function main() { + // Makes sure the thread code is linked and used + var name = sys.thread.Thread.current(); + for (f in ["GetThreadDescription", "SetThreadDescription"]) { + var imported:Bool = untyped __cpp__("importsFromKernel32({0}.utf8_str())", f); + Sys.println('imports $f: $imported'); + } + } +} diff --git a/test/regression/WindowsThreadNameImports/build.hxml b/test/regression/WindowsThreadNameImports/build.hxml new file mode 100644 index 000000000..b484e67e7 --- /dev/null +++ b/test/regression/WindowsThreadNameImports/build.hxml @@ -0,0 +1,2 @@ +-m Main +--cpp bin diff --git a/test/regression/WindowsThreadNameImports/stdout.txt b/test/regression/WindowsThreadNameImports/stdout.txt new file mode 100644 index 000000000..710090c05 --- /dev/null +++ b/test/regression/WindowsThreadNameImports/stdout.txt @@ -0,0 +1,2 @@ +imports GetThreadDescription: false +imports SetThreadDescription: false From fc12ab079f9f1d61236bd2b5667b86275c6db79b Mon Sep 17 00:00:00 2001 From: Chris Speciale Date: Thu, 19 Nov 2026 08:17:22 -0500 Subject: [PATCH 3/7] Read TLS slots of 64 and up through TlsGetValue on 32-bit Windows TLSData with FAST set, which the collector's tlsStackContext is, read fs:[0xE10 + slot*4] for any slot on 32-bit Windows. Only the first 64 slots are inline in the thread environment block; the others are in TlsExpansionSlots, so for a slot of 64 or more the read landed on other fields of the block. In a process that had used 64 TLS slots before the runtime started, as a host with many DLLs can, every thread looked unregistered and the first allocation stopped the process. Slots of 64 and up go through TlsGetValue, as on x64. Reading the expansion array directly is not safe either, since a thread that never set such a slot has no array yet. Test: TlsHighSlot, where an MSVC CRT initializer takes TLS slots up to 70 before the runtime starts, and a second thread allocates too. The 32-bit build stopped with "Bad local allocator" before and runs now; the 64-bit build runs either way. Fixes #458 --- Changes.md | 1 + include/hx/Tls.h | 18 +++++----- test/regression/TlsHighSlot/Main.hx | 49 ++++++++++++++++++++++++++ test/regression/TlsHighSlot/build.hxml | 2 ++ test/regression/TlsHighSlot/stdout.txt | 1 + 5 files changed, 61 insertions(+), 10 deletions(-) create mode 100644 test/regression/TlsHighSlot/Main.hx create mode 100644 test/regression/TlsHighSlot/build.hxml create mode 100644 test/regression/TlsHighSlot/stdout.txt diff --git a/Changes.md b/Changes.md index abbcd6a52..4dfbeb8b7 100644 --- a/Changes.md +++ b/Changes.md @@ -2,6 +2,7 @@ libhxcpp 6.0.0 (unreleased) ------------------------------------------------------------ +* Fixed 32-bit Windows programs stopping with "Bad local allocator - requesting memory from unregistered thread!" when the process had used 64 TLS slots before the runtime started, as a host with many DLLs, or a late-loaded hxcpp DLL, can: the collector read its per-thread context from the thread's inline TLS slots, of which there are only 64. Later slots are read with TlsGetValue, as on x64 (libhxcpp #458) * Fixed Windows programs not starting on Windows 8.1, Server 2012 R2 and Windows 10 before 1607 with a missing entry point, HXCPP_WINXP_COMPAT or not: every program imported GetThreadDescription and SetThreadDescription, which those lack. They are looked up when used now, and where they are missing a thread's name is null and setting it does nothing (libhxcpp #456) * Fixed Sys.sleep, Sys.command and Sys.getch ending the process with "GCFree Zone mismatch" when called inside a GC-free zone the caller had entered, as with cpp.vm.Gc.enterGCFreeZone around a blocking call: they left that zone themselves, so the caller's own exit failed. They keep the caller's zone, as Mutex, Lock and Semaphore do. CFFI's gc_safe_point, between gc_enter_blocking and gc_exit_blocking, no longer pauses as if outside the zone, which ended the process with "Bad Allocation while collecting" or waited for ever (libhxcpp #457) * Fixed a Haxe 4 haxe.atomic.AtomicInt kept in a field reading freed memory after a collection: Haxe 4's AtomicInt on C++ is a cpp.Pointer into a one-element array nothing else holds, and the collector ignored cpp.Pointer fields, so it freed the array (HaxeFoundation/hxcpp#1225; Haxe 5 changed AtomicInt instead, in HaxeFoundation/haxe#12236). A cpp.Pointer field, or a pointer boxed in a Dynamic, now keeps what it points into in the GC heap, as a pointer on the stack does, and pins it under HXCPP_GC_MOVING. Of 10,000 atomics in fields, 9,999 read freed memory after a collection and none do now diff --git a/include/hx/Tls.h b/include/hx/Tls.h index db1409a40..76764bdd3 100755 --- a/include/hx/Tls.h +++ b/include/hx/Tls.h @@ -44,10 +44,12 @@ #ifdef HXCPP_M64 mFastOffset = mSlot*sizeof(void *) + 0x1480; #else - if (FAST || mSlot < kMaxInlineSlots) - mFastOffset = mSlot*sizeof(void *) + 0xE10; - else - mFastOffset = mSlot - kMaxInlineSlots; + // Only the first 64 slots are in the thread's environment block. The + // rest are in an array of their own, which a thread that never set + // one does not have yet, so they go through TlsGetValue, as on x64. + // Reading every slot inline, as FAST once did, read other fields of + // the block once the process had used 64 slots (libhxcpp #458) + mFastOffset = mSlot < kMaxInlineSlots ? mSlot*sizeof(void *) + 0xE10 : 0; #endif } inline DATA *operator=(DATA *inData) @@ -59,13 +61,9 @@ inline operator DATA *() { #if !defined(HXCPP_M64) && (_MSC_VER >= 1400) - const size_t kTibExtraTlsOffset = 0xF94; - - if (FAST || mSlot < kMaxInlineSlots) + if (mSlot < kMaxInlineSlots) return (DATA *)__readfsdword(mFastOffset); - - DATA **extra = (DATA **)(__readfsdword(kTibExtraTlsOffset)); - return extra[mFastOffset]; + return (DATA *)TlsGetValue(mSlot); #elif (_MSC_VER >= 1400) & !defined(HXCPP_DEBUG) && !defined(HXCPP_ARM64)// 64 bit version... if (mSlot < 64) return (DATA *)__readgsqword(mFastOffset); diff --git a/test/regression/TlsHighSlot/Main.hx b/test/regression/TlsHighSlot/Main.hx new file mode 100644 index 000000000..5ece52d79 --- /dev/null +++ b/test/regression/TlsHighSlot/Main.hx @@ -0,0 +1,49 @@ +import sys.thread.Lock; +import sys.thread.Thread; + +// libhxcpp #458. On 32-bit Windows the collector's per-thread context was read +// from the thread environment block's inline TLS slots whatever its slot, but +// only the first 64 are inline. In a process that had used 64 slots before the +// runtime started, as a host with many DLLs can, every thread looked +// unregistered and the first allocation stopped the process. Here, with MSVC, a +// CRT initializer that runs before the runtime takes TLS slots up to 70. +@:cppFileCode(' +#if defined(_MSC_VER) && !defined(HX_WINRT) +#include +#include +extern "C" void __cdecl hxUseUpInlineTlsSlots(void) +{ + for (int i = 0; i < 128; i++) + { + DWORD last = TlsAlloc(); + if (last == TLS_OUT_OF_INDEXES || last >= 70) + break; + } +} +#pragma section(".CRT$XCT", read) +extern "C" __declspec(allocate(".CRT$XCT")) void (__cdecl *hxUseUpInlineTlsSlotsInit)(void) = hxUseUpInlineTlsSlots; +#ifdef _WIN64 +#pragma comment(linker, "/include:hxUseUpInlineTlsSlotsInit") +#else +#pragma comment(linker, "/include:_hxUseUpInlineTlsSlotsInit") +#endif +#endif +') +class Main { + static function strings(n:Int) + return [for (i in 0...n) Std.string(i)].length; + + static function main() { + var made = strings(100000); + var done = new Lock(); + var madeOnThread = 0; + Thread.create(() -> { + madeOnThread = strings(100000); + cpp.vm.Gc.run(true); + done.release(); + }); + done.wait(); + cpp.vm.Gc.run(true); + Sys.println('made $made strings, and $madeOnThread on another thread, and collected'); + } +} diff --git a/test/regression/TlsHighSlot/build.hxml b/test/regression/TlsHighSlot/build.hxml new file mode 100644 index 000000000..b484e67e7 --- /dev/null +++ b/test/regression/TlsHighSlot/build.hxml @@ -0,0 +1,2 @@ +-m Main +--cpp bin diff --git a/test/regression/TlsHighSlot/stdout.txt b/test/regression/TlsHighSlot/stdout.txt new file mode 100644 index 000000000..dc0535d1b --- /dev/null +++ b/test/regression/TlsHighSlot/stdout.txt @@ -0,0 +1 @@ +made 100000 strings, and 100000 on another thread, and collected From 2354e97f01a363dca12b4e6055d21e9bff36852b Mon Sep 17 00:00:00 2001 From: Chris Speciale Date: Thu, 19 Nov 2026 08:19:54 -0500 Subject: [PATCH 4/7] Wait for ever on tryAcquire(-1) only Since #420, CountingSemaphore::tryAcquire waited with no limit for any negative timeout, so that Haxe 5.0.0-preview.1's Lock.wait(), which passes -1 when given no timeout, would block. But a timeout computed as tryAcquire(deadline - now) goes negative once the deadline has passed, and then waited for ever, where Haxe documents tryAcquire as giving up when the timeout runs out. The Haxe nightly's Lock.wait calls acquire() when given no timeout, so only preview.1 needs the special case. Exactly -1 still waits with no limit, on every backend. Any other negative timeout returns at once, as 0 and NaN do. Test: ThreadWaitTimeouts checks -1 and -0.5 apart now. tryAcquire(-0.5) on an empty semaphore waited until the watchdog stopped the test before, and returns false at once now, on Windows x64. Fixes #461 --- Changes.md | 1 + src/hx/thread/CountingSemaphore.apple.cpp | 8 +++++--- src/hx/thread/CountingSemaphore.posix.cpp | 8 +++++--- src/hx/thread/CountingSemaphore.singlethreaded.cpp | 8 +++++--- src/hx/thread/CountingSemaphore.win32.cpp | 8 +++++--- test/regression/ThreadWaitTimeouts/Main.hx | 8 +++++--- test/regression/ThreadWaitTimeouts/stdout.txt | 1 + 7 files changed, 27 insertions(+), 15 deletions(-) diff --git a/Changes.md b/Changes.md index 4dfbeb8b7..0ed50938e 100644 --- a/Changes.md +++ b/Changes.md @@ -2,6 +2,7 @@ libhxcpp 6.0.0 (unreleased) ------------------------------------------------------------ +* Fixed Semaphore.tryAcquire with a negative timeout waiting for ever, as tryAcquire(deadline - now) does once the deadline has passed: since the fix for Haxe 5's Lock.wait() (libhxcpp #420), any negative timeout meant no limit. Only -1, which Haxe 5.0.0-preview.1's Lock.wait() passes for no timeout, waits with no limit now, and any other negative timeout returns at once, as 0 does (libhxcpp #461) * Fixed 32-bit Windows programs stopping with "Bad local allocator - requesting memory from unregistered thread!" when the process had used 64 TLS slots before the runtime started, as a host with many DLLs, or a late-loaded hxcpp DLL, can: the collector read its per-thread context from the thread's inline TLS slots, of which there are only 64. Later slots are read with TlsGetValue, as on x64 (libhxcpp #458) * Fixed Windows programs not starting on Windows 8.1, Server 2012 R2 and Windows 10 before 1607 with a missing entry point, HXCPP_WINXP_COMPAT or not: every program imported GetThreadDescription and SetThreadDescription, which those lack. They are looked up when used now, and where they are missing a thread's name is null and setting it does nothing (libhxcpp #456) * Fixed Sys.sleep, Sys.command and Sys.getch ending the process with "GCFree Zone mismatch" when called inside a GC-free zone the caller had entered, as with cpp.vm.Gc.enterGCFreeZone around a blocking call: they left that zone themselves, so the caller's own exit failed. They keep the caller's zone, as Mutex, Lock and Semaphore do. CFFI's gc_safe_point, between gc_enter_blocking and gc_exit_blocking, no longer pauses as if outside the zone, which ended the process with "Bad Allocation while collecting" or waited for ever (libhxcpp #457) diff --git a/src/hx/thread/CountingSemaphore.apple.cpp b/src/hx/thread/CountingSemaphore.apple.cpp index 25564db0e..8a448addf 100644 --- a/src/hx/thread/CountingSemaphore.apple.cpp +++ b/src/hx/thread/CountingSemaphore.apple.cpp @@ -54,9 +54,11 @@ void hx::thread::CountingSemaphore_obj::release() bool hx::thread::CountingSemaphore_obj::tryAcquire(Null timeout) { - // A negative timeout waits with no limit, as __hxcpp_lock_wait does: Haxe 5's - // Lock.wait() passes -1 when it is given no timeout. NaN still returns at once. - if (timeout.Default(0) < 0) + // -1 waits with no limit: Haxe 5.0.0-preview.1's Lock.wait() passes it when + // given no timeout. Any other negative timeout has run out already, as in + // tryAcquire(deadline - now), and returns at once like 0, as does NaN; it + // waited for ever before (libhxcpp #461) + if (timeout.Default(0) == -1) { acquire(); return true; diff --git a/src/hx/thread/CountingSemaphore.posix.cpp b/src/hx/thread/CountingSemaphore.posix.cpp index 8cbb3353d..aa0559ade 100644 --- a/src/hx/thread/CountingSemaphore.posix.cpp +++ b/src/hx/thread/CountingSemaphore.posix.cpp @@ -75,9 +75,11 @@ void hx::thread::CountingSemaphore_obj::release() bool hx::thread::CountingSemaphore_obj::tryAcquire(Null timeout) { - // A negative timeout waits with no limit, as __hxcpp_lock_wait does: Haxe 5's - // Lock.wait() passes -1 when it is given no timeout. NaN still returns at once. - if (timeout.Default(0) < 0) + // -1 waits with no limit: Haxe 5.0.0-preview.1's Lock.wait() passes it when + // given no timeout. Any other negative timeout has run out already, as in + // tryAcquire(deadline - now), and returns at once like 0, as does NaN; it + // waited for ever before (libhxcpp #461) + if (timeout.Default(0) == -1) { acquire(); return true; diff --git a/src/hx/thread/CountingSemaphore.singlethreaded.cpp b/src/hx/thread/CountingSemaphore.singlethreaded.cpp index 2b5430f56..2598ec5a6 100644 --- a/src/hx/thread/CountingSemaphore.singlethreaded.cpp +++ b/src/hx/thread/CountingSemaphore.singlethreaded.cpp @@ -54,9 +54,11 @@ void hx::thread::CountingSemaphore_obj::release() bool hx::thread::CountingSemaphore_obj::tryAcquire(Null timeout) { - // A negative timeout waits with no limit, as __hxcpp_lock_wait does: Haxe 5's - // Lock.wait() passes -1 when it is given no timeout. NaN still returns at once. - if (timeout.Default(0) < 0) + // -1 waits with no limit: Haxe 5.0.0-preview.1's Lock.wait() passes it when + // given no timeout. Any other negative timeout has run out already, as in + // tryAcquire(deadline - now), and returns at once like 0, as does NaN; it + // waited for ever before (libhxcpp #461) + if (timeout.Default(0) == -1) { acquire(); return true; diff --git a/src/hx/thread/CountingSemaphore.win32.cpp b/src/hx/thread/CountingSemaphore.win32.cpp index f9d9ab8e3..5389419cc 100644 --- a/src/hx/thread/CountingSemaphore.win32.cpp +++ b/src/hx/thread/CountingSemaphore.win32.cpp @@ -59,9 +59,11 @@ void hx::thread::CountingSemaphore_obj::release() bool hx::thread::CountingSemaphore_obj::tryAcquire(Null timeout) { - // A negative timeout waits with no limit, as __hxcpp_lock_wait does: Haxe 5's - // Lock.wait() passes -1 when it is given no timeout. NaN still returns at once. - if (timeout.Default(0) < 0) + // -1 waits with no limit: Haxe 5.0.0-preview.1's Lock.wait() passes it when + // given no timeout. Any other negative timeout has run out already, as in + // tryAcquire(deadline - now), and returns at once like 0, as does NaN; it + // waited for ever before (libhxcpp #461) + if (timeout.Default(0) == -1) { acquire(); return true; diff --git a/test/regression/ThreadWaitTimeouts/Main.hx b/test/regression/ThreadWaitTimeouts/Main.hx index 41f2a6fee..417ff0535 100644 --- a/test/regression/ThreadWaitTimeouts/Main.hx +++ b/test/regression/ThreadWaitTimeouts/Main.hx @@ -40,11 +40,13 @@ class Main { static function main() { untyped __cpp__("startWatchdog(60)"); var empty = new Semaphore(0); - var full = new Semaphore(2); + var full = new Semaphore(3); - // A negative timeout waits with no limit, which Haxe 5's Lock.wait() relies on + // -1 waits with no limit, which Haxe 5.0.0-preview.1's Lock.wait() relies on releaseLater(() -> empty.release()); - check("semaphore negative", attempt(() -> empty.tryAcquire(-0.5) && full.tryAcquire(-0.5))); + check("semaphore minus one", attempt(() -> empty.tryAcquire(-1) && full.tryAcquire(-1))); + // Any other negative timeout has run out, and returns at once (libhxcpp #461) + check("semaphore negative", attempt(() -> !empty.tryAcquire(-0.5) && full.tryAcquire(-0.5))); check("semaphore nan", attempt(() -> !empty.tryAcquire(Math.NaN) && full.tryAcquire(Math.NaN))); var huge = true; diff --git a/test/regression/ThreadWaitTimeouts/stdout.txt b/test/regression/ThreadWaitTimeouts/stdout.txt index 75619baa8..f7ba2c080 100644 --- a/test/regression/ThreadWaitTimeouts/stdout.txt +++ b/test/regression/ThreadWaitTimeouts/stdout.txt @@ -1,3 +1,4 @@ +semaphore minus one ok semaphore negative ok semaphore nan ok semaphore huge ok From 76935d26143bdf2e11b2eae774bf3d935dcbe120 Mon Sep 17 00:00:00 2001 From: Chris Speciale Date: Thu, 19 Nov 2026 08:31:41 -0500 Subject: [PATCH 5/7] Look up cppia's shared maps without inserting hx::Object::_hx_getInterface read a script class's interfaceScriptTables with std::map::operator[], so Std.isOfType on a script object, or an interface cast, inserted an entry for each host interface the class did not implement, from whichever thread asked, with no lock, while other threads searched the same map. Threads asking about the same classes hung or crashed. HaxeNativeClass::findClass, hxObject, link's super class lookup and HaxeNativeInterface::findInterface inserted the names they missed into the host's class maps the same way, while loading a module on another thread walked them in link(), so loads failed with "No base defined for non-interface" or hung. The maps are filled when classes link or the host starts and only read after, with find(). link(), which sets the shared super classes on every load, takes a lock, and the module id counter is atomic. Test: CppiaThreadedLookups, 8 threads asking Std.isOfType about the same 2000 script objects, then 8 threads loading one module at once, with a watchdog. Before, 3 of 4 runs hung; now 5 of 5 pass, on Windows x64. Fixes #466 --- Changes.md | 1 + src/hx/cppia/CppiaClasses.cpp | 7 +- src/hx/cppia/CppiaModule.cpp | 3 +- src/hx/cppia/HaxeNative.cpp | 21 ++++- test/regression/CppiaThreadedLookups/Gen.hx | 13 +++ test/regression/CppiaThreadedLookups/Main.hx | 94 +++++++++++++++++++ .../regression/CppiaThreadedLookups/Script.hx | 8 ++ .../CppiaThreadedLookups/build.hxml | 8 ++ .../CppiaThreadedLookups/stdout.txt | 2 + test/regression/emscripten-skip.txt | 1 + 10 files changed, 151 insertions(+), 7 deletions(-) create mode 100644 test/regression/CppiaThreadedLookups/Gen.hx create mode 100644 test/regression/CppiaThreadedLookups/Main.hx create mode 100644 test/regression/CppiaThreadedLookups/Script.hx create mode 100644 test/regression/CppiaThreadedLookups/build.hxml create mode 100644 test/regression/CppiaThreadedLookups/stdout.txt diff --git a/Changes.md b/Changes.md index 0ed50938e..bbc865942 100644 --- a/Changes.md +++ b/Changes.md @@ -2,6 +2,7 @@ libhxcpp 6.0.0 (unreleased) ------------------------------------------------------------ +* Fixed cppia hanging or crashing when several threads asked Std.isOfType about script objects, or cast them to host interfaces, at once: each question about an interface a class did not implement inserted into that class's table, unlocked, while other threads searched it. Loading modules on several threads at once could likewise fail with "No base defined for non-interface", or hang, as each load inserted the host class names it missed into a map the others walked. Both are only read after they are filled now (libhxcpp #466) * Fixed Semaphore.tryAcquire with a negative timeout waiting for ever, as tryAcquire(deadline - now) does once the deadline has passed: since the fix for Haxe 5's Lock.wait() (libhxcpp #420), any negative timeout meant no limit. Only -1, which Haxe 5.0.0-preview.1's Lock.wait() passes for no timeout, waits with no limit now, and any other negative timeout returns at once, as 0 does (libhxcpp #461) * Fixed 32-bit Windows programs stopping with "Bad local allocator - requesting memory from unregistered thread!" when the process had used 64 TLS slots before the runtime started, as a host with many DLLs, or a late-loaded hxcpp DLL, can: the collector read its per-thread context from the thread's inline TLS slots, of which there are only 64. Later slots are read with TlsGetValue, as on x64 (libhxcpp #458) * Fixed Windows programs not starting on Windows 8.1, Server 2012 R2 and Windows 10 before 1607 with a missing entry point, HXCPP_WINXP_COMPAT or not: every program imported GetThreadDescription and SetThreadDescription, which those lack. They are looked up when used now, and where they are missing a thread's name is null and setting it does nothing (libhxcpp #456) diff --git a/src/hx/cppia/CppiaClasses.cpp b/src/hx/cppia/CppiaClasses.cpp index 3b30bbfdc..6f50259c1 100644 --- a/src/hx/cppia/CppiaClasses.cpp +++ b/src/hx/cppia/CppiaClasses.cpp @@ -2246,8 +2246,11 @@ void *hx::Object::_hx_getInterface(int inId) if (!vtable) return 0; CppiaClassInfo *info = ((CppiaClassInfo *)vtable[-1]); - void *result = info->interfaceScriptTables[inId]; - return info->interfaceScriptTables[inId]; + // Filled when the class links. operator[] inserted an entry for each + // interface asked about that the class does not implement, from whichever + // thread asked, while others searched the same map (libhxcpp #466) + std::map::iterator it = info->interfaceScriptTables.find(inId); + return it==info->interfaceScriptTables.end() ? 0 : it->second; } diff --git a/src/hx/cppia/CppiaModule.cpp b/src/hx/cppia/CppiaModule.cpp index f8be08f8f..ce277288c 100644 --- a/src/hx/cppia/CppiaModule.cpp +++ b/src/hx/cppia/CppiaModule.cpp @@ -12,7 +12,8 @@ namespace hx { -static int sScriptId = 0; +// Modules load on any thread +static std::atomic sScriptId(0); static std::atomic sLiveModules(0); diff --git a/src/hx/cppia/HaxeNative.cpp b/src/hx/cppia/HaxeNative.cpp index 9bd24ac66..ca8feae7a 100644 --- a/src/hx/cppia/HaxeNative.cpp +++ b/src/hx/cppia/HaxeNative.cpp @@ -2,6 +2,7 @@ #include #include "Cppia.h" #include +#include namespace hx @@ -101,18 +102,27 @@ ScriptNamedFunction HaxeNativeClass::findStaticFunction(String inName) +// The maps are filled as the host starts, and only read after, by any thread +// loading or running a script: a lookup with operator[] inserted the names it +// missed, under threads walking or searching the same map (libhxcpp #466) HaxeNativeClass *HaxeNativeClass::findClass(const std::string &inName) { - return sScriptRegistered ? (*sScriptRegistered)[inName] : 0; + if (!sScriptRegistered) + return 0; + ScriptRegisteredMap::iterator it = sScriptRegistered->find(inName); + return it==sScriptRegistered->end() ? 0 : it->second; } HaxeNativeClass *HaxeNativeClass::hxObject() { - return sScriptRegistered ? (*sScriptRegistered)["hx.Object"] : 0; + return findClass("hx.Object"); } void HaxeNativeClass::link() { + // Each load links, and the super classes it sets are shared by every module + static std::mutex linkLock; + std::lock_guard lock(linkLock); HaxeNativeClass *hxObj = hxObject(); for(ScriptRegisteredMap::iterator i = sScriptRegistered->begin(); i!=sScriptRegistered->end();++i) { @@ -131,7 +141,7 @@ void HaxeNativeClass::link() hx::Class superClass = cls->GetSuper(); if (superClass.mPtr) { - HaxeNativeClass *superRef = (*sScriptRegistered)[superClass.mPtr->mName.utf8_str()]; + HaxeNativeClass *superRef = findClass(superClass.mPtr->mName.utf8_str()); if (superRef) { DBGLOG("registered %s\n",superClass.mPtr->mName.utf8_str()); @@ -282,7 +292,10 @@ void ScriptableRegisterInterface( String inName, HaxeNativeInterface *HaxeNativeInterface::findInterface(const std::string &inName) { - return sScriptRegisteredInterface ? (*sScriptRegisteredInterface)[inName] : 0; + if (!sScriptRegisteredInterface) + return 0; + HaxeNativeIntefaceMap::iterator it = sScriptRegisteredInterface->find(inName); + return it==sScriptRegisteredInterface->end() ? 0 : it->second; } diff --git a/test/regression/CppiaThreadedLookups/Gen.hx b/test/regression/CppiaThreadedLookups/Gen.hx new file mode 100644 index 000000000..12af6aa93 --- /dev/null +++ b/test/regression/CppiaThreadedLookups/Gen.hx @@ -0,0 +1,13 @@ +import haxe.macro.Context; +import haxe.macro.Expr; + +// Defines the script's classes C0 to C, each with a constructor +class Gen { + public static function classes(n:Int) { + var pos = Context.currentPos(); + for (i in 0...n) { + var ctor:Field = {name: "new", access: [APublic], kind: FFun({args: [], expr: macro {}}), pos: pos}; + Context.defineType({pack: [], name: 'C$i', pos: pos, kind: TDClass(), fields: [ctor], meta: [{name: ":keep", pos: pos}]}); + } + } +} diff --git a/test/regression/CppiaThreadedLookups/Main.hx b/test/regression/CppiaThreadedLookups/Main.hx new file mode 100644 index 000000000..887b8be2e --- /dev/null +++ b/test/regression/CppiaThreadedLookups/Main.hx @@ -0,0 +1,94 @@ +import haxe.io.Path; +import sys.thread.Lock; +import sys.thread.Mutex; +import sys.thread.Thread; + +interface I0 {} +interface I1 {} +interface I2 {} +interface I3 {} +interface I4 {} +interface I5 {} +interface I6 {} +interface I7 {} + +// libhxcpp #466. Two lookups any thread can make inserted into maps that other +// threads were searching at the same time: +// - Std.isOfType(scriptObject, HostInterface) inserted an entry into the +// script class's interface table for each interface it did not implement, +// which hung or crashed threads asking about the same classes. +// - Loading a module looked up host classes with operator[], inserting what it +// missed, while other loads walked the same map, which failed loads with "No +// base defined for non-interface" or hung. +// A native watchdog, which the collector does not know, ends a hang. +@:cppFileCode(' +#include +#include +#include +#include +#include +static std::atomic gStage(0); +static void startWatchdog() +{ + std::thread([]{ + for(int i=0; i<120 && gStage<2; i++) + std::this_thread::sleep_for(std::chrono::seconds(1)); + if (gStage<2) + { + printf("hung at stage %d\\n", (int)gStage); + fflush(stdout); + std::_Exit(1); + } + }).detach(); +} +') +class Main { + static function main() { + untyped __cpp__("startWatchdog()"); + var threads = 8; + var script = sys.io.File.getBytes(Path.join([Path.directory(Sys.programPath()), "script.cppia"])); + var module = cpp.cppia.Module.fromData(script.getData()); + module.boot(); + module.run(); + var objects:Array = Reflect.field(module.resolveClass("Script"), "objects"); + var ifaces:Array = [I0, I1, I2, I3, I4, I5, I6, I7]; + + var mutex = new Mutex(); + var done = new Lock(); + var wrong = 0; + for (t in 0...threads) + Thread.create(() -> { + var mine = 0; + for (round in 0...4) + for (o in objects) + for (k in 0...ifaces.length) + if (Std.isOfType(o, ifaces[(k + t) % ifaces.length])) + mine++; + mutex.acquire(); + wrong += mine; + mutex.release(); + done.release(); + }); + for (_ in 0...threads) + done.wait(); + untyped __cpp__("gStage = 1"); + Sys.println('${objects.length} script objects asked about ${ifaces.length} host interfaces on $threads threads: $wrong wrong'); + + var failed = 0; + for (t in 0...threads) + Thread.create(() -> { + try { + cpp.cppia.Module.fromData(script.getData()); + } catch (e:Dynamic) { + mutex.acquire(); + failed++; + mutex.release(); + } + done.release(); + }); + for (_ in 0...threads) + done.wait(); + untyped __cpp__("gStage = 2"); + Sys.println('the module loaded on $threads threads at once: $failed failed'); + } +} diff --git a/test/regression/CppiaThreadedLookups/Script.hx b/test/regression/CppiaThreadedLookups/Script.hx new file mode 100644 index 000000000..4e3c05ca3 --- /dev/null +++ b/test/regression/CppiaThreadedLookups/Script.hx @@ -0,0 +1,8 @@ +class Script { + public static var objects:Array = []; + + static function main() { + for (i in 0...2000) + objects.push(Type.createInstance(Type.resolveClass('C$i'), [])); + } +} diff --git a/test/regression/CppiaThreadedLookups/build.hxml b/test/regression/CppiaThreadedLookups/build.hxml new file mode 100644 index 000000000..c95a3dea9 --- /dev/null +++ b/test/regression/CppiaThreadedLookups/build.hxml @@ -0,0 +1,8 @@ +--main Script +--macro Gen.classes(2000) +-cppia bin/script.cppia +--next +--main Main +-cpp bin +-D scriptable +--dce no diff --git a/test/regression/CppiaThreadedLookups/stdout.txt b/test/regression/CppiaThreadedLookups/stdout.txt new file mode 100644 index 000000000..6859eeca7 --- /dev/null +++ b/test/regression/CppiaThreadedLookups/stdout.txt @@ -0,0 +1,2 @@ +2000 script objects asked about 8 host interfaces on 8 threads: 0 wrong +the module loaded on 8 threads at once: 0 failed diff --git a/test/regression/emscripten-skip.txt b/test/regression/emscripten-skip.txt index a7846a887..32dc9a43a 100644 --- a/test/regression/emscripten-skip.txt +++ b/test/regression/emscripten-skip.txt @@ -26,6 +26,7 @@ CppiaModuleFree: threads CppiaModuleRetained: threads CppiaNullDynamicArray: its host is the first --next section, which -D emscripten does not reach, and it runs the script under the JIT too, and Emscripten has no cppia JIT CppiaStackSize: threads (std::thread, which needs a pthreads build) +CppiaThreadedLookups: threads DateTimezoneCache: Emscripten takes the local zone from JavaScript and ignores TZ ExitWithBusyWorker: processes and threads FileFinalizerFlush: threads (std::thread, which needs a pthreads build) From 34d2147ca3d6a1b457287d35109d56f0727bbd60 Mon Sep 17 00:00:00 2001 From: Chris Speciale Date: Thu, 19 Nov 2026 08:34:43 -0500 Subject: [PATCH 6/7] Keep what a cpp.Pointer field points into wherever it points e1f6ac06 made a cpp.Pointer field, and a pointer boxed in a Dynamic, keep what it points into, through MarkConservativePointer. That went through the stack scan, which accepts a pointer only within sgCheckInternalOffset of an allocation's start: 64 bytes, or 32 on 32-bit. Right for a stack word, which is usually also held from the start somewhere, but a field may be all that holds it, and a pointer from cpp.Pointer.arrayElem(a, 100), 400 bytes in, kept nothing. MarkConservative's body takes the reach as a parameter now, and passes it to GetMemType, GetEnclosingAllocType and GetEnclosingNurseryType. A stack word keeps sgCheckInternalOffset; MarkConservativePointer reaches back to the start of the block, and a large allocation is checked against its own size. The walk back in a block stops at the block's start, and the large allocation bounds check no longer adds the reach to the highest base, which a large reach overflowed. Test: GcPointerFieldInterior, 2000 fields 40 and 400 bytes into Int arrays and one 200 KB into a large array. Before, 1999 of the 400-byte fields and the large one read freed memory, and on 32-bit all 4000 did; now none do, in the default, HXCPP_GC_MOVING and 32-bit builds on Windows. Fixes #451 --- Changes.md | 1 + src/hx/gc/Immix.cpp | 44 +++++++++----- .../regression/GcPointerFieldInterior/Main.hx | 58 +++++++++++++++++++ .../GcPointerFieldInterior/build.hxml | 2 + .../GcPointerFieldInterior/stdout.txt | 3 + 5 files changed, 94 insertions(+), 14 deletions(-) create mode 100644 test/regression/GcPointerFieldInterior/Main.hx create mode 100644 test/regression/GcPointerFieldInterior/build.hxml create mode 100644 test/regression/GcPointerFieldInterior/stdout.txt diff --git a/Changes.md b/Changes.md index bbc865942..3f381bda0 100644 --- a/Changes.md +++ b/Changes.md @@ -2,6 +2,7 @@ libhxcpp 6.0.0 (unreleased) ------------------------------------------------------------ +* Fixed a cpp.Pointer field, or a pointer boxed in a Dynamic, letting what it points into be freed when it points further in than 64 bytes, or 32 on 32-bit, as cpp.Pointer.arrayElem(a, 100) does: the fix for Haxe 4's AtomicInt kept only what a stack word would keep. A pointer field keeps its whole allocation now, large ones included, and pins it under HXCPP_GC_MOVING (libhxcpp #451) * Fixed cppia hanging or crashing when several threads asked Std.isOfType about script objects, or cast them to host interfaces, at once: each question about an interface a class did not implement inserted into that class's table, unlocked, while other threads searched it. Loading modules on several threads at once could likewise fail with "No base defined for non-interface", or hang, as each load inserted the host class names it missed into a map the others walked. Both are only read after they are filled now (libhxcpp #466) * Fixed Semaphore.tryAcquire with a negative timeout waiting for ever, as tryAcquire(deadline - now) does once the deadline has passed: since the fix for Haxe 5's Lock.wait() (libhxcpp #420), any negative timeout meant no limit. Only -1, which Haxe 5.0.0-preview.1's Lock.wait() passes for no timeout, waits with no limit now, and any other negative timeout returns at once, as 0 does (libhxcpp #461) * Fixed 32-bit Windows programs stopping with "Bad local allocator - requesting memory from unregistered thread!" when the process had used 64 TLS slots before the runtime started, as a host with many DLLs, or a late-loaded hxcpp DLL, can: the collector read its per-thread context from the thread's inline TLS slots, of which there are only 64. Later slots are read with TlsGetValue, as on x64 (libhxcpp #458) diff --git a/src/hx/gc/Immix.cpp b/src/hx/gc/Immix.cpp index 96dd27863..955e48704 100644 --- a/src/hx/gc/Immix.cpp +++ b/src/hx/gc/Immix.cpp @@ -1459,7 +1459,7 @@ struct BlockDataInfo } #ifdef HXCPP_GC_NURSERY - AllocType GetEnclosingNurseryType(uintptr_t inOffset, uintptr_t* outPtr) const + AllocType GetEnclosingNurseryType(uintptr_t inOffset, uintptr_t* outPtr, uintptr_t inReach=sgCheckInternalOffset) const { // The block did not get used in the previous cycle, so allocStart is invalid and // no new objects should be in here @@ -1505,7 +1505,7 @@ struct BlockDataInfo if (inOffset>=scan && inOffset scan + sgCheckInternalOffset) + if (inOffset > scan + inReach) return allocNone; if (nullptr != outPtr) @@ -1559,9 +1559,9 @@ struct BlockDataInfo return GetAllocTypeChecked(inOffset,inAllowPrevious); } - AllocType GetEnclosingAllocType(uintptr_t inOffset, uintptr_t* outPtr, bool inAllowPrevious) const + AllocType GetEnclosingAllocType(uintptr_t inOffset, uintptr_t* outPtr, bool inAllowPrevious, uintptr_t inReach=sgCheckInternalOffset) const { - for (uintptr_t dx{ 0 }; dx <= sgCheckInternalOffset; dx += 4) + for (uintptr_t dx{ 0 }; dx <= inReach && dx <= inOffset; dx += 4) { uintptr_t blockOffset{ inOffset - dx }; if (blockOffset >= 0) @@ -1601,7 +1601,7 @@ struct BlockDataInfo } #ifdef HXCPP_GC_NURSERY - return GetEnclosingNurseryType(inOffset,outPtr); + return GetEnclosingNurseryType(inOffset,outPtr,inReach); #endif // Not a actual start... return allocNone; @@ -7609,7 +7609,7 @@ class GlobalAllocator // allocation finds it too, as GetEnclosingAllocType does in a block, and // outBase is set to where it starts. Native code may hold only a field's // address, which kept a small object alive but not a large one (libhxcpp #373) - MemType GetMemType(uintptr_t inPtr, uintptr_t *outBase = nullptr) + MemType GetMemType(uintptr_t inPtr, uintptr_t *outBase = nullptr, size_t inReach = 0) { BlockData* block{ reinterpret_cast(inPtr & IMMIX_BLOCK_BASE_MASK) }; @@ -7621,9 +7621,10 @@ class GlobalAllocator // Most stack words are nowhere near the large allocations - reject on // the cached bounds before paying for the full list scan. The bounds // are of the starts, so a pointer into the highest reaches past them - size_t reach = outBase ? sgCheckInternalOffset : 0; + // inReach, for a pointer field, goes further: see MarkConservativePointer + size_t reach = outBase ? (inReach ? inReach : sgCheckInternalOffset) : 0; size_t p = (size_t)inPtr; - if (p < mLargeMin || p > mLargeMax + reach) + if (p < mLargeMin || (p > mLargeMax && p - mLargeMax > reach)) return memUnmanaged; uintptr_t base = 0; @@ -7791,7 +7792,9 @@ MarkChunk *MarkChunk::swapForNew() HX_NO_SANITIZE_ADDRESS -void MarkConservative(int *inBottom, int *inTop,hx::MarkContext *__inCtx) +// inReach is how far into an allocation a pointer may be and still keep it: +// 0 for a stack word, which uses sgCheckInternalOffset +static void MarkConservativeReach(int *inBottom, int *inTop,hx::MarkContext *__inCtx, uintptr_t inReach) { #ifdef VERIFY_STACK_READ VerifyStackRead(inBottom, inTop); @@ -7853,7 +7856,7 @@ void MarkConservative(int *inBottom, int *inTop,hx::MarkContext *__inCtx) hx::localCount++; #endif uintptr_t largeBase{}; - MemType mem{ sGlobalAlloc->GetMemType(potentialObject, &largeBase) }; + MemType mem{ sGlobalAlloc->GetMemType(potentialObject, &largeBase, inReach) }; #ifdef HX_WATCH isWatch = false; @@ -7898,8 +7901,8 @@ void MarkConservative(int *inBottom, int *inTop,hx::MarkContext *__inCtx) size_t pos{ potentialObject & IMMIX_BLOCK_OFFSET_MASK }; AllocType t{ - sgCheckInternalOffset ? - info->GetEnclosingAllocType(pos - sizeof(int), &potentialObject, allowPrevious) : + (inReach || sgCheckInternalOffset) ? + info->GetEnclosingAllocType(pos - sizeof(int), &potentialObject, allowPrevious, inReach ? inReach : sgCheckInternalOffset) : info->GetAllocType(pos - sizeof(int), allowPrevious) }; #ifdef HX_WATCH @@ -7982,15 +7985,28 @@ void MarkConservative(int *inBottom, int *inTop,hx::MarkContext *__inCtx) #endif } +void MarkConservative(int *inBottom, int *inTop,hx::MarkContext *__inCtx) +{ + MarkConservativeReach(inBottom, inTop, __inCtx, 0); +} + +// A cpp.Pointer field, or a pointer boxed in a Dynamic, keeps what it points +// into wherever it points in it, as cpp.Pointer.arrayElem(a, 100) does. A +// stack word keeps an allocation only within sgCheckInternalOffset of its +// start, since a pointer further in is usually held from the start too, but a +// field may be all there is (libhxcpp #451). In a block, nothing bigger than +// the block, and a large allocation is checked against its own size +static const uintptr_t sPointerFieldReach = ~(uintptr_t)0 >> 1; + void MarkConservativePointer(void *inPtr,hx::MarkContext *__inCtx) { void *word[1] = { inPtr }; // MarkConservative reads every pointer-sized word starting on an int in // the range, so this covers the one word and nothing past it #ifdef HXCPP_STACK_UP - MarkConservative(reinterpret_cast(word + 1) - 1, reinterpret_cast(word) + 1, __inCtx); + MarkConservativeReach(reinterpret_cast(word + 1) - 1, reinterpret_cast(word) + 1, __inCtx, sPointerFieldReach); #else - MarkConservative(reinterpret_cast(word), reinterpret_cast(word + 1), __inCtx); + MarkConservativeReach(reinterpret_cast(word), reinterpret_cast(word + 1), __inCtx, sPointerFieldReach); #endif } diff --git a/test/regression/GcPointerFieldInterior/Main.hx b/test/regression/GcPointerFieldInterior/Main.hx new file mode 100644 index 000000000..dd7a73957 --- /dev/null +++ b/test/regression/GcPointerFieldInterior/Main.hx @@ -0,0 +1,58 @@ +// libhxcpp #451. A cpp.Pointer field keeps what it points into, but only kept +// it within 64 bytes of the allocation's start (32 on 32-bit), the limit for a +// stack word. A pointer to element 100 of an Int array, 400 bytes in, or into +// the middle of a large array, kept nothing, and the array was freed under it. +class Holder { + public var near:cpp.Pointer; + public var far:cpp.Pointer; + + public function new() {} +} + +class Main { + static inline var COUNT = 2000; + + static var holders:Array; + static var large:cpp.Pointer; + + static function fill() { + for (i in 0...COUNT) { + var a = [for (j in 0...500) i * 1000 + j]; + holders[i].near = cpp.Pointer.arrayElem(a, 10); + var b = [for (j in 0...500) i * 1000 + j]; + holders[i].far = cpp.Pointer.arrayElem(b, 100); + } + // 400 KB, a large allocation, pointed to 200 KB in + var big = [for (j in 0...100000) j]; + large = cpp.Pointer.arrayElem(big, 50000); + } + + static function churn() { + var junk = []; + for (i in 0...20000) + junk.push([for (j in 0...100) 0x5a5a5a5a]); + var bigJunk = [for (i in 0...20) [for (j in 0...100000) 0x5a5a5a5a]]; + return junk.length + bigJunk.length; + } + + static function main() { + holders = [for (i in 0...COUNT) new Holder()]; + fill(); + for (round in 0...4) { + cpp.vm.Gc.run(true); + churn(); + } + cpp.vm.Gc.run(true); + var wrongNear = 0; + var wrongFar = 0; + for (i in 0...COUNT) { + if (holders[i].near.value != i * 1000 + 10) + wrongNear++; + if (holders[i].far.value != i * 1000 + 100) + wrongFar++; + } + Sys.println('pointers 40 bytes in, wrong: $wrongNear of $COUNT'); + Sys.println('pointers 400 bytes in, wrong: $wrongFar of $COUNT'); + Sys.println('pointer 200 KB into a large array: ' + (large.value == 50000 ? "right" : "wrong")); + } +} diff --git a/test/regression/GcPointerFieldInterior/build.hxml b/test/regression/GcPointerFieldInterior/build.hxml new file mode 100644 index 000000000..b484e67e7 --- /dev/null +++ b/test/regression/GcPointerFieldInterior/build.hxml @@ -0,0 +1,2 @@ +-m Main +--cpp bin diff --git a/test/regression/GcPointerFieldInterior/stdout.txt b/test/regression/GcPointerFieldInterior/stdout.txt new file mode 100644 index 000000000..1038c087c --- /dev/null +++ b/test/regression/GcPointerFieldInterior/stdout.txt @@ -0,0 +1,3 @@ +pointers 40 bytes in, wrong: 0 of 2000 +pointers 400 bytes in, wrong: 0 of 2000 +pointer 200 KB into a large array: right From 647216fadef19c2d732728a2cb9277a9f0b1a5cf Mon Sep 17 00:00:00 2001 From: Chris Speciale Date: Thu, 19 Nov 2026 08:37:52 -0500 Subject: [PATCH 7/7] Count live bytes and Gc.trace right with an incremental cycle With HXCPP_GC_INCREMENTAL: - A finished cycle that is not a full reclaim only runs countRows, whose bytesInUse is whole rows, and Collect stored that as mLiveBytes because the cycle had finished. MEM_INFO_LIVE, the bytes of the objects that survived the last full reclaim, then reported every used row as live, seven times the real figure in the test. It is set by full reclaims only. - Gc.trace counts what its collection marks. With a cycle part done, the collection finished it, and the objects its earlier slices had marked were not counted. A census already fails the partial cycle for this reason; Gc.trace does too now. Test: IncrementalGcCounts, which the incremental gc job builds in debug, as Gc.trace needs. Before, MEM_INFO_LIVE changed after the cycle, and Gc.trace counted 90668 and 108653 of 200000; now both are right, on Windows x64. Fixes #453 --- .github/workflows/test.yml | 9 ++- Changes.md | 1 + src/hx/gc/Immix.cpp | 18 +++-- test/regression/IncrementalGcCounts/Main.hx | 79 +++++++++++++++++++ .../regression/IncrementalGcCounts/build.hxml | 4 + .../regression/IncrementalGcCounts/stdout.txt | 3 + 6 files changed, 105 insertions(+), 9 deletions(-) create mode 100644 test/regression/IncrementalGcCounts/Main.hx create mode 100644 test/regression/IncrementalGcCounts/build.hxml create mode 100644 test/regression/IncrementalGcCounts/stdout.txt diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 007a0b254..6a9a75b9a 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -495,11 +495,14 @@ jobs: working-directory: test/regression run: | failed="" - for test in GcMutationRoots GcContainerMutation IncrementalGcSlices IncrementalGcContainers IncrementalGcArrayChunks IncrementalGcDequeMoves IncrementalGcPacing IncrementalGcStartSweep IncrementalGcStats IncrementalGcFinishBoundary IncrementalGcVerifier IncrementalWindowsHeaders GcWeakMapChain GcWeakMapWalkCollect GenerationalFinalizerLifecycle GenerationalLargeBufferLifecycle; do + for test in GcMutationRoots GcContainerMutation IncrementalGcSlices IncrementalGcContainers IncrementalGcArrayChunks IncrementalGcDequeMoves IncrementalGcPacing IncrementalGcStartSweep IncrementalGcStats IncrementalGcFinishBoundary IncrementalGcVerifier IncrementalGcCounts IncrementalWindowsHeaders GcWeakMapChain GcWeakMapWalkCollect GenerationalFinalizerLifecycle GenerationalLargeBufferLifecycle; do echo "::group::incremental gc: $test" # The lifecycle fixtures' hxml selects generational mode; compile - # the same Main with incremental mode here instead. - if ! (cd "$test" && haxe -main Main -cpp bin -lib hxcpp -D HXCPP_GC_INCREMENTAL -D HXCPP_GC_INCREMENTAL_VERIFY -D HXCPP_GC_INCREMENTAL_TEST -D $HXCPP_ARCH_FLAG && bin/Main > out.txt && diff <(tr -d '\r' < stdout.txt) <(tr -d '\r' < out.txt)); then + # the same Main with incremental mode here instead. Gc.trace, which + # IncrementalGcCounts checks, counts in debug builds only. + extra="" + [ "$test" = IncrementalGcCounts ] && extra="-debug -D DEBUGSUFFIX=" + if ! (cd "$test" && haxe -main Main -cpp bin -lib hxcpp -D HXCPP_GC_INCREMENTAL -D HXCPP_GC_INCREMENTAL_VERIFY -D HXCPP_GC_INCREMENTAL_TEST -D $HXCPP_ARCH_FLAG $extra && bin/Main > out.txt && diff <(tr -d '\r' < stdout.txt) <(tr -d '\r' < out.txt)); then failed="$failed $test" echo "::error::incremental gc failed: $test" fi diff --git a/Changes.md b/Changes.md index 3f381bda0..16b55242e 100644 --- a/Changes.md +++ b/Changes.md @@ -2,6 +2,7 @@ libhxcpp 6.0.0 (unreleased) ------------------------------------------------------------ +* Fixed two counts with HXCPP_GC_INCREMENTAL: Gc.memInfo64(MEM_INFO_LIVE) after an incremental cycle reported every used row as live, several times the bytes really live, where it is meant to report the last full reclaim, and Gc.trace with a cycle part done counted only the objects the cycle had not marked yet. MEM_INFO_LIVE changes only with a full reclaim now, and Gc.trace starts the cycle again, as __hxcpp_gc_census does (libhxcpp #453) * Fixed a cpp.Pointer field, or a pointer boxed in a Dynamic, letting what it points into be freed when it points further in than 64 bytes, or 32 on 32-bit, as cpp.Pointer.arrayElem(a, 100) does: the fix for Haxe 4's AtomicInt kept only what a stack word would keep. A pointer field keeps its whole allocation now, large ones included, and pins it under HXCPP_GC_MOVING (libhxcpp #451) * Fixed cppia hanging or crashing when several threads asked Std.isOfType about script objects, or cast them to host interfaces, at once: each question about an interface a class did not implement inserted into that class's table, unlocked, while other threads searched it. Loading modules on several threads at once could likewise fail with "No base defined for non-interface", or hang, as each load inserted the host class names it missed into a map the others walked. Both are only read after they are filled now (libhxcpp #466) * Fixed Semaphore.tryAcquire with a negative timeout waiting for ever, as tryAcquire(deadline - now) does once the deadline has passed: since the fix for Haxe 5's Lock.wait() (libhxcpp #420), any negative timeout meant no limit. Only -1, which Haxe 5.0.0-preview.1's Lock.wait() passes for no timeout, waits with no limit now, and any other negative timeout returns at once, as 0 does (libhxcpp #461) diff --git a/src/hx/gc/Immix.cpp b/src/hx/gc/Immix.cpp index 955e48704..8894e12ff 100644 --- a/src/hx/gc/Immix.cpp +++ b/src/hx/gc/Immix.cpp @@ -6547,13 +6547,20 @@ class GlobalAllocator { hx::IncrementalState &state = hx::IncrementalData(); bool active = hx::gIncrementalActive.load(std::memory_order_relaxed); - bool requested = hx::sIncrementalRequestMs > 0 && !sgCensus; + // A census, or Gc.trace, counts what this collection marks, so a cycle + // part done, whose earlier slices marked objects already, is failed and + // everything marked again: finishing it undercounted (libhxcpp #453) + bool counting = sgCensus; + #ifdef HXCPP_DEBUG + counting = counting || gCollectTrace; + #endif + bool requested = hx::sIncrementalRequestMs > 0 && !counting; if (active && !requested) { std::lock_guard lock(state.statsLock); state.forcedFinishes++; } - if (active && sgCensus) + if (active && counting) state.failed.store(true, std::memory_order_relaxed); if (active && hx::sIncrementalRequestMs == 0) { @@ -7046,11 +7053,10 @@ class GlobalAllocator STAMP(t5) size_t mem = mRowsInUse<