diff --git a/Changes.md b/Changes.md index 813617ac3..557dcf1b6 100644 --- a/Changes.md +++ b/Changes.md @@ -2,6 +2,7 @@ libhxcpp 6.0.0 (unreleased) ------------------------------------------------------------ +* Added adoptInstances to a loaded cppia module: called with the module it replaces, before that one is released, it moves the live instances of each of the old module's classes onto the new class of the same name when the two lay out their fields the same way, so code reloaded into a running process reaches objects made before the reload. Calls on them run the new code, through interfaces and dynamic calls too, and Type.getClass gives the new class. It returns a row for each old class with live instances: how many moved, and for a class whose fields changed, which field differs. Bound methods and closures made before keep the old code. See docs/CppiaModuleRelease.md (libhxcpp #265) * Fixed the incremental collector (HXCPP_GC_INCREMENTAL) freeing a thread object's arrays while the thread was being created, as the gc stress test's verifier caught on Windows. A slice could find the new object on the stack while its first array was being allocated, and the cycle could end before the constructor's barrier, with the arrays stored after that unscanned. The arrays are now made in a mutation scope, which the stop that finishes a cycle scans again * cpp.encoding.Utf8.decode replaces each malformed UTF-8 sequence with U+FFFD instead of throwing, with smart strings or without. Haxe 5's Bytes.toString and getString decode with it, so they threw for bytes that Haxe 4's, which go through String::create, decode with U+FFFD in the same places. Valid input still takes the decoder that counts first; only malformed input takes the replacing one. Utf8.codepoint still throws * Fixed String::utf8_str into a view, at API level 500, writing the bytes and then throwing when the view had room for them but not the terminator. It returns false before writing anything. It also stopped a UTF-16 string at its first NUL, which is a character, and reported that length (libhxcpp #408) diff --git a/docs/CppiaModuleRelease.md b/docs/CppiaModuleRelease.md index 3887f41fa..42c17d271 100644 --- a/docs/CppiaModuleRelease.md +++ b/docs/CppiaModuleRelease.md @@ -66,3 +66,30 @@ for (row in (untyped __global__.__scriptable_cppia_retained() : Array)) It works in release builds, like `__hxcpp_gc_census()`, on which it builds. Not counted, since the module holds them itself while it is loaded: its `Class` objects, its enum values, and its static functions used as values. A host holding one of those keeps the module loaded too. Code of the module still running on a thread, and stale pointers to it left on a stack (see Limitations), keep it loaded without anything being counted. To find what holds the instances of a class, build the host with `-debug` and call `cpp.vm.Gc.trace` with that class, kept from before the release, since `Type.resolveClass` no longer finds it: it prints the path from a root to each instance. + +## Moving live instances onto the new module's classes + +Instances made before a reload keep their class and its code, so a reload only changes objects made afterwards. A host can move them instead, after the new module has booted and before the old one is released: + +```haxe +var next = cpp.cppia.Module.fromData(bytes.getData()); +next.boot(); +var report:Array = untyped next.adoptInstances(previous); +untyped previous.release(); +``` + +`adoptInstances(previous)` runs a full collection, and in it gives each live instance of a previous module's class the class of the same name in this module, when the two lay out their fields the same way. Calls on a moved instance then run the new code, through the instance, its interfaces and dynamic calls alike, and `Type.getClass` gives the new class. It returns one row for each previous class that had live instances, in name order: + +- `name`, the class's name. +- `moved`, how many of its instances moved. +- `reason`, null when they could move, otherwise why not, naming the first field that differs. + +The rule is strict, as an instance stays where it is, at its size. Both classes must extend the same host class, through super classes of the same names, implement the same interfaces, and have the same fields, dynamic functions included, with the same names, storage types and offsets. Methods can differ in any way, so editing method bodies, the usual case, is covered. A class that adds, removes, retypes or reorders a field keeps its instances, which keep the old module loaded until they go. + +What does not move: + +- Bound methods and closures made from the old class's methods before the step keep calling the old code, and keep the old module loaded until they are dropped. The same goes for a dynamic function's value held in a field. +- Code running on any thread finishes in the old code. The fields are laid out the same, so it reads and writes them correctly. +- Statics: the new module has its own, and nothing is copied. + +Calling it twice moves nothing the second time. A host that does not call it sees no change. diff --git a/src/hx/cppia/CppiaModule.cpp b/src/hx/cppia/CppiaModule.cpp index e50e1c0c4..38334c1c7 100644 --- a/src/hx/cppia/CppiaModule.cpp +++ b/src/hx/cppia/CppiaModule.cpp @@ -441,6 +441,8 @@ void CppiaConst::fromStream(CppiaStream &stream) +static Array adoptInstances(CppiaModule *inNext, CppiaModule *inPrevious); + // Cppia Object - manage class CppiaObject : public hx::CppiaLoadedModule_obj { @@ -570,10 +572,47 @@ class CppiaObject : public hx::CppiaLoadedModule_obj #endif }; + // Moves the live instances of the previous module's classes onto this + // module's classes of the same name, where their fields are laid out the + // same, and reports each class that had any (libhxcpp #265) + Array adopt(Dynamic inPrevious) + { + CppiaObject *previous = dynamic_cast(inPrevious.mPtr); + if (!previous) + hx::Throw(HX_CSTRING("adoptInstances takes a cppia module")); + if (previous==this) + hx::Throw(HX_CSTRING("adoptInstances takes another module")); + if (previous->cppia->released) + hx::Throw(HX_CSTRING("adoptInstances takes a module before its release")); + // Moved instances run this module's code, which needs its statics + boot(); + return adoptInstances(cppia, previous->cppia); + } + + struct AdoptFunc : public hx::LocalFunc + { + hx::Object *module; + AdoptFunc(hx::Object *inModule) : module(inModule) { } + Dynamic __Run(const Array &inArgs) HXCPP_OVERRIDE + { + return static_cast(module)->adopt(inArgs->length ? inArgs[0] : Dynamic()); + } + #if (HXCPP_API_LEVEL<500) + Dynamic __run(const Dynamic &inPrevious) HXCPP_OVERRIDE { return static_cast(module)->adopt(inPrevious); } + #endif + int __ArgCount() const HXCPP_OVERRIDE { return 1; } + void __Mark(hx::MarkContext *__inCtx) HXCPP_OVERRIDE { HX_MARK_MEMBER(module); } + #ifdef HXCPP_VISIT_ALLOCS + void __Visit(hx::VisitContext *__inCtx) HXCPP_OVERRIDE { HX_VISIT_OBJECT(module); } + #endif + }; + hx::Val __Field(const String &inName, hx::PropertyAccess inCallProp) HXCPP_OVERRIDE { if (inName==HX_CSTRING("release")) return Dynamic(new ReleaseFunc(this)); + if (inName==HX_CSTRING("adoptInstances")) + return Dynamic(new AdoptFunc(this)); return hx::CppiaLoadedModule_obj::__Field(inName, inCallProp); } @@ -1202,6 +1241,171 @@ static hx::Anon retainedRow(const char *inName, const RetainedTally &inTally) return row; } +// The census visitor is one global, so its users take turns +static std::mutex *sCensusVisitorLock = new std::mutex(); + +static void censusWithVisitor(void (*inVisitor)(hx::Object *, size_t)) +{ + { + hx::AutoGCFreeZone wait; + sCensusVisitorLock->lock(); + } + gCensusVisitor = inVisitor; + __hxcpp_gc_census(); + gCensusVisitor = 0; + sCensusVisitorLock->unlock(); +} + + +// --- Moving live instances onto a replaced class (libhxcpp #265) --- +// +// An instance of a script class finds everything about its class through its +// script vtable, whose slot -1 is the class: its methods, its interfaces, how +// the collector marks it, Type.getClass. So an instance moves to another class +// by having that one pointer replaced. It stays where it is, at its size, so +// the new class must lay out every field as the old one did. + +// A previous module's class with live instances, and the class they move to, +// or why they cannot +struct Adoption +{ + CppiaClassInfo *to; + std::string reason; + size_t moved; + size_t live; + Adoption() : to(0), moved(0), live(0) { } +}; +static std::map *sAdoptions = 0; + +static std::string typeName(TypeData *inType) +{ + return std::string(inType->name.utf8_str()); +} + +// The interfaces a class and its script super classes implement, by name +static std::vector interfaceNames(CppiaClassInfo *inInfo) +{ + std::vector names; + for(CppiaClassInfo *info=inInfo; info; info=info->superType ? info->superType->cppiaClass : 0) + for(size_t i=0;iimplements.size();i++) + names.push_back(typeName(info->cppia.types[info->implements[i]])); + std::sort(names.begin(), names.end()); + return names; +} + +// The first difference between two lists of a class's storage, its fields or +// its dynamic functions, which are linked with their super classes' first +static std::string slotDifference(const char *inWhat, CppiaClassInfo *inFrom, std::vector &inOld, + CppiaClassInfo *inTo, std::vector &inNew) +{ + for(size_t i=0;i=inOld.size()) + return std::string(inWhat) + " " + inTo->cppia.identStr(inNew[i]->nameId) + " is not in the old class"; + std::string name = inFrom->cppia.identStr(inOld[i]->nameId); + if (i>=inNew.size()) + return std::string(inWhat) + " " + name + " is not in the new class"; + if (name!=inTo->cppia.identStr(inNew[i]->nameId)) + return std::string(inWhat) + " " + name + " is not where it was"; + if (inOld[i]->storeType!=inNew[i]->storeType || inOld[i]->offset!=inNew[i]->offset) + return std::string(inWhat) + " " + name + " changed its type"; + } + return std::string(); +} + +// Why instances of inFrom cannot become instances of inTo, or empty when they can +static std::string incompatibility(CppiaClassInfo *inFrom, CppiaClassInfo *inTo) +{ + if (inFrom->haxeBase!=inTo->haxeBase) + return "it extends another host class"; + TypeData *from = inFrom->superType; + TypeData *to = inTo->superType; + while(from || to) + { + if (!from || !to || typeName(from)!=typeName(to)) + return "its super classes are not the same"; + from = from->cppiaClass ? from->cppiaClass->superType : 0; + to = to->cppiaClass ? to->cppiaClass->superType : 0; + } + if (interfaceNames(inFrom)!=interfaceNames(inTo)) + return "it implements other interfaces"; + std::string field = slotDifference("field", inFrom, inFrom->memberVars, inTo, inTo->memberVars); + if (field.empty()) + field = slotDifference("dynamic function", inFrom, inFrom->dynamicFunctions, inTo, inTo->dynamicFunctions); + if (!field.empty()) + return field; + if (inFrom->classSize!=inTo->classSize || inFrom->extraData!=inTo->extraData || + inFrom->dynamicMapOffset!=inTo->dynamicMapOffset || inFrom->containsPointers!=inTo->containsPointers) + return "its instances are laid out otherwise"; + return std::string(); +} + +// Called with each object the census marks, on one thread with every other stopped +static void adoptVisit(hx::Object *inObj, size_t) +{ + // A module marks its own Class objects + if (getScriptClassInfo(inObj)) + return; + void **vtable = inObj->__GetScriptVTable(); + if (!vtable) + return; + std::map::iterator found = sAdoptions->find((CppiaClassInfo *)vtable[-1]); + if (found==sAdoptions->end()) + return; + Adoption &adoption = found->second; + adoption.live++; + if (!adoption.to) + return; + void ***slot = (void ***)((char *)inObj + found->first->getScriptVTableOffset()); + if (*slot!=vtable) + return; + *slot = adoption.to->vtable; + adoption.moved++; +} + +static Array adoptInstances(CppiaModule *inNext, CppiaModule *inPrevious) +{ + std::map adoptions; + for(size_t i=0;iclasses.size();i++) + { + CppiaClassInfo *info = inPrevious->classes[i]; + if (info->isInterface || info->isEnum || !info->vtable) + continue; + Adoption &adoption = adoptions[info]; + CppiaClassInfo *to = inNext->findClass(String::create(info->name.c_str())); + if (!to || to->isInterface || to->isEnum || !to->vtable) + adoption.reason = "the new module has no class of that name"; + else + { + adoption.reason = incompatibility(info, to); + if (adoption.reason.empty()) + adoption.to = to; + } + } + + sAdoptions = &adoptions; + censusWithVisitor(adoptVisit); + sAdoptions = 0; + + std::vector< std::pair > rows; + for(std::map::iterator i=adoptions.begin(); i!=adoptions.end(); ++i) + if (i->second.live) + rows.push_back(std::make_pair(i->first->name, i->second)); + std::sort(rows.begin(), rows.end(), [](const std::pair &a, const std::pair &b) + { return a.first < b.first; }); + + Array result = Array_obj::__new(0,(int)rows.size()); + for(size_t r=0;r__SetField(HX_CSTRING("name"), String::create(rows[r].first.c_str()), hx::paccDynamic); + row->__SetField(HX_CSTRING("moved"), (int)std::min(rows[r].second.moved, (size_t)0x7fffffff), hx::paccDynamic); + row->__SetField(HX_CSTRING("reason"), rows[r].second.reason.empty() ? String() : String::create(rows[r].second.reason.c_str()), hx::paccDynamic); + result->push(row); + } + return result; +} + } // end namespace hx // Collects, then reports each released module that is still loaded: its @@ -1223,9 +1427,7 @@ Array __scriptable_cppia_retained() } hx::sRetained = &retained; - hx::gCensusVisitor = hx::retainedVisit; - __hxcpp_gc_census(); - hx::gCensusVisitor = 0; + hx::censusWithVisitor(hx::retainedVisit); hx::sRetained = 0; // Copied out under the lock, and turned into Haxe values after it: a diff --git a/test/regression/CppiaAdoptInstances/Main.hx b/test/regression/CppiaAdoptInstances/Main.hx new file mode 100644 index 000000000..c644647a0 --- /dev/null +++ b/test/regression/CppiaAdoptInstances/Main.hx @@ -0,0 +1,111 @@ +import cpp.cppia.Host; +import cpp.cppia.Module; + +// A module's adoptInstances(previous) moves the live instances of each of the +// previous module's classes onto the class of the same name, when the two lay +// out their fields the same way, so their calls run the new code. Instances of +// a class whose fields changed stay, and the report says why. A method bound +// before stays on the old code and keeps the old module loaded until it is +// dropped. openfl/libhxcpp#265 +// +// Every call into script code runs on a joined native worker, so stale values +// it leaves on a stack do not decide what keeps a module loaded. The main +// thread only collects and prints. +@:cppFileCode(' +#include +#include +static void runJoined(Dynamic callback) { + std::thread worker([callback]() mutable { hx::NativeAttach attach; callback(); }); + hx::AutoGCFreeZone freeZone; + worker.join(); +} +') +class Main { + static var dir:String; + static var previous:Module; + static var next:Module; + static var counter:Dynamic; + static var doubler:Dynamic; + static var reshaped:Dynamic; + static var bound:Void->Int; + static var lines:Array; + + static function load(name:String):Module { + var module = Module.fromData(sys.io.File.getBytes(haxe.io.Path.join([dir, name])).getData()); + module.boot(); + module.run(); + return module; + } + + static function loadOld() { + previous = load("v1.cppia"); + counter = Type.createInstance(previous.resolveClass("Counter"), [5]); + counter.step(); + doubler = Type.createInstance(previous.resolveClass("Doubler"), [0]); + reshaped = Type.createInstance(previous.resolveClass("Reshaped"), []); + bound = Reflect.field(counter, "step"); + } + + static function adopt() { + next = load("v2.cppia"); + var report:Array = untyped next.adoptInstances(previous); + untyped previous.release(); + previous = null; + var rows = [for (r in report) '${r.name} moved ${r.moved}' + (r.reason == null ? "" : ': ${r.reason}')]; + rows.sort(Reflect.compare); + lines.push('report: ${rows.join(", ")}'); + } + + static function useMoved() { + // Moved, with their fields as they were + lines.push('counter runs the new code: ${counter.step() == 16 && counter.version() == "v2 counter"}'); + lines.push('the subclass runs its new override: ${doubler.step() == 20 && doubler.version() == "v2 counter"}'); + var stepper:Stepper = counter; + lines.push('the host interface reaches the new code: ${stepper.step() == 26}'); + var counterClass = next.resolveClass("Counter"); + lines.push('they are of the new classes: ${Type.getClass(counter) == counterClass && Std.isOfType(counter, counterClass) && Std.isOfType(doubler, counterClass) && Type.getClass(doubler) == next.resolveClass("Doubler")}'); + } + + static function useOld() { + // Stayed + lines.push('the reshaped instance keeps its class: ${reshaped.version() == "v1"}'); + lines.push('the bound method keeps the old code: ${bound() == 27 && counter.step() == 37}'); + } + + static function dropOld() { + bound = null; + reshaped = null; + } + + static function dropAll() { + counter = doubler = null; + untyped next.release(); + next = null; + } + + static function retained():String { + var modules:Array = untyped __global__.__scriptable_cppia_retained(); + return [ + for (m in modules) + [for (i in (m.instances : Array)) '${i.name} x${i.count}'].concat(['closures x${m.closures.count}']).join(", ") + ].join("; "); + } + + static function main() { + dir = haxe.io.Path.directory(Sys.programPath()); + for (jit in [false, true]) { + Host.enableJit(jit); + lines = []; + untyped __cpp__("runJoined({0})", loadOld); + untyped __cpp__("runJoined({0})", adopt); + untyped __cpp__("runJoined({0})", useMoved); + untyped __cpp__("runJoined({0})", useOld); + lines.push('the old module keeps only what stayed: ${retained()}'); + untyped __cpp__("runJoined({0})", dropOld); + lines.push('and is freed once they go: "${retained()}"'); + untyped __cpp__("runJoined({0})", dropAll); + for (line in lines) + Sys.println('jit $jit: $line'); + } + } +} diff --git a/test/regression/CppiaAdoptInstances/Stepper.hx b/test/regression/CppiaAdoptInstances/Stepper.hx new file mode 100644 index 000000000..383472407 --- /dev/null +++ b/test/regression/CppiaAdoptInstances/Stepper.hx @@ -0,0 +1,4 @@ +// A host interface the script classes implement +interface Stepper { + function step():Int; +} diff --git a/test/regression/CppiaAdoptInstances/build.hxml b/test/regression/CppiaAdoptInstances/build.hxml new file mode 100644 index 000000000..ac4136d43 --- /dev/null +++ b/test/regression/CppiaAdoptInstances/build.hxml @@ -0,0 +1,12 @@ +-cp v1 +--main Script +-cppia bin/v1.cppia +--next +-cp v2 +--main Script +-cppia bin/v2.cppia +--next +--main Main +-cpp bin +-D scriptable +--dce no diff --git a/test/regression/CppiaAdoptInstances/stdout.txt b/test/regression/CppiaAdoptInstances/stdout.txt new file mode 100644 index 000000000..6e4b52f9d --- /dev/null +++ b/test/regression/CppiaAdoptInstances/stdout.txt @@ -0,0 +1,18 @@ +jit false: report: Counter moved 1, Doubler moved 1, Reshaped moved 0: field b is not in the old class +jit false: counter runs the new code: true +jit false: the subclass runs its new override: true +jit false: the host interface reaches the new code: true +jit false: they are of the new classes: true +jit false: the reshaped instance keeps its class: true +jit false: the bound method keeps the old code: true +jit false: the old module keeps only what stayed: Reshaped x1, closures x1 +jit false: and is freed once they go: "" +jit true: report: Counter moved 1, Doubler moved 1, Reshaped moved 0: field b is not in the old class +jit true: counter runs the new code: true +jit true: the subclass runs its new override: true +jit true: the host interface reaches the new code: true +jit true: they are of the new classes: true +jit true: the reshaped instance keeps its class: true +jit true: the bound method keeps the old code: true +jit true: the old module keeps only what stayed: Reshaped x1, closures x1 +jit true: and is freed once they go: "" diff --git a/test/regression/CppiaAdoptInstances/v1/Script.hx b/test/regression/CppiaAdoptInstances/v1/Script.hx new file mode 100644 index 000000000..3eb630834 --- /dev/null +++ b/test/regression/CppiaAdoptInstances/v1/Script.hx @@ -0,0 +1,35 @@ +class Counter implements Stepper { + public var count:Int; + public var label:String; + + public function new(start:Int) { + count = start; + label = "counter"; + } + + public function step():Int + return count += 1; + + public function version():String + return "v1 " + label; +} + +class Doubler extends Counter { + override public function step():Int + return count += 2; +} + +// v2 adds a field, so its instances cannot move +class Reshaped { + public var a:Int; + + public function new() + a = 1; + + public function version():String + return "v1"; +} + +class Script { + public static function main() {} +} diff --git a/test/regression/CppiaAdoptInstances/v2/Script.hx b/test/regression/CppiaAdoptInstances/v2/Script.hx new file mode 100644 index 000000000..44ef7727c --- /dev/null +++ b/test/regression/CppiaAdoptInstances/v2/Script.hx @@ -0,0 +1,35 @@ +class Counter implements Stepper { + public var count:Int; + public var label:String; + + public function new(start:Int) { + count = start; + label = "counter"; + } + + public function step():Int + return count += 10; + + public function version():String + return "v2 " + label; +} + +class Doubler extends Counter { + override public function step():Int + return count += 20; +} + +class Reshaped { + public var a:Int; + public var b:Int; + + public function new() + a = 1; + + public function version():String + return "v2"; +} + +class Script { + public static function main() {} +} diff --git a/test/regression/emscripten-skip.txt b/test/regression/emscripten-skip.txt index e4b0c22f6..8ae0141d6 100644 --- a/test/regression/emscripten-skip.txt +++ b/test/regression/emscripten-skip.txt @@ -12,6 +12,7 @@ CffiNekoValueTags: runs neko, which loads an ndll the test builds CffiPrimeAfterCffi: loads an ndll it builds, and Emscripten links statically CffiPrimeUnicodeFallback: loads an ndll it builds, and Emscripten links statically CffiQuietMissingPrimitive: loads an ndll it builds, and Emscripten links statically +CppiaAdoptInstances: threads (std::thread, which needs a pthreads build) CppiaArrayAccess: its host is the first --next section, which -D emscripten does not reach CppiaBreakpoints: threads CppiaHostArgs: processes