Automate local source preparation and bounded near-match repairs

README.md

@@ -29,6 +29,21 @@ under a tenth of a second (1.4 s before), and agents restart by themselves after
   stand-in for a model that hands in the demo's own C; the app compiles and judges every function for real.</sub>
 </p>
 
+## Automatic repairs before paid agents
+
+The permuter's bounded repair pass tries unused scalar locals, local and parameter `register` storage, and adjacent
+local declaration order before random permutation. It keeps original line layout and leaves shared headers alone.
+These hypotheses run through the same compiler and exact judge as agents, in ROM and supported 32-bit imported
+projects. They use local CPU, with no model API calls. Rejected candidates do not replace the best saved C.
+
+For close ROM misses it also tests cached compiler flag alternatives, two probes per pass. Later passes skip
+already tested choices and can reach optimization levels as well as debug flags. Only exact results qualify for a
+narrow compiler-range change. When automatic compiler changes are disabled, the result becomes a suggestion for
+you to review. Imported projects retain their own build settings.
+
+This cannot identify an unknown original-source function or reconstruct a substantially wrong algorithm. Those
+still need source-assisted agent work. It also cannot guarantee that a nearly matching function will match.
+
 ## Changing a ROM project's compiler
 
 Changing the game's compiler or an address range checks every affected matched function before saving.

ROADMAP.md

@@ -71,6 +71,7 @@ folder, `src/decompnet/consoles/<id>/`; a new console adds a column (CONTRIBUTIN
 | Drafts follow the header, made in the background; `redraft` | ✓ | ✓ | ✓ | ✓ | — | ✓ Ghidra's | — | — |
 | C++ methods drafted as `Class::method` | ✓ | ✓ | ✓ | ✓ | — | ✓ (RTTI, and classes from the vtables constructors store) | — | — |
 | Twins: free matches from identical code | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
+| Local mapped source preparation | ✓ | ✓ | ✓ | ✓ | body only¹ | body only¹ | body only¹ | body only¹ |
 | Permuter on near misses | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
 | Hand-written assembly set aside | ✓ | ✓ | ✓ | ✓ | ✓ | Visual C++'s exception-handling stubs | BIOS call stubs, pieces of functions | MIPS, PowerPC, ARM's BIOS stubs and pieces of functions |
 | Pieces of functions joined to their function | — | — | — | — | — | ✓ (`scratch.merge_pieces`) | ✓ (`recover.merge_pieces`) | — |
@@ -834,6 +835,22 @@ Repeated-output limits now remember bounded fingerprints across a claim and rese
 Function lists show check/improvement recency. Agent work selection supports oldest
 checked first and a recorded-model untried filter in ROM and imported projects.
 
+The shared deterministic repair pass now tries bounded unused scalar removal, scalar/pointer storage and adjacent local declaration order on supported 32-bit ROM/imported projects, preserving line layout and using normal exact judges. Close ROM misses rotate cached debug and optimization probes, two per pass, with manual compiler approval respected. Imported compiler flags remain owned by their build.
+
+### Local source preparation and renewed free searches
+
+¹ Imported builds retain their own declarations headers; reference-header injection remains a planned gap.
+
+Opt-in Settings → Local source preparation reads explicitly mapped local reference bodies, preserves formatting,
+applies supplied numeric retail macros and judges each seed before saving it. ROM mode installs improved seeds
+with private per-function declarations; imported mode uses its existing build headers and objdiff judge. Both
+modes share bounded preparation, cached trials and diff-focused permutation. External input changes renew bounded
+random turns; the permuter's own improvements do not. See [the setup guide](docs/LOCAL_SOURCE_PREPARATION.md).
+
+Remaining gap: imported projects cannot automatically inject a reference declarations header into the project
+build. Their body must already compile against the project's own headers. This needs a reviewed header edit and
+regression checking of affected units before it can become automatic.
+
 
 Agent work selection now reaches small-function batches through the shared scheduler (including percentage,
 model-history, attempt and lease filters). Paused queues wait without starting a paid session. Cooldowns and
@@ -848,3 +865,4 @@ ROM and imported briefs via get_function(section="quality"). Existing notes are
 These are not semantic proofs: best scores, saved best C and the exact judge remain unchanged. Alternatives must
 be rejudged in their current context. Custom generated dependencies outside the observed build/include files
 still need richer build-system dependency records.
+

docs/LOCAL_SOURCE_PREPARATION.md

@@ -0,0 +1,70 @@
+# Free local source preparation and permutation
+
+The permuter can use explicitly mapped local reference source before its usual deterministic and random searches.
+This is optional: ROM-only projects keep working without it. Reference code from another version is a useful seed,
+not proof that a function matches the retail game.
+
+## Set it up
+
+Open the game's **Settings → Local source preparation**:
+
+1. Select the reference source folder and any SDK/include folders it needs. These folders are read-only inputs;
+   the app does not execute reference build scripts or download SDKs.
+2. Map each intended destination explicitly, one per line:
+   `func_80001000 = update_objects @ objects.c`.
+   The path is relative to the reference folder. Ambiguous/duplicate mappings are refused. Matching a name or a
+   source location does not establish a match; no source-derived address is added to the destination symbol map.
+3. Supply a declarations-only header with the necessary types and prototypes when possible. It must live inside a
+   selected folder. Otherwise the app tries the mapped file's includes and declarations, or the current project
+   header if there are no includes. Complex conditional includes, compiler extensions or initialized globals may
+   need a smaller prepared header. An unresolved include is reported rather than guessed.
+4. If retail and debug layouts differ, enter numeric overrides such as `OBJECT_LIMIT=128`. Existing definitions
+   are replaced before dependent structs are compiled. Overrides do not infer the correct value or add a missing
+   definition. Add any known global aliases separately: `gReferenceObjects = D_80010000`.
+5. Enable preparation and **Save source settings**, then **Try preparation pass**. The button runs a background job
+   with a progress bar and per-function diagnostics. The permuter also runs a preparation pass before its usual work.
+
+A pass is bounded to eight compiler attempts and 20 seconds between attempts. One compiler invocation can finish
+past that deadline, within the compiler's normal timeout. Unchanged trials (including compiler failures) are
+cached. Another pass continues with untried mapped functions. Edited reference bodies, headers, mappings, macro
+values or destination build inputs give those candidates another chance. The original function body keeps its
+line breaks and original line number: IDO can schedule code differently when source line information changes.
+Token replacements leave comments and string literals alone. A mapped `static` entry point is made externally
+visible for the isolated compile, without shifting its lines. Its original body remains unchanged.
+
+## What gets saved
+
+For ROM projects, a prepared candidate is compiled and linked at the original address using the same judge as
+agent-written C. A strictly better candidate gets a private per-function header and becomes the saved seed. Both
+are committed together only if the target, current seed, header version, symbols and project config still agree
+with the state used for judging. Failed or worse candidates leave the project unchanged. Already-matched functions
+and shared headers are never rewritten by preparation. Extra emitted function bodies are refused because they
+can collide in a whole-game export.
+
+Imported projects use their existing build and objdiff judge. Their reference bodies must compile against that
+build's headers; the app does not inject a reference header into their build setup. An incompatible header needs a
+normal reviewed source/header edit in that project. This limitation is tracked in ROADMAP.md.
+
+Local reference paths, mappings and macro overrides stay in machine settings and are not added to portable project
+config or saved exports. **Accepted candidate C and its declarations are project work**: they are included when you
+save/export that project. Review them before publishing a source-assisted decompilation.
+
+## How the free search uses the result
+
+- Deterministic repairs use measured mismatch evidence to rank applicable families: stack storage/declarations,
+  register lifetimes, call signatures, constants or control flow. No family is excluded solely by this heuristic.
+- Each random turn first tries up to eight cached local ordering probes: adjacent equal-constant stores and
+  shift/add operand order. They share the turn deadline and normal judge, avoiding random drift for these cheap cases.
+- Random decomp-permuter search boosts relevant existing compiler weights by three, retaining every other upstream
+  weight. Focused batches alternate with upstream defaults so a misleading category cannot dominate the turn.
+  The fallback rewrites use the same focus. This changes search order, never the verdict.
+- A fresh baseline compile supplies evidence for the exact seed and context. An in-process bounded cache avoids
+  compiling that same baseline every pass. Every generated candidate still goes through the normal judge.
+- An improved external seed or changed header, effective compiler, target or symbols renews the existing bounded
+  random-turn allowance. The permuter's own improvements do not renew it. Existing counters are adopted on first
+  observation, and claimed functions wait until released. For imported projects, common build manifests and the
+  generated per-function header are tracked too; an explicit new pass/session retries imported preparation trials.
+
+This makes more work automatic without paid model calls. It cannot guarantee all remaining functions: version
+changes, incorrect mappings, unknown data layouts, missing source, assembly and undiscovered compiler settings
+still need investigation. An isolated function match is also not a replacement for a verified whole-image rebuild.

src/decompnet/app/draft_repairs.py

@@ -18,6 +18,7 @@
     near_repairs,
     pattern_repairs,
     register_repairs,
+    stack_repairs,
     storage_scope_repairs,
 )
 
@@ -42,6 +43,9 @@ def variants(source: str, name: str, arch: str, asm: str = '', flags=(), labels=
     offset = opening + 1
     proposals = []
     labels = labels if labels is not None else {}
+    for family, text in (stack_repairs.variants(source, name) if arch in {"mips", "ppc", "x86", "arm", "thumb"} else []):
+        proposals.append(text)
+        labels[text] = family
     for family, text in register_repairs.variants(source, name, arch):
         proposals.append(text)
         labels[text] = family
@@ -66,7 +70,7 @@ def variants(source: str, name: str, arch: str, asm: str = '', flags=(), labels=
             proposals.append(seeds.M2C_MACROS + refined)
             labels[proposals[-1]] = 'm2c_refinement'
     if local := branch_locals.variant(source, name):
-        proposals.append(local)
+        proposals.insert(0, local)
         labels[local] = 'branch_locals'
     for text in storage_scope_repairs.variants(source, name, arch, flags):
         proposals.append(text)

src/decompnet/app/engine/gui.py

@@ -56,8 +56,9 @@ def overview(self, slug: str) -> dict:
         twins = self.twins.jobs.get(proj["slug"])
         base = self._overview(slug, proj)
         from ..libultra import report as libultra_report
+        from ..source_preparation import settings as source_settings
         return base | {"libultra": libultra_report(self, proj), "last_activity": last, "autopilot": self.autopilot_status(slug),
-                       "permuter": self.permuter.status(proj["slug"]),
+                       "permuter": self.permuter.status(proj["slug"]), "source_preparation": source_settings(self, proj),
                        "twins": None if twins is None else {
                            k: getattr(twins, k) for k in ("state", "pairs", "tried", "matched", "asm",
                                                         "cross_candidates", "cross_tried", "cross_matched")},