diff --git a/README.md b/README.md index 582832f6..bbcf9821 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,21 @@ under a tenth of a second (1.4 s before), and agents restart by themselves after stand-in for a model that hands in the demo's own C; the app compiles and judges every function for real.

+## Automatic repairs before paid agents + +The permuter's bounded repair pass tries unused scalar locals, local and parameter `register` storage, and adjacent +local declaration order before random permutation. It keeps original line layout and leaves shared headers alone. +These hypotheses run through the same compiler and exact judge as agents, in ROM and supported 32-bit imported +projects. They use local CPU, with no model API calls. Rejected candidates do not replace the best saved C. + +For close ROM misses it also tests cached compiler flag alternatives, two probes per pass. Later passes skip +already tested choices and can reach optimization levels as well as debug flags. Only exact results qualify for a +narrow compiler-range change. When automatic compiler changes are disabled, the result becomes a suggestion for +you to review. Imported projects retain their own build settings. + +This cannot identify an unknown original-source function or reconstruct a substantially wrong algorithm. Those +still need source-assisted agent work. It also cannot guarantee that a nearly matching function will match. + ## Changing a ROM project's compiler Changing the game's compiler or an address range checks every affected matched function before saving. diff --git a/ROADMAP.md b/ROADMAP.md index 82acae80..23572f22 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -71,6 +71,7 @@ folder, `src/decompnet/consoles//`; a new console adds a column (CONTRIBUTIN | Drafts follow the header, made in the background; `redraft` | ✓ | ✓ | ✓ | ✓ | — | ✓ Ghidra's | — | — | | C++ methods drafted as `Class::method` | ✓ | ✓ | ✓ | ✓ | — | ✓ (RTTI, and classes from the vtables constructors store) | — | — | | Twins: free matches from identical code | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| Local mapped source preparation | ✓ | ✓ | ✓ | ✓ | body only¹ | body only¹ | body only¹ | body only¹ | | Permuter on near misses | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | Hand-written assembly set aside | ✓ | ✓ | ✓ | ✓ | ✓ | Visual C++'s exception-handling stubs | BIOS call stubs, pieces of functions | MIPS, PowerPC, ARM's BIOS stubs and pieces of functions | | Pieces of functions joined to their function | — | — | — | — | — | ✓ (`scratch.merge_pieces`) | ✓ (`recover.merge_pieces`) | — | @@ -834,6 +835,22 @@ Repeated-output limits now remember bounded fingerprints across a claim and rese Function lists show check/improvement recency. Agent work selection supports oldest checked first and a recorded-model untried filter in ROM and imported projects. +The shared deterministic repair pass now tries bounded unused scalar removal, scalar/pointer storage and adjacent local declaration order on supported 32-bit ROM/imported projects, preserving line layout and using normal exact judges. Close ROM misses rotate cached debug and optimization probes, two per pass, with manual compiler approval respected. Imported compiler flags remain owned by their build. + +### Local source preparation and renewed free searches + +¹ Imported builds retain their own declarations headers; reference-header injection remains a planned gap. + +Opt-in Settings → Local source preparation reads explicitly mapped local reference bodies, preserves formatting, +applies supplied numeric retail macros and judges each seed before saving it. ROM mode installs improved seeds +with private per-function declarations; imported mode uses its existing build headers and objdiff judge. Both +modes share bounded preparation, cached trials and diff-focused permutation. External input changes renew bounded +random turns; the permuter's own improvements do not. See [the setup guide](docs/LOCAL_SOURCE_PREPARATION.md). + +Remaining gap: imported projects cannot automatically inject a reference declarations header into the project +build. Their body must already compile against the project's own headers. This needs a reviewed header edit and +regression checking of affected units before it can become automatic. + Agent work selection now reaches small-function batches through the shared scheduler (including percentage, model-history, attempt and lease filters). Paused queues wait without starting a paid session. Cooldowns and @@ -848,3 +865,4 @@ ROM and imported briefs via get_function(section="quality"). Existing notes are These are not semantic proofs: best scores, saved best C and the exact judge remain unchanged. Alternatives must be rejudged in their current context. Custom generated dependencies outside the observed build/include files still need richer build-system dependency records. + diff --git a/docs/LOCAL_SOURCE_PREPARATION.md b/docs/LOCAL_SOURCE_PREPARATION.md new file mode 100644 index 00000000..8e0efe7a --- /dev/null +++ b/docs/LOCAL_SOURCE_PREPARATION.md @@ -0,0 +1,70 @@ +# Free local source preparation and permutation + +The permuter can use explicitly mapped local reference source before its usual deterministic and random searches. +This is optional: ROM-only projects keep working without it. Reference code from another version is a useful seed, +not proof that a function matches the retail game. + +## Set it up + +Open the game's **Settings → Local source preparation**: + +1. Select the reference source folder and any SDK/include folders it needs. These folders are read-only inputs; + the app does not execute reference build scripts or download SDKs. +2. Map each intended destination explicitly, one per line: + `func_80001000 = update_objects @ objects.c`. + The path is relative to the reference folder. Ambiguous/duplicate mappings are refused. Matching a name or a + source location does not establish a match; no source-derived address is added to the destination symbol map. +3. Supply a declarations-only header with the necessary types and prototypes when possible. It must live inside a + selected folder. Otherwise the app tries the mapped file's includes and declarations, or the current project + header if there are no includes. Complex conditional includes, compiler extensions or initialized globals may + need a smaller prepared header. An unresolved include is reported rather than guessed. +4. If retail and debug layouts differ, enter numeric overrides such as `OBJECT_LIMIT=128`. Existing definitions + are replaced before dependent structs are compiled. Overrides do not infer the correct value or add a missing + definition. Add any known global aliases separately: `gReferenceObjects = D_80010000`. +5. Enable preparation and **Save source settings**, then **Try preparation pass**. The button runs a background job + with a progress bar and per-function diagnostics. The permuter also runs a preparation pass before its usual work. + +A pass is bounded to eight compiler attempts and 20 seconds between attempts. One compiler invocation can finish +past that deadline, within the compiler's normal timeout. Unchanged trials (including compiler failures) are +cached. Another pass continues with untried mapped functions. Edited reference bodies, headers, mappings, macro +values or destination build inputs give those candidates another chance. The original function body keeps its +line breaks and original line number: IDO can schedule code differently when source line information changes. +Token replacements leave comments and string literals alone. A mapped `static` entry point is made externally +visible for the isolated compile, without shifting its lines. Its original body remains unchanged. + +## What gets saved + +For ROM projects, a prepared candidate is compiled and linked at the original address using the same judge as +agent-written C. A strictly better candidate gets a private per-function header and becomes the saved seed. Both +are committed together only if the target, current seed, header version, symbols and project config still agree +with the state used for judging. Failed or worse candidates leave the project unchanged. Already-matched functions +and shared headers are never rewritten by preparation. Extra emitted function bodies are refused because they +can collide in a whole-game export. + +Imported projects use their existing build and objdiff judge. Their reference bodies must compile against that +build's headers; the app does not inject a reference header into their build setup. An incompatible header needs a +normal reviewed source/header edit in that project. This limitation is tracked in ROADMAP.md. + +Local reference paths, mappings and macro overrides stay in machine settings and are not added to portable project +config or saved exports. **Accepted candidate C and its declarations are project work**: they are included when you +save/export that project. Review them before publishing a source-assisted decompilation. + +## How the free search uses the result + +- Deterministic repairs use measured mismatch evidence to rank applicable families: stack storage/declarations, + register lifetimes, call signatures, constants or control flow. No family is excluded solely by this heuristic. +- Each random turn first tries up to eight cached local ordering probes: adjacent equal-constant stores and + shift/add operand order. They share the turn deadline and normal judge, avoiding random drift for these cheap cases. +- Random decomp-permuter search boosts relevant existing compiler weights by three, retaining every other upstream + weight. Focused batches alternate with upstream defaults so a misleading category cannot dominate the turn. + The fallback rewrites use the same focus. This changes search order, never the verdict. +- A fresh baseline compile supplies evidence for the exact seed and context. An in-process bounded cache avoids + compiling that same baseline every pass. Every generated candidate still goes through the normal judge. +- An improved external seed or changed header, effective compiler, target or symbols renews the existing bounded + random-turn allowance. The permuter's own improvements do not renew it. Existing counters are adopted on first + observation, and claimed functions wait until released. For imported projects, common build manifests and the + generated per-function header are tracked too; an explicit new pass/session retries imported preparation trials. + +This makes more work automatic without paid model calls. It cannot guarantee all remaining functions: version +changes, incorrect mappings, unknown data layouts, missing source, assembly and undiscovered compiler settings +still need investigation. An isolated function match is also not a replacement for a verified whole-image rebuild. diff --git a/src/decompnet/app/draft_repairs.py b/src/decompnet/app/draft_repairs.py index 020ec638..8e2a86fd 100644 --- a/src/decompnet/app/draft_repairs.py +++ b/src/decompnet/app/draft_repairs.py @@ -18,6 +18,7 @@ near_repairs, pattern_repairs, register_repairs, + stack_repairs, storage_scope_repairs, ) @@ -42,6 +43,9 @@ def variants(source: str, name: str, arch: str, asm: str = '', flags=(), labels= offset = opening + 1 proposals = [] labels = labels if labels is not None else {} + for family, text in (stack_repairs.variants(source, name) if arch in {"mips", "ppc", "x86", "arm", "thumb"} else []): + proposals.append(text) + labels[text] = family for family, text in register_repairs.variants(source, name, arch): proposals.append(text) labels[text] = family @@ -66,7 +70,7 @@ def variants(source: str, name: str, arch: str, asm: str = '', flags=(), labels= proposals.append(seeds.M2C_MACROS + refined) labels[proposals[-1]] = 'm2c_refinement' if local := branch_locals.variant(source, name): - proposals.append(local) + proposals.insert(0, local) labels[local] = 'branch_locals' for text in storage_scope_repairs.variants(source, name, arch, flags): proposals.append(text) diff --git a/src/decompnet/app/engine/gui.py b/src/decompnet/app/engine/gui.py index cb64b817..25ab15ca 100644 --- a/src/decompnet/app/engine/gui.py +++ b/src/decompnet/app/engine/gui.py @@ -56,8 +56,9 @@ def overview(self, slug: str) -> dict: twins = self.twins.jobs.get(proj["slug"]) base = self._overview(slug, proj) from ..libultra import report as libultra_report + from ..source_preparation import settings as source_settings return base | {"libultra": libultra_report(self, proj), "last_activity": last, "autopilot": self.autopilot_status(slug), - "permuter": self.permuter.status(proj["slug"]), + "permuter": self.permuter.status(proj["slug"]), "source_preparation": source_settings(self, proj), "twins": None if twins is None else { k: getattr(twins, k) for k in ("state", "pairs", "tried", "matched", "asm", "cross_candidates", "cross_tried", "cross_matched")}, diff --git a/src/decompnet/app/permute.py b/src/decompnet/app/permute.py index fdfc200c..804010fc 100644 --- a/src/decompnet/app/permute.py +++ b/src/decompnet/app/permute.py @@ -207,6 +207,11 @@ def stop(self, slug: str) -> None: st.stop.set() def candidates(self, proj, limit: int = CANDIDATES) -> list: + from . import permuter_inputs + rows = self.net.store.all("SELECT * FROM functions WHERE project_id=? AND status='open' " + "AND best_source IS NOT NULL AND best_score>0 AND best_score<=n_insns*100*? " + "AND n_insns<=?", proj['id'], NEAR, self.max_insns) + permuter_inputs.refresh(self.net, proj, rows) return self.net.store.all( "SELECT * FROM functions WHERE project_id=? AND status='open' AND best_source IS NOT NULL " "AND best_score > 0 AND best_score <= n_insns * 100 * ? AND n_insns <= ? AND permuted < ? " @@ -222,6 +227,9 @@ def _loop(self, slug: str, st: State) -> None: proj = self.net.project_row(slug) except NetworkError: return # the project is gone + from .source_preparation import run as prepare_sources + prepare_sources(self, proj, st) + proj = self.net.project_row(slug) self._repair_round(agent, proj, st) todo = self.candidates(proj) if not todo: @@ -240,6 +248,8 @@ def repair_pass(self, slug: str, stop: threading.Event) -> State: st = State(stop=stop) if slug not in self.net.links: agent, _ = self.net.local_agent("permuter") + from .source_preparation import run as prepare_sources + prepare_sources(self, self.net.project_row(slug), st) self._repair_round(agent, self.net.project_row(slug), st) return st @@ -268,11 +278,10 @@ def _repair_round(self, agent, proj, st: State, *, function_id=None, budget=32, labels = {} candidates = draft_repairs.variants(fn['best_source'], fn['name'], proj['arch'], fn['asm'] or '', flags=compiler_for(self.net.cfg(proj),fn['vram']).get('flags',[]) if not fork else [], labels=labels) from . import repair_ranking - ranked = repair_ranking.order(dict.fromkeys(labels.values()),repair_ranking.evidence(self.net,fn['id'],agent.id)) + ranked = repair_ranking.order(dict.fromkeys(labels.get(text, 'draft_repair') for text in candidates),repair_ranking.evidence(self.net,fn['id'],agent.id)) from . import agent_tokens effect_scope=agent_tokens.target_scope(self.net,fn,'') if hasattr(self.net,'_context') else None ranked=repair_ranking.demote(self.net,effect_scope,ranked) - candidates.sort(key=lambda text: ranked.index(labels[text]) if labels.get(text) in ranked else len(ranked)) if not self._hold(agent, fn, count_turn=False): continue st.holding(fn['name'], True) @@ -288,6 +297,15 @@ def _repair_round(self, agent, proj, st: State, *, function_id=None, budget=32, context = repr((scope, header, self.net.cfg(proj), symbols, proj['arch'], proj['big_endian'], fn['unit'], fn['target'], fn['relocs'], fn['vram'])) best = fn['best_score'] + if hasattr(self.net, 'keep_diff') and attempts < budget and time.monotonic() < deadline: + measured, compiled = self._baseline(agent, proj, fn, st, fork) + attempts += compiled + if measured == 0: + st.add(improved=1, matched=1) + continue + ranked = repair_ranking.order(dict.fromkeys(labels.get(text, 'draft_repair') for text in candidates), repair_ranking.evidence(self.net, fn['id'], agent.id)) + ranked = repair_ranking.demote(self.net, effect_scope, ranked) + candidates.sort(key=lambda text: ranked.index(labels.get(text, 'draft_repair')) if labels.get(text, 'draft_repair') in ranked else len(ranked)) for candidate in candidates: if st.stop.is_set() or attempts >= budget or time.monotonic() >= deadline: break @@ -370,6 +388,19 @@ def _permute(self, agent, proj, fn, st: State) -> None: start_src = best_src = cur_src = m2cfix.expand_macros(fn["best_source"]) best = cur = fn["best_score"] deadline = time.monotonic() + seconds + from . import permuter_focus, repair_ranking + if hasattr(self.net, 'keep_diff'): + measured, _ = self._baseline(agent, proj, fn, st, fork) + if measured == 0: + st.add(improved=1, matched=1) + return + if hasattr(self.net, 'keep_diff'): + best, best_src = self._local_order(agent, proj, fn, st, fork, deadline, best, best_src) + cur, cur_src = best, best_src + if best == 0: + return + rows = repair_ranking.evidence(self.net, fn['id'], agent.id) + focus, boosts = permuter_focus.classify(rows), permuter_focus.boosts(rows) batches = misses = 0 while time.monotonic() < deadline and best > 0 and not st.stop.is_set(): if gen is not None: @@ -378,8 +409,10 @@ def _permute(self, agent, proj, fn, st: State) -> None: # the demo ROM (at 45); taking turns, every one matched within 12 s base = best_src if batches % 2 else start_src batches += 1 + # Alternate upstream defaults with focused weights: preserve an exploratory lane when + # the advisory mismatch category is incomplete or points at the wrong repair. batch = gen.batch({"header": header, "source": base, "function": fn["name"], "family": family, - "seed": rng.randrange(MAX_SEED), "count": BATCH}) + "seed": rng.randrange(MAX_SEED), "count": BATCH, "weight_boosts": boosts if batches % 2 == 0 else {}}) if batch is None: # C it can't read: the simple rewrites can still try gen.close() gen = None @@ -387,7 +420,7 @@ def _permute(self, agent, proj, fn, st: State) -> None: if not batch: # nothing new left to write from this C break else: # one simple rewrite, mostly of the last C kept - batch = [mutate(cur_src if rng.random() < 0.85 else best_src, rng)] + batch = [mutate(cur_src if rng.random() < 0.85 else best_src, rng, focus)] # none of them applies to this C (lerp's `a + (b - a) * t`): a minute of trying burnt a core misses = 0 if batch[0] else misses + 1 if misses >= REWRITE_MISSES: @@ -402,6 +435,8 @@ def _permute(self, agent, proj, fn, st: State) -> None: continue if score < best: best, best_src = score, cand + rows = repair_ranking.evidence(self.net, fn['id'], agent.id) + focus, boosts = permuter_focus.classify(rows), permuter_focus.boosts(rows) st.add(improved=1) if score == 0: st.add(matched=1) @@ -416,7 +451,7 @@ def _permute(self, agent, proj, fn, st: State) -> None: self._let_go(agent, fn) def _repair_flags(self, agent, proj, fn, st, deadline): - """Two cached debug-flag probes on close ROM misses; persist only a byte-exact function range. + """Two cached compiler-flag probes per pass on close ROM misses; persist only an exact function range. Imported projects own their build flags: never rewrite their build setup from this phase. """ @@ -430,12 +465,15 @@ def _repair_flags(self, agent, proj, fn, st, deadline): count = 0 symbols = [tuple(row) for row in self.net.store.all('SELECT name,address FROM symbols WHERE project_id=? ORDER BY name', proj['id'])] choices = [f for f in self.net.debug_flag_variants(own['toolchain'], list(own['flags'])) if f != own['flags']] - for flags in choices[:2]: - if st.stop.is_set() or time.monotonic() >= deadline: + if hasattr(self.net, 'flag_variants'): + choices += [f for f in self.net.flag_variants(own['toolchain'], list(own['flags'])) + if f != own['flags'] and f not in choices] + for flags in choices: + if count >= 2 or st.stop.is_set() or time.monotonic() >= deadline: break digest = hashlib.sha256(repr((current['target'], current['relocs'], current['best_source'], ctx['header'], self.net.cfg(proj), symbols, flags)).encode()).hexdigest() - key = 'draft-debug-v1:' + digest + key = 'draft-flags-v2:' + digest if self.net.store.one('SELECT 1 FROM reuse_trials WHERE function_id=? AND signature=?', fn['id'], key): continue count += 1 @@ -443,9 +481,14 @@ def _repair_flags(self, agent, proj, fn, st, deadline): exact, _, _ = self.net._judge_with(proj, current, own['toolchain'], flags, current['best_source']) with self.net.store.tx() as db: db.execute('INSERT OR REPLACE INTO reuse_trials(function_id,signature,outcome,reason) VALUES(?,?,?,?)', - (fn['id'], key, 'match' if exact else 'miss', 'near-miss debug flag probe')) + (fn['id'], key, 'match' if exact else 'miss', 'near-miss compiler flag probe')) if exact: - # Changing this function's compiler settings cannot invalidate another function's match. + if hasattr(self.net, 'auto_adjusts') and not self.net.auto_adjusts(self.net.project_row(proj['slug'])): + self.net.suggest(agent, proj['slug'], 'The permuter found an exact match with these compiler flags.', + compiler=own['toolchain'], flags=' '.join(flags), function=current['name'], + ranges=f"0x{fn['vram']:X}-0x{fn['vram']+fn['size']:X}", source=current['best_source']) + break # leave the proven change for the user, as their project setting requests + # Range changes still protect any affected existing match before saving. self.net.set_compiler_ranges(proj['slug'], [[fn['vram'], fn['vram'] + fn['size']]], own['toolchain'], flags) fresh = self.net.project_row(proj['slug']) count += 1 @@ -455,6 +498,60 @@ def _repair_flags(self, agent, proj, fn, st, deadline): break return count + def _local_order(self, agent, proj, fn, st, fork, deadline, best, best_src): + """Eight cached cheap hypotheses, inside the turn's existing time/claim allowance.""" + from . import permuter_inputs, permuter_local, source_preparation + state = (st.started_at, proj['config'], self.net.forks.permuter_header(fork, fn['name']), + permuter_inputs.build_state(proj)) if fork else source_preparation.inputs(self.net, proj, fn) + for family, candidate in permuter_local.variants(fn['best_source'], fn['name']): + if st.stop.is_set() or time.monotonic() >= deadline: + break + key = 'local-order-v1:' + hashlib.sha256(repr((state, candidate)).encode()).hexdigest() + if self.net.store.one('SELECT 1 FROM reuse_trials WHERE function_id=? AND signature=?', fn['id'], key): + continue + st.add(variants=1, repairs=1) + with repair_trial({'kind': 'local order before permutation', 'family': family}): + score = self._score(agent, proj, fn, candidate, best, fork) + with self.net.store.tx() as db: + db.execute('INSERT OR REPLACE INTO reuse_trials(function_id,signature,outcome,reason) VALUES(?,?,?,?)', + (fn['id'], key, str(score), family)) + if score is not None and score < best: + best, best_src = score, candidate + st.add(improved=1) + if score == 0: + st.add(matched=1) + break + return best, best_src + + def _baseline(self, agent, proj, fn, st, fork): + """Fresh evidence once per exact input in this process; a bounded cache avoids repeated baseline compiles.""" + from . import permuter_inputs, source_preparation + header = self.net.forks.permuter_header(fork, fn['name']) if fork else self.net._context(proj['id'], fn['unit'])['header'] + state = (st.started_at, header, proj['config'], fn['best_source'], fn['target'], permuter_inputs.build_state(proj)) if fork else source_preparation.inputs(self.net, proj, fn) + key = (fn['id'], hashlib.sha256(repr(state).encode()).hexdigest()) + cache = getattr(self, '_baselines', {}) + self._baselines = cache + if key in cache: + score, rows = cache[key] + self.net.keep_diff(agent.id, fn['id'], rows) + if not hasattr(self.net, '_agent_outcomes'): + self.net._agent_outcomes = {} + self.net._agent_outcomes[agent.id, fn['id']] = {'source': fn['best_source'], 'score': score, 'rows': rows} + return score, 0 + st.add(variants=1, repairs=1) + # Clear old evidence so a compile/link error cannot leave a misleading diff as today's baseline. + getattr(self.net, '_last_diffs', {}).pop((agent.id, fn['id']), None) + score = self._score(agent, proj, fn, fn['best_source'], fn['best_score'], fork) + rows = getattr(self.net, '_last_diffs', {}).get((agent.id, fn['id']), ([], ''))[0] if score is not None else [] + self.net.keep_diff(agent.id, fn['id'], rows) + if not hasattr(self.net, '_agent_outcomes'): + self.net._agent_outcomes = {} + self.net._agent_outcomes[agent.id, fn['id']] = {'source': fn['best_source'], 'score': score, 'rows': rows} + cache[key] = (score, rows) + while len(cache) > 256: + del cache[next(iter(cache))] + return score, 1 + def _score(self, agent, proj, fn, source: str, best: int, fork: str | None) -> int | None: """A variant's score; recorded (as the function's best attempt, or its match) only when it beats `best`.""" if fork: @@ -470,6 +567,10 @@ def _score(self, agent, proj, fn, source: str, best: int, fork: str | None) -> i d = scoring.diff(self.net._target_obj(fn, proj), cand) if verdict is not None: d = scoring.DiffResult(0 if verdict else max(1, d.score), d.max_score, d.rows, verdict) + self.net.keep_diff(agent.id, fn['id'], d.rows) + if not hasattr(self.net, '_agent_outcomes'): + self.net._agent_outcomes = {} + self.net._agent_outcomes[agent.id, fn['id']] = {'source': source, 'score': d.score, 'rows': d.rows} if d.score < best: self.net._record(fn, agent, d.score, source, ctx["version"], matched=d.bytes_match, via="permuter", effective_settings=getattr(result,"effective_settings",None)) return 0 if d.bytes_match else max(1, d.score) diff --git a/src/decompnet/app/permuter_focus.py b/src/decompnet/app/permuter_focus.py new file mode 100644 index 00000000..d409e5f3 --- /dev/null +++ b/src/decompnet/app/permuter_focus.py @@ -0,0 +1,30 @@ +"""Advisory search weights from the ordinary judge's mismatches; all mutations remain available.""" +import re + +BOOSTS = { + 'stack': ('perm_reorder_decls', 'perm_temp_for_expr', 'perm_split_assignment', 'perm_randomize_internal_type'), + 'register': ('perm_reorder_decls', 'perm_reorder_stmts', 'perm_expand_expr', 'perm_temp_for_expr', 'perm_commutative'), + 'call': ('perm_randomize_function_type', 'perm_randomize_external_type', 'perm_refer_to_var'), + 'constant': ('perm_float_literal', 'perm_cast_simple', 'perm_add_mask', 'perm_xor_zero'), + 'control': ('perm_condition', 'perm_reorder_stmts', 'perm_ins_block', 'perm_empty_stmt'), +} + + +def classify(rows): + changed = [row for row in rows if row and row[0] != ' '] + markers = {row[0] for row in changed} + text = ' '.join(str(cell) for row in changed for cell in row[1:]) + if re.search(r'\b(?:jalr?|callq?|bl|bctrl)\b', text): + return 'call' + if 's' in markers: + return 'stack' + if markers and markers <= {'r'}: + return 'register' + if 'i' in markers or re.search(r'\b(?:lui|li|ldc1|lwc1|movabs)\b', text): + return 'constant' + return 'control' if markers & {'<', '>', '|', 'o'} else '' + + +def boosts(rows): + # Multiply existing upstream weights, never invent unknown mutations or remove the compiler's defaults. + return dict.fromkeys(BOOSTS.get(classify(rows), ()), 3) diff --git a/src/decompnet/app/permuter_inputs.py b/src/decompnet/app/permuter_inputs.py new file mode 100644 index 00000000..1ec3b1e8 --- /dev/null +++ b/src/decompnet/app/permuter_inputs.py @@ -0,0 +1,45 @@ +"""Renew bounded permutation allowances when external inputs change, never for the permuter's own steps.""" +import hashlib +from pathlib import Path + +from ..project import compiler_for + + +def build_state(proj): + """Imported builds own these files; inspect their content without executing their commands.""" + result = [] + for name in ('objdiff.json', 'build.ninja', 'Makefile', 'compile_commands.json', 'config.json'): + path = Path(proj['root']) / name + if path.is_file(): + result.append((name, hashlib.sha256(path.read_bytes()).hexdigest())) + return result + + +def refresh(net, proj, rows): + """Adopt existing counters on first observation; later context/target/build changes reopen only idle work.""" + cfg = net.cfg(proj) + fork = net.fork_slug(proj['slug']) + symbols = [tuple(r) for r in net.store.all('SELECT name,address,size,kind FROM symbols WHERE project_id=? ORDER BY name', proj['id'])] + build = build_state(proj) if fork else [] + changed = [] + for fn in rows: + header = net.forks.permuter_header(fork, fn['name']) if fork else net._context(proj['id'], fn['unit'])['header'] + own = cfg.get('compiler') if fork else compiler_for(cfg, fn['vram']) + fields = (header, own, symbols, build, fn['unit'], fn['vram'], fn['target'], fn['relocs'], fn['name']) + digest = hashlib.sha256(repr(fields).encode()).hexdigest() + key = f'permuter-inputs-v1:{fn["id"]}' + with net.store.tx() as db: + old = db.execute('SELECT value FROM settings WHERE key=?', (key,)).fetchone() + if old and old['value'] != digest: + if db.execute('SELECT 1 FROM claims WHERE function_id=?', (fn['id'],)).fetchone(): + continue # retry after the current turn, without consuming its future allowance + db.execute("UPDATE functions SET permuted=0 WHERE id=? AND status='open'", (fn['id'],)) + changed.append(fn['id']) + db.execute('INSERT OR REPLACE INTO settings VALUES(?,?)', (key, digest)) + return changed + + +def external_seed(db, current, source, agent_name): + """Called only for an improving saved candidate. Identical text and our own hill climbing keep their cap.""" + if agent_name != 'permuter' and source != current['best_source']: + db.execute("UPDATE functions SET permuted=0 WHERE id=? AND status='open'", (current['id'],)) diff --git a/src/decompnet/app/permuter_local.py b/src/decompnet/app/permuter_local.py new file mode 100644 index 00000000..d4111fd9 --- /dev/null +++ b/src/decompnet/app/permuter_local.py @@ -0,0 +1,32 @@ +"""Cheap local ordering hypotheses before random drift; only the normal judge can accept one.""" +import re + +from ..forks.csource import _blank, find + + +def variants(source, name): + span = find(source, name) + if not span: + return [] + clean = _blank(source) + opening = clean.find('{', *span) + if opening < 0 or re.search(r'\bvolatile\b|^\s*#', clean[span[0]:span[1]], re.M): + return [] + result = [] + # Adjacent constant stores through the same pointer often differ only in scheduling. Do not + # cross statements, branches, indentation levels or a call; preserve the original lines verbatim. + store = re.compile(r'(?m)^([ \t]+)([A-Za-z_]\w*)->([A-Za-z_]\w*)\s*=\s*' + r'(-?(?:0[xX][0-9a-fA-F]+|[0-9]+(?:\.[0-9]+)?)[uUlLfF]*)\s*;[ \t]*$') + matches = list(store.finditer(clean, opening, span[1])) + for left, right in zip(matches, matches[1:]): + if (clean[left.end():right.start()] != '\n' or (left[1], left[2]) != (right[1], right[2]) + or left[3] == right[3] or left[4] != right[4]): + continue + result.append(('adjacent_constant_stores', source[:left.start()] + source[right.start():right.end()] + '\n' + + source[left.start():left.end()] + source[right.end():])) + # A common integer multiply-by-33 idiom: operand order can choose which value occupies the + # destination register. Both operands remain once each; the compiler/link judge remains mandatory. + for match in re.finditer(r'\b([A-Za-z_]\w*)\s*\+\s*\(\s*\1\s*<<\s*([0-9]+)\s*\)', clean[opening:span[1]]): + a, b = opening + match.start(), opening + match.end() + result.append(('shift_add_order', source[:a] + f'({match[1]} << {match[2]}) + {match[1]}' + source[b:])) + return result[:8] diff --git a/src/decompnet/app/repair_ranking.py b/src/decompnet/app/repair_ranking.py index bb6cac13..b2001dcd 100644 --- a/src/decompnet/app/repair_ranking.py +++ b/src/decompnet/app/repair_ranking.py @@ -1,10 +1,16 @@ """Rank applicable hypotheses by observed mismatches without excluding other families.""" def order(families, rows): + from .permuter_focus import classify + focus = classify(rows) markers = {row[0] for row in rows if row[0] != ' '} - if markers and markers <= {'r', 's'}: - preferred = ('stack_array', 'parameter_width', 'declaration_order') if 's' in markers else ( - 'comparison_reuse', 'integer_address_order', 'typed_temporary', 'value_lifetime', 'split_scalar_lifetime', 'array_index_width', 'declaration_order') + if focus == 'call': + preferred = ('call_arity', 'call_result_inline', 'parameter_width', 'parameter_storage') + elif markers and markers <= {'r', 's'}: + preferred = ('unused_scalar_local', 'local_storage', 'parameter_storage', 'local_declaration_order', + 'stack_array', 'parameter_width', 'declaration_order') if 's' in markers else ( + 'comparison_reuse', 'integer_address_order', 'typed_temporary', 'value_lifetime', + 'split_scalar_lifetime', 'array_index_width', 'declaration_order') elif markers & {'<', '>', '|', 'o'}: preferred = ('paired_load_order', 'pointer_before_store', 'branch_orientation', 'loop_induction', 'call_result_inline', 'parameter_width') elif 'i' in markers: diff --git a/src/decompnet/app/routes/projects.py b/src/decompnet/app/routes/projects.py index 5f9b84a5..809f29bc 100644 --- a/src/decompnet/app/routes/projects.py +++ b/src/decompnet/app/routes/projects.py @@ -221,6 +221,34 @@ def p_boundary(slug, request, data): net.resolve_boundary(slug, int(data.get("id", 0)), data.get("action") == "apply") return net.overview(slug) + def p_source_preparation(slug, request, data): + from .. import source_preparation + from ..permute import State + proj = net.project_row(slug) + if request.method == "GET": + return source_preparation.settings(net, proj) + if data.get('action') not in (None, 'save', 'run'): + raise ValueError('Choose save or run for source preparation.') + if data.get('action') == 'run': + def work(log): + report = source_preparation.run(net.permuter, net.project_row(slug), State(), log) + log(f"Prepared {report['tested']}: {report['improved']} improved, {report['matched']} matched.") + return report + return {'job': jobs.start('source-preparation', work)} + return source_preparation.configure(net, proj, data) + + def p_permuter(slug, request, data): + """Run free local permutations without starting a paid helper or the full autopilot.""" + net.project_row(slug) + action = data.get("action", "start") + if action not in {"start", "stop"}: + raise ValueError("Choose start or stop for the permuter.") + if action == "stop": + net.permuter.stop(slug) + else: + net.permuter.start(slug) + return {"permuter": net.permuter.status(slug)} + def p_autopilot(slug, request, data): if data.get("action") == "stop": net.stop_autopilot(slug) @@ -295,6 +323,8 @@ def p_join_rom(slug, request, data): "/api/delete": (p_delete, ["POST"]), "/api/try": (p_try, ["POST"]), "/api/release": (p_release, ["POST"]), "/api/reopen": (p_reopen, ["POST"]), "/api/setaside": (p_set_aside, ["POST"]), "/api/recheck": (p_recheck, ["POST"]), + "/api/permuter": (p_permuter, ["POST"]), + "/api/source-preparation": (p_source_preparation, ["GET", "POST"]), "/api/autopilot": (p_autopilot, ["POST"]), "/api/instructions": (p_instructions, ["POST"]), "/api/title": (p_title, ["POST"]), "/api/cross-game": (lambda s, r, d: {"results": net.twins.cross_game_results(s), "suggestions": net.twins.suggestions(s, r.query_params.get("function", "")) diff --git a/src/decompnet/app/source_preparation.py b/src/decompnet/app/source_preparation.py new file mode 100644 index 00000000..3f1be474 --- /dev/null +++ b/src/decompnet/app/source_preparation.py @@ -0,0 +1,321 @@ +"""Read-only, opt-in local source seeds. A mapping is a hypothesis; only the ordinary judge is progress. + +Paths and mappings live in machine settings, not the portable project's config. No source build script runs. +ROM candidates get a private per-function header only on improvement. Imported projects keep their own build +headers: candidates must compile in that build, so an incompatible reference header is a reported limitation. +""" +import hashlib +import json +import logging +import re +import time +from pathlib import Path + +from .. import scoring +from ..forks import csource +from ..network import NetworkError +from ..repair_feedback import trial as repair_trial + +log = logging.getLogger('decompdecomp') +MAX_FILES, MAX_BYTES = 512, 16 * 1024 * 1024 +MAX_HEADER = 1024 * 1024 +_IDENT = r'[A-Za-z_]\w*' +_TOKEN = re.compile(r'//[^\n]*|/\*.*?\*/|"(?:\\.|[^"\\])*"|\'(?:\\.|[^\'\\])*\'|\b' + _IDENT + r'\b', re.S) +_INCLUDE = re.compile(r'^\s*#\s*include\s*[<"]([^>"\n]+)[>"]\s*$', re.M) + + +def settings(net, proj): + row = net.store.one('SELECT value FROM settings WHERE key=?', f'source-preparation:{proj["id"]}') + return json.loads(row['value']) if row else {} + + +def mappings(text): + result, targets, aliases = [], set(), {} + for line in text.splitlines(): + if not line.strip() or line.lstrip().startswith('#'): + continue + match = re.fullmatch(rf'\s*({_IDENT})\s*=\s*({_IDENT})\s*@\s*(.+?)\s*', line) + if not match: + raise NetworkError('Use one mapping per line: destination = source_symbol @ relative/file.c') + destination, source, file = match.groups() + if destination in targets or (source in aliases and aliases[source] != destination): + raise NetworkError('Each destination and source symbol must have one unambiguous mapping.') + targets.add(destination) + aliases[source] = destination + result.append((destination, source, file)) + if len(result) > MAX_FILES: + raise NetworkError('Use at most 512 source mappings per project.') + return result + + +def overrides(text): + result = {} + for line in text.splitlines(): + if not line.strip(): + continue + match = re.fullmatch(rf'\s*({_IDENT})\s*=\s*(-?(?:0[xX][0-9a-fA-F]+|[0-9]+)[uUlL]*)\s*', line) + if not match: + raise NetworkError('Macro overrides must be numeric, for example OBJECT_LIMIT=128.') + result[match[1]] = match[2] + return result + + +def symbol_aliases(text): + result = {} + for line in text.splitlines(): + if not line.strip(): + continue + match = re.fullmatch(rf'\s*({_IDENT})\s*=\s*({_IDENT})\s*', line) + if not match or match[1] in result: + raise NetworkError('Use one unambiguous symbol alias per line: reference_name = destination_name.') + result[match[1]] = match[2] + return result + + +def configure(net, proj, data): + cfg = {key: str(data.get(key, '')).strip() for key in ('root', 'header', 'include_roots', 'mappings', 'overrides', 'aliases')} + cfg['enabled'] = bool(data.get('enabled')) + if any(len(value) > 128_000 for value in cfg.values() if isinstance(value, str)): + raise NetworkError('Source preparation settings are too large; choose a smaller mapping.') + mappings(cfg['mappings']) + overrides(cfg['overrides']) + symbol_aliases(cfg['aliases']) + if cfg['enabled']: + reader = Reader(cfg) + for _, _, file in mappings(cfg['mappings']): + reader.file(reader.root / file) + if cfg['header']: + reader.file(Path(cfg['header']).expanduser()) + with net.store.tx() as db: + db.execute('INSERT OR REPLACE INTO settings VALUES(?,?)', + (f'source-preparation:{proj["id"]}', json.dumps(cfg))) + return cfg + + +class Reader: + """Bound all reads to the explicitly selected roots, including symlink targets and recursive includes.""" + def __init__(self, cfg): + if not cfg.get('root'): + raise NetworkError('Choose a local reference source folder first.') + self.root = Path(cfg['root']).expanduser().resolve() + self.roots = [self.root] + [Path(p).expanduser().resolve() for p in cfg.get('include_roots', '').splitlines() if p.strip()] + if not all(p.is_dir() for p in self.roots): + raise NetworkError('The reference folder and each include root must be existing folders.') + self.overrides = overrides(cfg.get('overrides', '')) + self.cache, self.bytes = {}, 0 + + def file(self, path): + path = path.resolve() + if not any(path.is_relative_to(root) for root in self.roots): + raise NetworkError('A reference file leaves the selected source/include folders. Add its folder explicitly.') + if path.suffix.lower() not in {'.c', '.h', '.cc', '.cpp', '.hpp', '.inc'} or not path.is_file(): + raise NetworkError('Choose an existing C/C++ source or declarations header.') + if path not in self.cache: + if len(self.cache) >= MAX_FILES or path.stat().st_size + self.bytes > MAX_BYTES: + raise NetworkError('Reference reads exceed 512 files / 16 MB. Choose a smaller source subset.') + text = path.read_bytes() + self.bytes += len(text) + self.cache[path] = text.decode('utf-8', errors='replace').replace('\r\n', '\n') + return self.cache[path] + + def flatten(self, path, seen=None): + seen = set() if seen is None else seen + path = path.resolve() + if path in seen: + return '' # guards/cycles do not warrant an unbounded include walk + if len(seen) >= 64: + raise NetworkError('Reference includes exceed 64 levels; supply a prepared declarations header.') + seen.add(path) + text = self.file(path) + def include(match): + name = match[1] + if Path(name).is_absolute() or '..' in Path(name).parts: + raise NetworkError('Use relative includes within the selected reference folders.') + candidates = [path.parent / name] + [root / name for root in self.roots] + found = next((p for p in candidates if p.is_file()), None) + if found is None: + raise NetworkError(f'Include {name!r} was not found. Add its SDK folder as an include root or supply a prepared declarations header.') + return self.flatten(found, seen) + text = _INCLUDE.sub(include, text) + # Apply retail values where they are defined, before any dependent struct declarations. + for name, value in self.overrides.items(): + pattern = re.compile(r'^(\s*#\s*define\s+' + re.escape(name) + r')[ \t]+(?:[^\n\\]|\\\n)*', re.M) + text = pattern.sub(lambda m, value=value: m[1] + ' ' + value, text) + if len(text.encode()) > MAX_HEADER: + raise NetworkError('Flattened declarations exceed 1 MB; supply a smaller declarations header.') + seen.remove(path) + return text + + +def rename(text, aliases): + return _TOKEN.sub(lambda m: aliases.get(m[0], m[0]), text) + + +def declarations(text): + """Remove function bodies without collapsing their original line breaks. This is deliberately conservative.""" + clean, out, start, depth, at = csource._blank(text), [], None, 0, 0 + for i, char in enumerate(clean): + if char == '{': + if depth == 0 and clean[:i].rstrip().endswith(')'): + start = i + depth += 1 + elif char == '}': + depth -= 1 + if depth == 0 and start is not None: + out.extend((text[at:start], ';' + re.sub(r'[^\n]', ' ', text[start + 1:i + 1]))) + at, start = i + 1, None + return ''.join(out) + text[at:] + + +def prepare(reader, cfg, mapping, aliases, fallback): + destination, symbol, file = mapping + path = reader.root / file + original = reader.file(path) + span = csource.find(original, symbol) + if not span or csource.is_stub(original[span[0]:span[1]]): + raise NetworkError(f'No C definition of {symbol!r} in its mapped file. Check the mapping.') + if csource.find(original[span[1]:], symbol): + raise NetworkError(f'More than one definition of {symbol!r} is present. Supply a source subset with the intended version.') + # Keep the exact body formatting and its original line number, including __LINE__ uses. + body = original[span[0]:span[1]] + # An unused static definition can disappear entirely when compiled alone. Expose only this mapped + # entry point; retain columns/line breaks and let the normal destination judge settle the result. + clean = csource._blank(body) + name_at = clean.find(symbol) + if name_at >= 0: + for match in reversed(list(re.finditer(r'\bstatic\b', clean[:name_at]))): + body = body[:match.start()] + ' ' * len(match[0]) + body[match.end():] + source = f'#line {original[:span[0]].count(chr(10)) + 1} "reference.c"\n' + rename(body, aliases) + if cfg.get('header'): + header = reader.flatten(Path(cfg['header']).expanduser()) + elif '#include' in original[:span[0]]: + header = declarations(reader.flatten(path.resolve())) + else: + header = fallback + return source, rename(header, aliases) + + +def inputs(net, proj, fn): + """The destination state that a private-header compile was judged against, checked again before promotion.""" + ctx = net._context(proj['id'], fn['unit']) + symbols = [tuple(r) for r in net.store.all('SELECT name,address,size,kind FROM symbols WHERE project_id=? ORDER BY name', proj['id'])] + return (proj['config'], fn['unit'], fn['target'], fn['relocs'], fn['vram'], fn['best_source'], ctx['version'], symbols) + + +def promote_context(net, db, fn, expected, header, agent): + """Called inside _record's transaction, after an ordinary successful compile/link, before writing the attempt.""" + proj = db.execute('SELECT * FROM projects WHERE id=?', (fn['project_id'],)).fetchone() + claim = db.execute('SELECT agent_id,expires_at FROM claims WHERE function_id=?', (fn['id'],)).fetchone() + if not claim or claim['agent_id'] != agent.id or claim['expires_at'] <= time.time(): + raise NetworkError('The function claim changed during preparation. Run the pass again.') + if fn['status'] != 'open' or inputs(net, proj, fn) != expected: + raise NetworkError('The function, header or build inputs changed during preparation. Run the pass again.') + version = net._next_version(db, proj['id']) + unit = f'local_source_{fn["id"]}' + db.execute('INSERT INTO context_versions(project_id,version,header,agent_id,message,created_at,unit) VALUES(?,?,?,?,?,?,?)', + (proj['id'], version, header, agent.id, 'Judge-verified local reference declarations', time.time(), unit)) + db.execute('UPDATE functions SET unit=?, permuted=0 WHERE id=?', (unit, fn['id'])) + net._event(db, proj['id'], agent.id, 'context', f'Local reference declarations for {fn["name"]}') + return version + + +def attempt(permuter, proj, fn, source, header, agent, fork): + net = permuter.net + if fork: + # The existing build must supply its own compatible types. Never edit its build setup or headers. + return net.forks.attempt(agent, fork, fn['name'], source, keep=False, quiet=fn['best_score']).get('score') + expected = inputs(net, proj, fn) + result = net._compile(proj, header, source, fn) + if not result.ok: + raise NetworkError('Prepared C did not compile: ' + str(result.log)[-1200:]) + from ..rom.export import _single_definition + if net.is_rom(proj) and not _single_definition(result.obj, fn['name']): + raise NetworkError('Prepared declarations emitted extra function bodies. Supply a declarations-only header.') + candidate, verdict, problems = net._evaluate(fn, proj, result.obj) + if candidate is None: + raise NetworkError('Prepared C could not be linked: ' + '; '.join(problems)[:1200]) + diff = scoring.diff(net._target_obj(fn, proj), candidate) + if verdict is not None: + diff = scoring.DiffResult(0 if verdict else max(1, diff.score), diff.max_score, diff.rows, verdict) + score = 0 if diff.bytes_match else max(1, diff.score) + if fn['best_score'] is None or score < fn['best_score']: + net._record(fn, agent, score, source, 0, matched=diff.bytes_match, via='local-source', + effective_settings=getattr(result, 'effective_settings', None), + prepared_header=header, prepared_inputs=expected) + net.keep_diff(agent.id, fn['id'], diff.rows) + return score + + +def run(permuter, proj, st, log_line=lambda line: None, budget=8, seconds=20): + """A bounded, cached preparation round before ordinary deterministic/random permutations.""" + net = permuter.net + cfg = settings(net, proj) + report = {'tested': 0, 'improved': 0, 'matched': 0, 'errors': []} + if not cfg.get('enabled') or proj['slug'] in net.links: + return report + try: + reader = Reader(cfg) + mapped = mappings(cfg.get('mappings', '')) + aliases = symbol_aliases(cfg.get('aliases', '')) + for dest, source, _ in mapped: + if source in aliases and aliases[source] != dest: + raise NetworkError('A function mapping conflicts with a symbol alias. Keep one destination per source symbol.') + aliases[source] = dest + agent, _ = net.local_agent('autopilot') + fork = net.fork_slug(proj['slug']) + deadline = time.monotonic() + seconds + for mapping in mapped: + if st.stop.is_set() or report['tested'] >= budget or time.monotonic() >= deadline: + break + fn = net.store.one("SELECT * FROM functions WHERE project_id=? AND name=? AND status='open'", proj['id'], mapping[0]) + if not fn or not permuter._hold(agent, fn, count_turn=False, seconds=seconds): + continue + st.holding(fn['name'], True) + cache_key = None + try: + fresh = net.project_row(proj['slug']) + fn = net.store.one('SELECT * FROM functions WHERE id=?', fn['id']) + if fn['status'] != 'open': + continue + fallback = net.forks.permuter_header(fork, fn['name']) if fork else net._context(proj['id'], fn['unit'])['header'] + source, header = prepare(reader, cfg, mapping, aliases, fallback) + # Imported build files can change independently: retry on a new explicit pass/session. + from .permuter_inputs import build_state + state = (st.started_at, fresh['config'], build_state(fresh)) if fork else inputs(net, fresh, fn) + digest = hashlib.sha256(repr((state, source, header, cfg)).encode()).hexdigest() + cache_key = 'local-source-v1:' + digest + if net.store.one('SELECT 1 FROM reuse_trials WHERE function_id=? AND signature=?', fn['id'], cache_key): + continue + report['tested'] += 1 + st.add(variants=1, repairs=1) + log_line(f'Preparing {fn["name"]} ({report["tested"]}/{budget})') + with repair_trial({'kind': 'local source preparation', 'source_sha256': hashlib.sha256(source.encode()).hexdigest(), + 'header_sha256': hashlib.sha256(header.encode()).hexdigest()}): + score = attempt(permuter, fresh, fn, source, header, agent, fork) + if score is not None and (fn['best_score'] is None or score < fn['best_score']): + report['improved'] += 1 + st.add(improved=1) + if score == 0: + report['matched'] += 1 + st.add(matched=1) + with net.store.tx() as db: + db.execute('INSERT OR REPLACE INTO reuse_trials(function_id,signature,outcome,reason) VALUES(?,?,?,?)', + (fn['id'], cache_key, str(score), 'local source preparation')) + except (NetworkError, OSError, ValueError) as exc: + message = f'{fn["name"]}: {exc}' + report['errors'].append(message) + log_line(message) + # Compile failures are cached too, but a changed destination remains retryable. + if cache_key and not str(exc).startswith(('The function, header or build inputs changed', 'The function claim changed')): + with net.store.tx() as db: + db.execute('INSERT OR REPLACE INTO reuse_trials(function_id,signature,outcome,reason) VALUES(?,?,?,?)', + (fn['id'], cache_key, 'error', str(exc)[:1200])) + finally: + st.holding(fn['name'], False) + permuter._let_go(agent, fn) + except (NetworkError, OSError, ValueError) as exc: + report['errors'].append(str(exc)) + log_line(str(exc)) + for message in report['errors']: + log.info('permuter preparation: %s', message) + return report diff --git a/src/decompnet/app/stack_repairs.py b/src/decompnet/app/stack_repairs.py new file mode 100644 index 00000000..516f036a --- /dev/null +++ b/src/decompnet/app/stack_repairs.py @@ -0,0 +1,85 @@ +"""Small storage and declaration hypotheses for excess stack frames, before random permutation. + +Keep source layout and shared headers intact. These are candidates, not equivalence proofs: every result must +pass the ordinary compiler and linked/object judge. The shared draft pass reaches ROM and imported projects. +""" +import re + +from ..forks.csource import _blank, find + +IDENT = r'[A-Za-z_]\w*' +SCALAR = r'(?:s8|u8|s16|u16|s32|u32|s64|u64|f32|f64|char|short|int|long|float|double|unsigned\s+(?:char|short|int|long)|signed\s+(?:char|short|int|long))' +DECL = re.compile(r'\s*(?Pregister\s+)?(?P(?:' + SCALAR + r'\s*\**|'+IDENT+r'\s*\*+))(?:\s+|(?<=\*))(?P'+IDENT+r')\s*;') +LIMIT = 16 + + +def uses(text, name): + """A field with the same spelling is not a use of the local variable.""" + return [m for m in re.finditer(r'\b'+re.escape(name)+r'\b', text) + if not text[:m.start()].rstrip().endswith(('.', '->'))] + + +def variants(source, name): + span = find(source, name) + if not span: + return [] + clean = _blank(source) + signature = re.search(r'\b'+re.escape(name)+r'\s*\((?P[^{};]*)\)\s*{', clean[span[0]:span[1]]) + if not signature: + return [] + opening = span[0]+signature.end()-1 + body = clean[opening+1:span[1]] + if '#' in body or re.search(r'\b(?:asm|__asm__|__asm|goto)\b', body): + return [] + declarations, pos = [], 0 + while match := DECL.match(body, pos): + declarations.append(match) + pos = match.end() + if not declarations: + return [] # do not spend repair trials on parameter-only wrappers with no local storage + result = [] + + def emit(family, edits): + text = source + for a, b, value in sorted(edits, reverse=True): + text = text[:a]+value+text[b:] + if text != source and text not in [s for _, s in result]: + result.append((family, text)) + + # Only ordinary, uninitialized declarations in the leading local block. Blank their bytes rather + # than deleting lines; a historical compiler can schedule differently when source layout changes. + unused = [] + for decl in declarations: + if len(uses(body, decl['name'])) == 1: + a, b = opening+1+decl.start(), opening+1+decl.end() + unused.append((a,b,''.join(char if clean[a+i].isspace() else ' ' for i,char in enumerate(source[a:b])))) + emit('unused_scalar_local', [unused[-1]]) + if len(unused) > 1: + emit('unused_scalar_local', unused) + for decl in declarations[:6]: + var = decl['name'] + if re.search(r'&\s*'+re.escape(var)+r'\b', body): + continue + a = opening+1+(decl.start('storage') if decl['storage'] else decl.start('type')) + b = opening+1+decl.end('storage') if decl['storage'] else a + emit('local_storage', [(a,b,'' if decl['storage'] else 'register ')]) + # Parameter homes can account for a frame mismatch too. Do not try register for address-taken parameters. + params = signature['params'] + for param in list(re.finditer(r'(?:^|,)\s*(?Pregister\s+)?(?:'+SCALAR+r'\s*\**|'+IDENT+r'\s*\*+)(?:\s+|(?<=\*))(?P'+IDENT+r')\s*(?=,|$)', params))[:3]: + if re.search(r'&\s*'+re.escape(param['name'])+r'\b', body): + continue + base = span[0]+signature.start('params') + if param['storage']: + a,b = base+param.start('storage'),base+param.end('storage') + else: + a = base+param.start()+len(param[0])-len(param[0].lstrip(' ,\t\r\n')) + b = a + emit('parameter_storage', [(a,b,'' if param['storage'] else 'register ')]) + # Adjacent uninitialized locals only, not statements, initialized objects or nested scopes. + for first, second in list(zip(declarations, declarations[1:]))[:3]: + a,b = opening+1+first.start('type'),opening+1+first.end() + c,d = opening+1+second.start('type'),opening+1+second.end() + if first['storage'] or second['storage']: + continue + emit('local_declaration_order', [(a,b,source[c:d]),(c,d,source[a:b])]) + return result[:LIMIT] diff --git a/src/decompnet/app/static/index.html b/src/decompnet/app/static/index.html index d1f03b8d..ee06ca69 100644 --- a/src/decompnet/app/static/index.html +++ b/src/decompnet/app/static/index.html @@ -368,6 +368,23 @@

ROM map

+
+

Local source preparation

optional · no AI tokens
+
+

Have reference source you want to use? Map its functions explicitly. The app reads it without changing it, preserves line breaks and tries the ordinary compiler/link judge. A mapping alone never counts as a match.

+ + + + + + + +

Reference paths and mappings stay on this machine; accepted C and declarations are included in saved/exported work. Use a declarations header with the retail layouts, not debug settings. Imported projects keep their existing build headers; the reference body must compile against those. Each preparation pass is limited to 8 compiles / 20 seconds; unchanged trials are skipped.

+
+
+
+
+

Compiler

diff --git a/src/decompnet/app/static/js/steering.js b/src/decompnet/app/static/js/steering.js index da0f16db..aacd415d 100644 --- a/src/decompnet/app/static/js/steering.js +++ b/src/decompnet/app/static/js/steering.js @@ -458,6 +458,9 @@ async function renderProjectSettings() { renderCompilerWaiting(ov); loadCompilerAnswer(); } + const prep = ov.source_preparation || {}; + $("sourcePrepEnabled").checked = !!prep.enabled; + for (const [id, key] of [["Root", "root"], ["Header", "header"], ["Includes", "include_roots"], ["Mappings", "mappings"], ["Overrides", "overrides"], ["Aliases", "aliases"]]) $("sourcePrep" + id).value = prep[key] || ""; await renderSdkSelection(); const facts = [["Platform", ov.platform_name], ["Game", ov.rom?.title || ov.title], [fork ? "Version" : "Game code", ov.rom?.game_code || ov.version], !fork && ov.rom?.region ? ["Region", ov.rom.region] : null, fork ? ["Layout", ov.origin === "scratch" ? (ov.layout || "decomp") + " (from scratch)" : ov.fork_kind] : ["Compiler", `${ov.compiler.name} ${ov.compiler.flags.join(" ")}`], @@ -782,3 +785,25 @@ function renderCompilerImprovements(slug, rows, filter = "improved") { } catch (err) { status.textContent = err.message; } finally { finish(); } }; } + + +$("sourcePrepSave").onclick = async () => { + const log = $("sourcePrepLog"), button = $("sourcePrepSave"); + const finish = beginWork(log, "Saving reference source settings", [button], "Reference source settings saved."); + try { + const body = { slug: S.slug, enabled: $("sourcePrepEnabled").checked }; + for (const [id, key] of [["Root", "root"], ["Header", "header"], ["Includes", "include_roots"], ["Mappings", "mappings"], ["Overrides", "overrides"], ["Aliases", "aliases"]]) body[key] = $("sourcePrep" + id).value; + const saved = await api("/api/source-preparation", { method: "POST", body }); + if (S.ov) S.ov.source_preparation = saved; + } catch (e) { showJobError(log, e.message); } finally { finish(); } +}; + +$("sourcePrepRun").onclick = async () => { + const log = $("sourcePrepLog"), button = $("sourcePrepRun"); + const finish = beginWork(log, "Preparing local source candidates", [button], "Local source preparation finished."); + try { + const { job } = await api("/api/source-preparation", { method: "POST", body: { slug: S.slug, action: "run" } }); + await followJob(job, log); + refreshMap(); + } catch (e) { showJobError(log, e.message); } finally { finish(); } +}; diff --git a/src/decompnet/network.py b/src/decompnet/network.py index 39666743..caa28000 100644 --- a/src/decompnet/network.py +++ b/src/decompnet/network.py @@ -1409,7 +1409,8 @@ def dependents_note(self, function_id: int) -> str: f"claim_function(function=\"{rows[0]['name']}\") takes one next.") def _record(self, fn, agent: Agent, score: int | None, source: str, ctx_version: int, matched: bool = False, - compiled: bool | None = None, via: str = "agent", effective_settings=None): + compiled: bool | None = None, via: str = "agent", effective_settings=None, + prepared_header=None, prepared_inputs=None): now = time.time() compiled = score is not None if compiled is None else compiled with self.store.tx() as db: @@ -1419,7 +1420,13 @@ def _record(self, fn, agent: Agent, score: int | None, source: str, ctx_version: config = json.loads(db.execute("SELECT config FROM projects WHERE id=?", (fn["project_id"],)).fetchone()["config"]) used = (effective_settings or {}).get("compiler") own = compiler_for(config, current["vram"]) if used else None - stale = bool(used and any(used.get(k) != own.get(k) for k in ("toolchain", "flags"))) + stale = bool(score is not None and used and any(used.get(k) != own.get(k) for k in ("toolchain", "flags"))) + if prepared_header is not None: + from .app.source_preparation import promote_context + if stale or score is None or (current['best_score'] is not None and score >= current['best_score']): + raise NetworkError("The prepared candidate no longer improves this function. Run the pass again.") + ctx_version = promote_context(self, db, current, prepared_inputs, prepared_header, agent) + current = db.execute("SELECT * FROM functions WHERE id=?", (fn['id'],)).fetchone() attempt = db.execute("INSERT INTO attempts(function_id,agent_id,score,compiled,context_version,source,created_at,via) " "VALUES(?,?,?,?,?,?,?,?)", (fn["id"], agent.id, score, compiled, ctx_version, source, now, via)).lastrowid self.repair_feedback.attempt(db, current, attempt, ctx_version, effective_settings) @@ -1427,9 +1434,12 @@ def _record(self, fn, agent: Agent, score: int | None, source: str, ctx_version: self.repair_feedback.capture(db, current, source, agent.id, via, ctx_version, attempt) db.execute("UPDATE functions SET attempts=attempts+1 WHERE id=?", (fn["id"],)) if score is not None and not stale: - db.execute("UPDATE functions SET best_score=?, best_source=?, best_agent_id=? " - "WHERE id=? AND status='open' AND (best_score IS NULL OR best_score > ?)", - (score, source, agent.id, fn["id"], score)) + improved = db.execute("UPDATE functions SET best_score=?, best_source=?, best_agent_id=? " + "WHERE id=? AND status='open' AND (best_score IS NULL OR best_score > ?)", + (score, source, agent.id, fn["id"], score)).rowcount + if improved: + from .app.permuter_inputs import external_seed + external_seed(db, current, source, agent.name) if matched and not stale: db.execute("UPDATE functions SET status='matched', matched_source=?, matched_agent_id=?, " "matched_at=?, best_score=0, best_source=? WHERE id=?", diff --git a/src/decompnet/permuter.py b/src/decompnet/permuter.py index 54a02aed..b66b0763 100644 --- a/src/decompnet/permuter.py +++ b/src/decompnet/permuter.py @@ -116,10 +116,14 @@ def _add_cast(lines, rng): MUTATIONS = [_swap_statements, _shuffle_declarations, _retype_local, _toggle_register, _rewrite, _rewrite, _add_cast] -def mutate(source: str, rng: random.Random) -> str | None: +def mutate(source: str, rng: random.Random, focus: str = "") -> str | None: lines = source.split("\n") + preferred = {"stack": [_shuffle_declarations, _toggle_register], + "register": [_swap_statements, _shuffle_declarations], + "call": [_retype_local, _add_cast], "constant": [_rewrite, _add_cast], + "control": [_rewrite, _swap_statements]}.get(focus, []) for _ in range(6): - out = rng.choice(MUTATIONS)(list(lines), rng) + out = rng.choice(MUTATIONS + preferred * 2)(list(lines), rng) if out is not None and out != lines: return "\n".join(out) return None diff --git a/src/decompnet/permuter_gen.py b/src/decompnet/permuter_gen.py index 534f2011..da3cb54e 100644 --- a/src/decompnet/permuter_gen.py +++ b/src/decompnet/permuter_gen.py @@ -63,7 +63,10 @@ def variants(job: dict): f"{plain_c(source, directives=False)}\n") rng = random.Random(job["seed"]) try: - weights = get_default_randomization_weights(job.get("family") or "base") + weights = dict(get_default_randomization_weights(job.get("family") or "base")) + for key, factor in job.get("weight_boosts", {}).items(): + if key in weights and isinstance(factor, (int, float)) and 1 <= factor <= 4: + weights[key] *= factor first = Candidate.from_source(unit, EvalState(), name, weights, rng.randrange(1, MAX_SEED)) except Exception as e: # the C or the header isn't C pycparser reads (a macro call, a compiler extension) yield {"error": f"{type(e).__name__}: {e}"} diff --git a/tests/gui_progress.js b/tests/gui_progress.js index 5f4044a2..5cc7d425 100644 --- a/tests/gui_progress.js +++ b/tests/gui_progress.js @@ -35,6 +35,7 @@ const cases = [ ["pages", 'b.onclick = async () => {', '\n };', "log", "b"], ["project", '$("localScan").onclick = async () => {', '\n};', "localScanLog", "localScan"], ["project", 'go.onclick = async () => {', '\n };', "localImportLog", "localImport"], + ["steering", '$("sourcePrepRun").onclick = async () => {', '\n};', "sourcePrepLog", "sourcePrepRun"], ]; function setup(file, marker, end, logId, buttonId, ranges = "") { const elements = new Map(); diff --git a/tests/test_app_server.py b/tests/test_app_server.py index 543433fd..67dcfe60 100644 --- a/tests/test_app_server.py +++ b/tests/test_app_server.py @@ -877,3 +877,48 @@ def test_reset_agent_metrics_requires_app_auth_and_rejects_missing_helper(app): client, _, _ = app assert client.post('/api/agents/reset-metrics', json={'id': 'missing'}, headers={'X-Decomp': '1'}).status_code == 400 assert client.post('/api/agents/reset-metrics', json={'id': 'missing'}).status_code == 403 + + +def test_permuter_can_run_without_starting_paid_helpers(app, monkeypatch): + client, net, manager = app + calls=[] + monkeypatch.setattr(net, 'project_row', lambda slug: {'slug':slug}) + monkeypatch.setattr(net.permuter, 'start', lambda slug: calls.append(('start',slug))) + monkeypatch.setattr(net.permuter, 'stop', lambda slug: calls.append(('stop',slug))) + monkeypatch.setattr(net.permuter, 'status', lambda slug: {'state':'running'}) + headers={'X-Decomp':'1'} + assert client.post('/api/permuter',json={'slug':'game'}).status_code == 403 + response=client.post('/api/permuter',json={'slug':'game'},headers=headers) + assert response.status_code == 200 and response.json()['permuter']['state']=='running' + assert client.post('/api/permuter',json={'slug':'game','action':'stop'},headers=headers).status_code==200 + assert client.post('/api/permuter',json={'slug':'game','action':'bad'},headers=headers).status_code==400 + assert calls==[('start','game'),('stop','game')] + + +def test_source_preparation_settings_and_bounded_job(app, tmp_path, monkeypatch): + from decompnet.app import source_preparation + client, net, _ = app + with net.store.tx() as db: + db.execute("INSERT INTO projects(slug,name,arch,root,config) VALUES('source-test','Game','mips',?,'{}')", (str(tmp_path),)) + headers = {'X-Decomp': '1'} + root = tmp_path / 'reference' + root.mkdir() + (root / 'game.c').write_text('int original(void) { return 1; }') + data = {'slug': 'source-test', 'root': str(root), 'enabled': True, 'mappings': 'f = original @ game.c'} + saved = client.post('/api/source-preparation', json=data, headers=headers) + assert saved.status_code == 200 + assert client.get('/api/source-preparation?slug=source-test').json()['root'] == str(root) + invalid = client.post('/api/source-preparation', json=data | {'mappings': 'not a mapping'}, headers=headers) + assert invalid.status_code == 400 + calls = [] + monkeypatch.setattr(source_preparation, 'run', lambda *args: calls.append(args) or {'tested': 1, 'improved': 1, 'matched': 0, 'errors': []}) + response = client.post('/api/source-preparation', json={'slug': 'source-test', 'action': 'run'}, headers=headers) + assert response.status_code == 200 + job = response.json()['job'] + for _ in range(100): + result = client.get('/api/job', params={'id': job}).json() + if result['state'] in {'done', 'error'}: + break + time.sleep(.01) + assert result['state'] == 'done' and calls + assert result['result']['improved'] == 1 diff --git a/tests/test_local_repairs.py b/tests/test_local_repairs.py index 9598bf8e..5020a656 100644 --- a/tests/test_local_repairs.py +++ b/tests/test_local_repairs.py @@ -121,7 +121,8 @@ def test_targeted_phase_respects_lease_and_trial_budget(tmp_path,monkeypatch): with net.store.tx() as db:db.execute('DELETE FROM claims') p._repair_round(machine,net._project('game'),st,function_id=fn['id'],budget=6,seconds=8) assert len(seen)==6 and st.repairs==6 - assert net.store.one('SELECT COUNT(*) AS n FROM reuse_trials WHERE function_id=?',fn['id'])['n']==6 + assert seen[0] == fn['best_source'] # the verified baseline shares the six-compile allowance + assert net.store.one('SELECT COUNT(*) AS n FROM reuse_trials WHERE function_id=?',fn['id'])['n']==5 assert not net.store.one('SELECT * FROM claims') diff --git a/tests/test_near_repairs.py b/tests/test_near_repairs.py index d1ed9bb2..5534fb65 100644 --- a/tests/test_near_repairs.py +++ b/tests/test_near_repairs.py @@ -93,3 +93,57 @@ def test_other_function_and_packed_layout_are_left_alone(): assert not variants(src, 'f', 'mips', asm(0x3c028001, 0x8c430010)) src = 'typedef struct __attribute__((packed)) { u8 padA[32]; s16 value; u8 padB[4]; } Item;\nvoid f(void) { return; }' assert not variants(src, 'f', 'mips') + + +def test_near_flag_search_rotates_cached_debug_probes_to_optimization_flags(): + import time + + from decompnet.app.permute import Permuter, State + + from .test_permute import AGENT + net = _flag_probe_net(); proj = net.project_row('game'); fn = net.function() + net.debug_flag_variants = lambda tc, flags: [flags, ['-O2','-g'], ['-O2','-g3']] + net.flag_variants = lambda tc, flags: [flags, ['-O1'], ['-O3']] + tested=[]; changes=[] + def judge(proj, fn, tc, flags, source): + tested.append(flags) + return flags == ['-O3'], None, '' + net._judge_with=judge + net.set_compiler_ranges=lambda *args:changes.append(args) + perm=Permuter(net); perm._score=lambda *args:0 + state=State() + assert perm._repair_flags(AGENT,proj,fn,state,time.monotonic()+10)==2 + assert not changes + assert perm._repair_flags(AGENT,proj,fn,state,time.monotonic()+10)==3 + assert tested==[['-O2','-g'],['-O2','-g3'],['-O1'],['-O3']] + assert changes[0][-1]==['-O3'] and state.matched==1 + + +def test_near_flag_match_respects_manual_compiler_approval(): + import time + + from decompnet.app.permute import Permuter, State + + from .test_permute import AGENT + net=_flag_probe_net(); proj=net.project_row('game'); fn=net.function() + net.debug_flag_variants=lambda tc, flags:[flags,['-O2','-g']] + net._judge_with=lambda *args:(True,None,'') + net.auto_adjusts=lambda proj:False + suggestions=[] + net.suggest=lambda *args,**kwargs:suggestions.append(kwargs) + net.set_compiler_ranges=lambda *args:(_ for _ in ()).throw(AssertionError('must ask the user')) + state=State() + assert Permuter(net)._repair_flags(AGENT,proj,fn,state,time.monotonic()+10)==1 + assert suggestions[0]['function']==fn['name'] and suggestions[0]['source']==fn['best_source'] + assert state.matched==0 + + +def _flag_probe_net(): + import json + + from .test_permute import FakeNet + net=FakeNet() + with net.store.tx() as db: + db.execute('UPDATE projects SET config=?',(json.dumps({'compiler':{'toolchain':'gcc-2.7.2-kmc','flags':['-O2']}}),)) + db.execute('UPDATE functions SET best_score=5,size=40') + return net diff --git a/tests/test_permuter_focus.py b/tests/test_permuter_focus.py new file mode 100644 index 00000000..d4f68d11 --- /dev/null +++ b/tests/test_permuter_focus.py @@ -0,0 +1,51 @@ +from decompnet.app import permuter_focus, repair_ranking + + +def test_actual_calls_take_precedence_over_generic_instruction_changes(): + rows = [('i', 'jal original', 'jal candidate')] + assert permuter_focus.classify(rows) == 'call' + assert repair_ranking.order(['signedness', 'call_arity', 'stack_array'], rows)[0] == 'call_arity' + assert permuter_focus.boosts(rows)['perm_randomize_function_type'] == 3 + + +def test_stack_register_constants_and_unknown_evidence(): + assert permuter_focus.classify([('s', 'sw a0,32(sp)', 'sw a0,40(sp)')]) == 'stack' + assert permuter_focus.classify([('r', 'add t0,a0,a1', 'add t1,a0,a1')]) == 'register' + assert permuter_focus.classify([('i', 'lui a0,3', 'lui a0,4')]) == 'constant' + assert permuter_focus.boosts([]) == {} + families = ['unknown', 'stack_array', 'call_arity'] + assert set(repair_ranking.order(families, [('s', 'x', 'y')])) == set(families) + + +def test_baseline_cached_per_input_and_clears_unrelated_evidence(): + from decompnet.app.permute import State + from tests.test_permute import AGENT, FakeNet, ScriptedPermuter + net = FakeNet() + net._last_diffs = {(999, 1): ([('s', 'stale', 'other source')], '')} + net.keep_diff = lambda agent, fn, rows: net._last_diffs.update({(agent, fn): (rows, '')}) + p = ScriptedPermuter(net, {}) + st = State() + proj, fn = net.project_row('game'), net.function() + score, compiled = p._baseline(AGENT, proj, fn, st, None) + assert score == 1000 and compiled == 1 + assert repair_ranking.evidence(net, 1, AGENT.id) == [] + assert p._baseline(AGENT, proj, fn, st, None) == (1000, 0) + assert len(p.judged) == 1 + net._context = lambda *args: {'version': 2, 'header': 'new context'} + assert p._baseline(AGENT, proj, fn, st, None) == (1000, 1) + assert len(p.judged) == 2 + + +def test_random_generator_receives_advisory_boosts_without_changing_verdict(monkeypatch): + from decompnet.app.permute import State + from tests.test_permute import AGENT, FakeNet, ScriptedPermuter, batches + net = FakeNet() + net._last_diffs = {(AGENT.id, 1): ([('s', 'sw a0,32(sp)', 'sw a0,40(sp)')], '')} + p = ScriptedPermuter(net, {'candidate': 0}) + jobs = batches(monkeypatch, ['miss'], ['candidate']) + state = State() + p._permute(AGENT, net.project_row('game'), net.function(), state) + assert jobs[0]['family'] == 'gcc' + assert jobs[0]['weight_boosts'] == {} + assert jobs[1]['weight_boosts']['perm_reorder_decls'] == 3 + assert state.matched == 1 and p.judged == ['miss', 'candidate'] diff --git a/tests/test_permuter_inputs.py b/tests/test_permuter_inputs.py new file mode 100644 index 00000000..19f60b63 --- /dev/null +++ b/tests/test_permuter_inputs.py @@ -0,0 +1,68 @@ +"""Changed external inputs get another bounded search; own hill climbing cannot renew it forever.""" +from decompnet.app import permuter_inputs +from tests.test_permute import AGENT, HEADER, FakeNet + + +def set_turns(net, turns=4): + with net.store.tx() as db: + db.execute('UPDATE functions SET permuted=? WHERE id=1', (turns,)) + + +def refresh(net): + return permuter_inputs.refresh(net, net.project_row('game'), [net.function()]) + + +def test_existing_counters_adopted_then_header_change_renews_once(): + net = FakeNet() + set_turns(net) + assert refresh(net) == [] + assert net.function()['permuted'] == 4 + net._context = lambda *args: {'version': 2, 'header': HEADER + '\nextern int value;'} + assert refresh(net) == [1] + assert net.function()['permuted'] == 0 + set_turns(net) + assert refresh(net) == [] + assert net.function()['permuted'] == 4 + + +def test_claimed_input_change_waits_until_release(): + net = FakeNet() + refresh(net) + set_turns(net) + with net.store.tx() as db: + db.execute('INSERT INTO claims VALUES(1,?,?,?)', (AGENT.id, 1, 9999999999)) + db.execute("UPDATE functions SET target=? WHERE id=1", (b'changed',)) + assert refresh(net) == [] + assert net.function()['permuted'] == 4 + with net.store.tx() as db: + db.execute('DELETE FROM claims') + assert refresh(net) == [1] + + +def test_only_external_new_source_renews_budget(): + net = FakeNet() + for name, source, expected in [('permuter', 'better', 4), ('helper', net.function()['best_source'], 4), + ('autopilot', 'new external seed', 0)]: + set_turns(net) + with net.store.tx() as db: + permuter_inputs.external_seed(db, net.function(), source, name) + assert net.function()['permuted'] == expected + + +def test_imported_build_content_change_not_timestamp_renews(tmp_path): + net = FakeNet() + net.fork_slug = lambda slug: slug + class Forks: + def permuter_header(self, *args): + return HEADER + net.forks = Forks() + with net.store.tx() as db: + db.execute('UPDATE projects SET root=?', (str(tmp_path),)) + manifest = tmp_path / 'build.ninja' + manifest.write_text('flags=-O2') + refresh(net) + set_turns(net) + manifest.touch() + assert refresh(net) == [] + manifest.write_text('flags=-O3') + assert refresh(net) == [1] diff --git a/tests/test_permuter_local.py b/tests/test_permuter_local.py new file mode 100644 index 00000000..544fbeb2 --- /dev/null +++ b/tests/test_permuter_local.py @@ -0,0 +1,41 @@ +from decompnet.app.permuter_local import variants + + +def test_adjacent_constant_stores_preserve_exact_lines_and_other_functions(): + text = 'void other(void){return;}\nvoid f(Item *p) {\n p->y = 0;\n p->x = 0;\n}\n' + assert variants(text, 'f') == [('adjacent_constant_stores', text.replace(' p->y = 0;\n p->x = 0;', ' p->x = 0;\n p->y = 0;'))] + + +def test_does_not_cross_calls_blocks_values_or_volatile_code(): + for middle in (' call();\n', '}\n{\n'): + assert not variants('void f(Item *p) {\n p->y = 0;\n' + middle + ' p->x = 0;\n}', 'f') + assert not variants('void f(Item *p) {\n p->y = 1;\n p->x = 0;\n}', 'f') + assert not variants('void f(volatile Item *p) {\n p->y = 0;\n p->x = 0;\n}', 'f') + + +def test_shift_add_is_local_and_leaves_comments_and_strings_opaque(): + text = 'int f(int sum) {\n // sum + (sum << 5)\n return sum + (sum << 5);\n}' + result = variants(text, 'f') + assert len(result) == 1 and result[0][0] == 'shift_add_order' + assert '// sum + (sum << 5)' in result[0][1] + assert 'return (sum << 5) + sum;' in result[0][1] + + +def test_probe_count_bounded_and_unchanged_trials_cached(): + import time + + from decompnet.app.permute import State + from tests.test_permute import AGENT, FakeNet, ScriptedPermuter + source = 'void f(Item *p) {\n' + ''.join(f' p->v{i} = 0;\n' for i in range(20)) + '}\n' + assert len(variants(source, 'f')) == 8 + net = FakeNet() + with net.store.tx() as db: + db.execute('UPDATE functions SET best_source=? WHERE id=1', (source,)) + p = ScriptedPermuter(net, {}) + proj, fn, state = net.project_row('game'), net.function(), State() + for _ in range(2): + assert p._local_order(AGENT, proj, fn, state, None, time.monotonic() + 10, 40, source) == (40, source) + assert len(p.judged) == 8 and state.repairs == 8 + state.stop.set() + p._local_order(AGENT, proj, fn, state, None, time.monotonic() + 10, 40, source) + assert len(p.judged) == 8 diff --git a/tests/test_repair_feedback.py b/tests/test_repair_feedback.py index 28280c54..3e91c94c 100644 --- a/tests/test_repair_feedback.py +++ b/tests/test_repair_feedback.py @@ -200,14 +200,17 @@ def test_comparable_failure_previous_trial_effective_settings_and_provenance(set from decompnet.repair_feedback import trial net,winner,_,pid,fid=setup effective={'compiler':{'toolchain':'example','flags':['-O2']},'effective_command':['cc','-O2','{src}']} - with net.store.tx() as db: - db.execute('UPDATE projects SET config=? WHERE id=?',(json.dumps({'compiler':effective['compiler']}),pid)) + def configure(compiler): + with net.store.tx() as db: + cfg = json.loads(db.execute('SELECT config FROM projects WHERE id=?', (pid,)).fetchone()['config']) + cfg['compiler'] = compiler + db.execute('UPDATE projects SET config=? WHERE id=?', (json.dumps(cfg), pid)) + configure(effective['compiler']) net._record(fn(net,fid),winner,20,'comparable',1,effective_settings=effective) - with net.store.tx() as db: - db.execute('UPDATE projects SET config=? WHERE id=?',(json.dumps({'compiler':{'toolchain':'other','flags':['-O1']}}),pid)) + configure({'toolchain': 'other', 'flags': ['-O1']}) net._record(fn(net,fid),winner,1,'different settings',1,effective_settings={'compiler':{'toolchain':'other','flags':['-O1']}}) - with net.store.tx() as db: - db.execute('UPDATE projects SET config=? WHERE id=?',(json.dumps({'compiler':effective['compiler']}),pid)) + configure(effective['compiler']) + with trial({'kind':'agent batch','family':'casts','variant':2}): net._record(fn(net,fid),winner,0,'matched',1,matched=True,effective_settings=effective) case=net.repair_feedback.listing('demo')['examples'][0]['id'] @@ -262,6 +265,7 @@ def test_autopilot_capture_uses_actual_compile_settings_instead_of_project_snaps cfg=json.loads(db.execute('SELECT config FROM projects WHERE id=?',(pid,)).fetchone()['config']) cfg['compiler_ranges']=[{'start':0x80001000,'end':0x80001010,**effective['compiler']}] db.execute('UPDATE projects SET config=? WHERE id=?',(json.dumps(cfg),pid)) + result=SimpleNamespace(ok=True,obj=b'object',effective_settings=effective) monkeypatch.setattr(net,'_compile',lambda *args:result) monkeypatch.setattr(net,'_evaluate',lambda *args:(SimpleNamespace(code=b'code',relocs=[]),True,[])) diff --git a/tests/test_source_preparation.py b/tests/test_source_preparation.py new file mode 100644 index 00000000..500f72b7 --- /dev/null +++ b/tests/test_source_preparation.py @@ -0,0 +1,200 @@ +"""Only explicitly mapped local source is read, with byte layout preserved and normal judging on promotion.""" +import json +from types import SimpleNamespace + +import pytest + +from decompnet.app import source_preparation as prep +from decompnet.network import Agent, Network, NetworkError +from decompnet.store import IN_MEMORY +from tests.test_permute import FakeNet + + +def config(tmp_path): + return {'root': str(tmp_path), 'mappings': 'f = original @ game.c', 'overrides': 'LIMIT=128', + 'header': str(tmp_path / 'types.h'), 'include_roots': '', 'enabled': True} + + +def test_preserves_body_line_breaks_literals_and_original_line_number(tmp_path): + cfg = config(tmp_path) + original = '// first\n// second\nint original(int a)\n{\n return a + LIMIT; /* original */\n}\n' + (tmp_path / 'game.c').write_text(original) + (tmp_path / 'types.h').write_text('#define LIMIT 2\nstruct Item { int values[LIMIT]; };\n') + source, header = prep.prepare(prep.Reader(cfg), cfg, prep.mappings(cfg['mappings'])[0], {'original': 'f'}, '') + assert source == '#line 3 "reference.c"\n' + original[original.index('int original'):].replace('int original(', 'int f(') + assert '#define LIMIT 128\nstruct Item' in header + assert '/* original */' in source + assert prep.rename('original("original"); // original', {'original': 'f'}) == 'f("original"); // original' + + +def test_flatten_explicit_includes_retail_macro_before_struct(tmp_path): + cfg = config(tmp_path) + (tmp_path / 'types.h').write_text('#include "config.h"\nstruct Item { int values[LIMIT]; };') + (tmp_path / 'config.h').write_text('#define LIMIT 2\n') + assert prep.Reader(cfg).flatten(tmp_path / 'types.h').startswith('#define LIMIT 128\n') + (tmp_path / 'types.h').write_text('#include "missing.h"') + with pytest.raises(NetworkError, match='include root'): + prep.Reader(cfg).flatten(tmp_path / 'types.h') + + +def test_no_reads_outside_selected_roots_or_through_symlinks(tmp_path): + root = tmp_path / 'selected' + root.mkdir() + (tmp_path / 'private.h').write_text('private') + (root / 'link.h').symlink_to(tmp_path / 'private.h') + cfg = config(root) + for path in (tmp_path / 'private.h', root / 'link.h'): + with pytest.raises(NetworkError, match='leaves the selected'): + prep.Reader(cfg).file(path) + cfg['include_roots'] = str(tmp_path) + assert prep.Reader(cfg).file(root / 'link.h') == 'private' + + +def test_configuration_explicit_and_machine_local(tmp_path): + net = FakeNet() + cfg = config(tmp_path) + (tmp_path / 'game.c').write_text('int original(void) { return 1; }') + (tmp_path / 'types.h').write_text('') + old = net.project_row('game')['config'] + saved = prep.configure(net, net.project_row('game'), cfg) + assert prep.settings(net, net.project_row('game')) == saved + assert net.project_row('game')['config'] == old + for bad in ('f = original', 'f = original @ x.c\nf = other @ y.c', 'f = original @ x.c\ng = original @ y.c'): + with pytest.raises(NetworkError): + prep.mappings(bad) + with pytest.raises(NetworkError): + prep.overrides('LIMIT=128\n#include "private.h"') + + +def test_body_removal_retains_layout_and_declarations(): + text = 'struct X { int n; };\nint f(void) {\n return 1;\n}\nint g(void) { return 2; }\n' + header = prep.declarations(text) + assert header.count('\n') == text.count('\n') + assert 'struct X { int n; };' in header + assert 'return' not in header + assert 'int f(void) ;' in header + + +def prepared_net(): + net = Network(IN_MEMORY) + with net.store.tx() as db: + db.execute("INSERT INTO projects(id,slug,name,arch,root,config) VALUES(1,'game','Game','mips','.',?)", (json.dumps({'compiler': {}}),)) + db.execute("INSERT INTO agents(id,name,token_hash) VALUES(1,'autopilot','x')") + db.execute("INSERT INTO context_versions(project_id,version,header,agent_id,message,created_at,unit) VALUES(1,1,'old',1,'initial',1,'')") + db.execute("INSERT INTO functions(id,project_id,name,n_insns,best_score,best_source,vram,permuted) VALUES(1,1,'f',10,40,'old source',4096,4)") + db.execute('INSERT INTO claims VALUES(1,1,1,9999999999)') + fn = net.store.one('SELECT * FROM functions WHERE id=1') + proj = net.store.one('SELECT * FROM projects WHERE id=1') + return net, proj, fn + + +def test_private_header_and_candidate_promoted_atomically_without_shared_edit(): + net, proj, fn = prepared_net() + expected = prep.inputs(net, proj, fn) + net._record(fn, Agent(1, 'autopilot', None), 20, 'new source', 0, prepared_header='new', prepared_inputs=expected) + current = net.store.one('SELECT * FROM functions WHERE id=1') + assert current['best_source'] == 'new source' and current['best_score'] == 20 + assert current['permuted'] == 0 and current['unit'] == 'local_source_1' + assert net._context(1, '')['header'] == 'old' + assert net._context(1, current['unit'])['header'] == 'new' + assert net.store.one('SELECT context_version FROM attempts')['context_version'] == 2 + + +def test_changed_inputs_or_worse_candidate_never_installs_header(): + for change in ('config', 'target', 'source', 'header', 'symbols', 'matched', 'worse', 'claim'): + net, proj, fn = prepared_net() + expected = prep.inputs(net, proj, fn) + with net.store.tx() as db: + if change == 'config': + db.execute("UPDATE projects SET config='{}'") + elif change == 'target': + db.execute("UPDATE functions SET target=?", (b'changed',)) + elif change == 'source': + db.execute("UPDATE functions SET best_source='changed'") + elif change == 'header': + db.execute("UPDATE context_versions SET version=2") + elif change == 'symbols': + db.execute("INSERT INTO symbols(project_id,name,address,size,kind) VALUES(1,'value',8192,4,'data')") + elif change == 'claim': + db.execute('DELETE FROM claims') + elif change == 'matched': + db.execute("UPDATE functions SET status='matched'") + with pytest.raises(NetworkError): + net._record(fn, Agent(1, 'autopilot', None), 50 if change == 'worse' else 20, 'new source', 0, + prepared_header='new', prepared_inputs=expected) + assert net.store.one('SELECT COUNT(*) AS n FROM attempts')['n'] == 0 + assert net.store.one('SELECT COUNT(*) AS n FROM context_versions')['n'] == 1 + assert net.store.one('SELECT unit FROM functions')['unit'] in ('', None) + + +def test_imported_mode_uses_existing_judge_without_editing_headers(): + calls = [] + net = SimpleNamespace(forks=SimpleNamespace(attempt=lambda *args, **kwargs: calls.append((args, kwargs)) or {'score': 0})) + fn = {'name': 'f', 'best_score': 40} + assert prep.attempt(SimpleNamespace(net=net), {}, fn, 'int f(void){return 1;}', 'reference header', 'agent', 'game') == 0 + assert calls[0][1] == {'keep': False, 'quiet': 40} + assert 'reference header' not in calls[0][0] + + +def test_changed_header_renews_cached_trial_and_existing_match_is_skipped(tmp_path, monkeypatch): + import threading + + from decompnet.app.permute import Permuter, State + net, proj, fn = prepared_net() + net.cfg = lambda proj: json.loads(proj['config']) + net.project_row = lambda slug: net.store.one('SELECT * FROM projects WHERE id=1') + net.fork_slug = lambda slug: None + net.links = {} + net.local_agent = lambda name: (Agent(1, 'autopilot', None), '') + cfg = config(tmp_path) + (tmp_path / 'game.c').write_text('int original(void) { return LIMIT; }') + (tmp_path / 'types.h').write_text('#define LIMIT 2\n') + prep.configure(net, proj, cfg) + calls = [] + monkeypatch.setattr(prep, 'attempt', lambda *args: calls.append(args) or 100) + with net.store.tx() as db: + db.execute('DELETE FROM claims') + permuter = Permuter(net) + assert prep.run(permuter, proj, State())['tested'] == 1 + assert prep.run(permuter, proj, State())['tested'] == 0 + (tmp_path / 'types.h').write_text('#define LIMIT 2\nextern int new_type;\n') + assert prep.run(permuter, proj, State())['tested'] == 1 + assert len(calls) == 2 + with net.store.tx() as db: + db.execute("UPDATE functions SET status='matched'") + assert prep.run(permuter, proj, State())['tested'] == 0 + stop = threading.Event() + stop.set() + assert prep.run(permuter, proj, State(stop=stop))['tested'] == 0 + + +def test_failed_compile_never_changes_saved_candidate_or_shared_header(monkeypatch): + net, proj, fn = prepared_net() + net._compile = lambda *args: SimpleNamespace(ok=False, log='missing struct declaration') + with pytest.raises(NetworkError, match='missing struct declaration'): + prep.attempt(SimpleNamespace(net=net), proj, fn, 'new source', 'bad header', Agent(1, 'autopilot', None), None) + assert net.store.one('SELECT best_source,unit FROM functions')['best_source'] == 'old source' + assert net._context(1, '')['header'] == 'old' + + +def test_reference_aliases_are_explicit_and_token_safe(): + assert prep.symbol_aliases('gReference = D_80010000') == {'gReference': 'D_80010000'} + with pytest.raises(NetworkError): + prep.symbol_aliases('gReference = first\ngReference = second') + + + +def test_static_entry_point_is_emitted_without_changing_body_layout(tmp_path): + cfg = config(tmp_path) + (tmp_path / 'game.c').write_text('static int original(int a)\n{\n return a;\n}\n') + (tmp_path / 'types.h').write_text('') + source, _ = prep.prepare(prep.Reader(cfg), cfg, prep.mappings(cfg['mappings'])[0], {'original': 'f'}, '') + assert source == '#line 1 "reference.c"\n int f(int a)\n{\n return a;\n}\n' + + + +def test_multiple_reference_definitions_require_an_explicit_subset(tmp_path): + cfg = config(tmp_path) + (tmp_path / 'game.c').write_text('#ifdef DEBUG\nint original(void){return 1;}\n#else\nint original(void){return 2;}\n#endif\n') + with pytest.raises(NetworkError, match='More than one definition'): + prep.prepare(prep.Reader(cfg), cfg, prep.mappings(cfg['mappings'])[0], {}, '') diff --git a/tests/test_speed.py b/tests/test_speed.py index 5d306400..1d53a7a6 100644 --- a/tests/test_speed.py +++ b/tests/test_speed.py @@ -304,14 +304,13 @@ def test_the_permuter_finishes_a_near_miss(game, monkeypatch): assert st.matched == 1 and net._function("entity_update_physics", slug)["status"] == "matched", st -@pytest.mark.skipif(toolchains.installed("decomp-permuter") is None, reason="needs decomp-permuter") @pytest.mark.parametrize("name,written,near", [ ("entity_spawn", "e->vel_x = 0;\n e->vel_y = 0;", "e->vel_y = 0;\n e->vel_x = 0;"), - ("checksum_save", "(sum << 5) + sum", "sum + (sum << 5)"), # no simple rewrite turns operands around + ("checksum_save", "(sum << 5) + sum", "sum + (sum << 5)"), # a local probe avoids waiting for random drift ]) -def test_decomp_permuter_brings_a_near_miss_back_to_a_match(game, name, written, near): +def test_permuter_repairs_local_ordering_before_random_search(game, name, written, near): """A matching function with two statements in another order, or an addition's operands the other way round, - left as an agent's near miss: decomp-permuter's variants, judged by our judge, match it again in one turn.""" + left as an agent's near miss: bounded local ordering probes, judged normally, match it in one turn.""" import time from decompnet.app import permute @@ -337,3 +336,44 @@ def test_decomp_permuter_brings_a_near_miss_back_to_a_match(game, name, written, net.permuter._permute(permuter, proj, net._function(name, slug), st) assert st.matched == 1 and net._function(name, slug)["status"] == "matched", st assert time.monotonic() - started < permute.SECONDS + + +def test_local_source_preparation_uses_real_rom_judge_and_private_context(game, tmp_path): + """The opt-in seed path must earn an exact linked verdict, not just a successful compiler invocation.""" + import sqlite3 + + from decompnet.app import source_preparation + from decompnet.app.permute import State + original, slug, _ = game + database = tmp_path / 'copy.db' + with sqlite3.connect(database) as dest: + original.store.conn.backup(dest) + proj = original.project_row(slug) + folder = tmp_path / 'project' + shutil.copytree(proj['root'], folder) + net = RomNetwork(database, tmp_path) + net.on_match = lambda *args: None + with net.store.tx() as db: + db.execute('UPDATE projects SET root=? WHERE slug=?', (str(folder), slug)) + db.execute("UPDATE functions SET status='open',matched_source=NULL,best_source='int clamp_s32(int x,int lo,int hi){return 0;}',best_score=10000 WHERE project_id=? AND name='clamp_s32'", (proj['id'],)) + db.execute('DELETE FROM claims') + fn = net._function('clamp_s32', slug) + old_header = net._context(proj['id'], fn['unit'])['header'] + header = tmp_path / 'reference.h' + header.write_text(old_header) + cfg = {'enabled': True, 'root': str(GAME), 'header': str(header), 'include_roots': str(tmp_path), + 'mappings': 'clamp_s32 = clamp_s32 @ game.c'} + source_preparation.configure(net, net.project_row(slug), cfg) + report = source_preparation.run(net.permuter, net.project_row(slug), State()) + assert report['errors'] == [] + assert report['tested'] == report['matched'] == 1 + matched = net._function('clamp_s32', slug) + assert matched['status'] == 'matched' and matched['unit'] == f'local_source_{fn["id"]}' + assert net._context(proj['id'], fn['unit'])['header'] == old_header + assert net.try_quietly(net.local_agent('you')[0], net.project_row(slug), matched, matched['matched_source'])['matched'] + + from decompnet.rom.export import export_rom + exported = export_rom(net, slug, tmp_path / 'export', net.console(net.project_row(slug)), log=lambda *args: None) + assert exported['identical'] + assert 'clamp_s32' not in [name for name, _ in exported['fallback']] + assert list((tmp_path / 'export' / 'include').glob('common_local_source_*.h')) diff --git a/tests/test_stack_repairs.py b/tests/test_stack_repairs.py new file mode 100644 index 00000000..41203c76 --- /dev/null +++ b/tests/test_stack_repairs.py @@ -0,0 +1,68 @@ +"""Storage hypotheses stay local and pass through the existing bounded repair pipeline.""" +from decompnet.app import draft_repairs, stack_repairs + + +def test_unused_scalar_ignores_field_names_and_keeps_lines_literals_and_other_functions(): + source = 'int other(void) { return 7; }\nvoid f(Obj *p)\n{\n int flags; /* flags stays a comment */\n p->flags=1;\n say("flags");\n}\n' + variants = stack_repairs.variants(source, 'f') + repaired = next(text for family,text in variants if family == 'unused_scalar_local') + assert 'int flags;' not in repaired and 'p->flags=1;' in repaired + assert '/* flags stays a comment */' in repaired and 'say("flags")' in repaired + assert repaired.count('\n') == source.count('\n') + assert repaired.startswith('int other(void) { return 7; }\n') + + +def test_live_address_taken_and_sizeof_locals_are_not_removed_or_made_register(): + for expression in ['use(&x);', 'use(sizeof(x));', 'return x;']: + variants = stack_repairs.variants('int f(void) { int x; '+expression+' }', 'f') + assert not any(f == 'unused_scalar_local' for f,_ in variants) + if '&x' in expression: + assert not any(f == 'local_storage' for f,_ in variants) + + +def test_initialized_volatile_aggregate_and_preprocessor_storage_are_not_guessed(): + for declaration in ['volatile int x;', 'int x=side_effect();', 'Obj x;']: + assert not stack_repairs.variants('void f(void) { '+declaration+' use(x); }', 'f') + assert not stack_repairs.variants('void f(void) { int x;\n#if X\nuse(x);\n#endif\n}', 'f') + + +def test_pointer_and_scalar_parameters_support_register_without_touching_prototypes(): + src = 'int f(int n, Obj *p);\nint f(int n, Obj *p) { int i; i=n; return p->x+i; }' + variants = stack_repairs.variants(src, 'f') + params = [text for family,text in variants if family == 'parameter_storage'] + assert any('int f(register int n, Obj *p)' in text for text in params) + assert any('int f(int n, register Obj *p)' in text for text in params) + assert all(text.startswith('int f(int n, Obj *p);\n') for text in params) + assert not any(f == 'parameter_storage' for f,_ in stack_repairs.variants('void f(int x) { use(&x); }', 'f')) + + +def test_adjacent_declaration_order_and_register_removal_are_bounded(): + src = 'int f(void) { register int x; int y; x=1; y=2; return x+y; }' + assert any(f == 'local_storage' and 'register' not in text for f,text in stack_repairs.variants(src,'f')) + src = 'int f(void) { int x; float y; x=1; y=2; return x+y; }' + assert any(f == 'local_declaration_order' and 'float y; int x;' in text for f,text in stack_repairs.variants(src,'f')) + labels = {} + results = draft_repairs.variants(src, 'f', 'mips', labels=labels) + assert any(labels[text] == 'local_declaration_order' for text in results) + assert len(results) <= draft_repairs.LIMIT + + +def test_background_repair_pass_scores_new_storage_candidates(): + from decompnet.app.permute import State + + from .test_draft_repairs import setup_repair_net + from .test_permute import AGENT, ScriptedPermuter + net=setup_repair_net() + src='int f1(int x) { int unused; return send(x, 7); }' + with net.store.tx() as db: + db.execute('UPDATE functions SET best_source=? WHERE id=1',(src,)) + candidate=next(text for family,text in stack_repairs.variants(src,'f1') if family=='unused_scalar_local') + perm=ScriptedPermuter(net,{candidate:0});state=State() + perm._repair_round(AGENT,net.project_row('game'),state) + assert candidate in perm.judged and state.matched==1 + + +def test_unused_declaration_keeps_embedded_comments(): + src='void f(void) { int /* field note */ unused; use(); }' + candidate=next(t for f,t in stack_repairs.variants(src,'f') if f=='unused_scalar_local') + assert '/* field note */' in candidate and 'unused' not in candidate