Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -621,6 +621,23 @@ Verified full-image imports include linked assembly routines as target-only unit
as recovered C. Assembly wrappers containing whole binary programs are not treated as single functions; an
explicit coverage notice identifies their exclusion from function progress.

### Repair preflight and reuse discovery

Saved ROM drafts get an advisory warning preflight in agent briefs and deep-search output when their pinned
GCC supports it. It uses separate warning flags and caches diagnostics for unchanged compiler, header and
source inputs. An uninitialized-variable warning is a reason to check the draft before tuning it; warnings
never change project settings or count as a match. Imported projects retain diagnostics from their own build;
extra diagnostic builds for those projects and other compiler families are not supported yet.

Generated local repairs skip direct calls that disagree with an unambiguous plain C prototype. Macros,
includes, indirect calls, unspecified argument lists, variable arguments and C++ remain the compiler's job.
The normal compiler and exact judge still decide every accepted improvement.

Template/address/SDK discovery remembers its last visited function, including interrupted passes. Cached
misses are read together per function, and this phase takes at most half the remaining reuse time so wider
cross-project discovery can run too. Both phases retain the shared compile budget; a cursor does not skip
new candidates permanently.

### Permuter budgets

For a longer search of one function, open its drawer and expand **Deep permuter · this function**.
Expand Down
7 changes: 7 additions & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -881,3 +881,10 @@ projects. It bypasses the background queue's size and turn filters, visits the g
then uses ordinary random rewrites. Time/compile limits, progress, cancellation, normal judges and immediate
saving remain mandatory. Scoped misses survive another deep run; failed compiles stay retryable and cached
improvements must be rejudged. This does not claim to exhaust all possible C programs or compiler options.

Repair preflight filters provable plain-prototype arity errors in shared generated local families and ROM
permuter/reuse candidates, before the ordinary judges. Saved ROM GCC drafts get cached advisory warnings in
agent briefs and deep-search output. Planned: compiler-specific diagnostic flags for other backends and an
opt-in diagnostic build command for imported projects (their ordinary build diagnostics remain available).
Reuse discovery batches fingerprint names/cache reads and persists a separate cheap-discovery cursor; at most
half the remaining shared reuse time goes to that phase, leaving time for wider matching.
46 changes: 46 additions & 0 deletions src/decompnet/app/compiler_preflight.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
"""Advisory diagnostics for saved ROM drafts, separate from the exact match judge.

A private N64 near miss used an uninitialized pointer. One warning-enabled compile per input snapshot
makes that evidence available before more permutations; it never supplies a score or changes project flags.
"""
import hashlib
import json
import logging
import threading

from .. import toolchains
from ..project import compiler_for

log = logging.getLogger('decompdecomp')


def warnings(net, proj, fn):
"""Return cached diagnostics where the pinned compiler supports warning flags, without keeping its object."""
if not fn['best_source'] or net.fork_slug(proj['slug']):
return '' # Imported projects own their build commands; their ordinary build retains its diagnostics.
cfg = net.cfg(proj)
settings = compiler_for(cfg, fn['vram'])
flags = toolchains.diagnostic_flags(settings.get('toolchain', ''))
if not flags or settings.get('cmd') or not net.is_rom(proj):
return ''
header = net._context(proj['id'], fn['unit'])['header']
key = hashlib.sha256(repr((proj['root'], settings, header, fn['best_source'])).encode()).hexdigest()
cache = net.__dict__.setdefault('_compiler_preflights', {})
lock = net.__dict__.setdefault('_compiler_preflight_lock', threading.Lock())
with lock:
if key in cache:
return cache[key]
diagnostic = {**cfg, 'compiler_ranges': [], 'compiler': {**settings, 'flags': [*settings.get('flags', []), *flags]}}
try:
result = net._compile({**dict(proj), 'config': json.dumps(diagnostic)}, header, fn['best_source'], fn)
lines = [line for line in result.log.splitlines() if 'warning' in line.lower() or 'uninitialized' in line.lower()]
text = '\n'.join(lines)[:1000] if result.ok else ''
except Exception: # advisory evidence must never stand between the user and the real judge
log.exception('compiler warning preflight failed')
return '' # Tool failures remain retryable.
# Different functions may compile concurrently. Only publication/eviction needs the lock.
with lock:
cache[key] = text
while len(cache) > 256:
del cache[next(iter(cache))]
return text
19 changes: 15 additions & 4 deletions src/decompnet/app/deep_permuter.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ def __init__(self, net, minutes, max_variants, log):
self.seconds = minutes*60
self.limit = max_variants
self.log = log
self.tested = self.skipped = self.visited = 0
self.tested = self.skipped = self.visited = self.rejected = 0
self.best = None
self.phase = 'local repairs'
self.measuring_baseline = False
Expand All @@ -54,7 +54,7 @@ def check(self):

def report(self):
self.log.progress({'phase': self.phase, 'tested': self.tested, 'skipped': self.skipped,
'best_score': self.best, 'variant_limit': self.limit,
'best_score': self.best, 'variant_limit': self.limit, 'rejected': self.rejected,
'seconds_left': max(0, round(self.deadline-time.monotonic()))})

def _baseline(self, *args):
Expand All @@ -69,6 +69,13 @@ def _baseline(self, *args):
def _score(self, agent, proj, fn, source, best, fork):
self.check()
self.visited += 1
if not fork and not self.measuring_baseline and source != fn['best_source']:
from .repair_preflight import invalid_call
header = self.net._context(proj['id'], fn['unit'])['header']
if invalid_call(source, fn['name'], header):
self.rejected += 1
self.report()
return None
# Scope includes the actual compiler/header/target and imported build dependencies. A previous
# miss is reusable only in that scope. Better cached results are rejudged before being kept.
scope = agent_tokens.target_scope(self.net, {**dict(fn), 'best_source': source}, '')
Expand Down Expand Up @@ -97,6 +104,10 @@ def run(net, slug, name, minutes, max_variants, log):
state = State()
agent, _ = net.local_agent('permuter')
reason = 'Current candidate families finished'
from .compiler_preflight import warnings
warning = warnings(net, proj, fn)
if warning:
log('Compiler preflight (advisory): '+warning)
log('Testing supported local repairs, then bounded random rewrites. This cannot exhaust every possible C program.')
try:
if not search._hold(agent, fn, count_turn=False):
Expand Down Expand Up @@ -145,6 +156,6 @@ def run(net, slug, name, minutes, max_variants, log):
except SearchLimit as error:
reason = str(error)
search.report()
log(f'{reason}: {search.tested} candidates tested, {search.skipped} cached misses skipped. Verified improvements are saved immediately.')
log(f'{reason}: {search.tested} candidates tested, {search.skipped} cached misses skipped, {search.rejected} invalid calls skipped. Verified improvements are saved immediately.')
return {'function': name, 'reason': reason, 'tested': search.tested, 'skipped': search.skipped,
'matched': net._function(name, slug)['status'] == 'matched', 'best_score': search.best}
'rejected': search.rejected, 'matched': net._function(name, slug)['status'] == 'matched', 'best_score': search.best}
3 changes: 3 additions & 0 deletions src/decompnet/app/draft_repairs.py
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,10 @@ def add(a, b, text):
labels.setdefault(text,'expression_or_near_repair')
# Round-robin families so a new family cannot crowd existing repairs out of a short pass.
groups = {}
from .repair_preflight import invalid_call
for text in dict.fromkeys(proposals):
if invalid_call(text, name):
continue
groups.setdefault(labels[text],[]).append(text)
ordered = []
while any(groups.values()):
Expand Down
8 changes: 7 additions & 1 deletion src/decompnet/app/engine/briefs.py
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,13 @@ def _brief(self, fn, agent, header):
fn = self._function(fn["name"], proj["slug"])
if fn["seed"]:
fn = dict(fn) | {"seed": self._draft_shown(fn, proj)}
return super()._brief(fn, agent, header)
text = super()._brief(fn, agent, header)
if fn['status'] == 'open' and fn['best_source']:
from .. import compiler_preflight
warning = compiler_preflight.warnings(self, proj, fn)
if warning:
text += '\n\nCompiler preflight (advisory; check the draft before tuning):\n' + warning
return text

def xrefs(self, proj) -> dict:
"""calls / called_by / uses / used_by, all keyed by address."""
Expand Down
Loading
Loading