Reject malformed diffs, report unapplied diffs, opt-in EVOLVE-BLOCK enforcement, and quick fixes (v0.4.0) - #494
Merged
codelion merged 9 commits intoSep 28, 2026
Conversation
|
|
This was referenced Sep 27, 2026
brriorda
marked this pull request as ready for review
September 27, 2026 10:58
codelion
force-pushed
the
openevolve-diff-format-validation
branch
2 times, most recently
from
September 28, 2026 02:28
57306cc to
9187975
Compare
Pass token_usage through the malformed-delimiter error result so LLM usage is still tracked (main added token_usage to worker error results). Add tests for extra, nested and stray SEARCH/REPLACE markers, custom diff_pattern behaviour, and the config default matching the standard pattern that validation is keyed on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cover malformed shapes (extra separators in SEARCH or REPLACE, a bad later block, missing markers, indented or trailing-whitespace markers, CRLF responses), look-alike lines that must still be accepted (longer or shorter equals runs, markers inside code or prose), valid forms (empty SEARCH or REPLACE, code fences, no trailing newline), and that apply_diff makes no partial changes. Worker tests check a malformed response is rejected without evaluation and keeps token usage, including in changes-description mode, and that custom diff patterns stay permissive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Diffs whose SEARCH text matched nothing were silently skipped, so the unchanged parent was evaluated again as a new child. The worker now rejects responses where no SEARCH block matches or the result equals the parent (including identical full rewrites), warns on partial matches, and SEARCH lines match even when they differ only in trailing whitespace. EVOLVE-BLOCK markers were only prompt hints. The new enforce_evolve_blocks option (default false) restores everything outside the blocks to the parent code before evaluation and rejects responses that change or remove the markers. Markers are recognised after any comment syntax (#, //, /*, --, %), so evolve blocks work in C, Java, etc. Closes algorithmicsuperintelligence#346 Closes algorithmicsuperintelligence#422 Closes algorithmicsuperintelligence#106 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add database.embedding_api_base (or OPENAI_EMBEDDING_BASE_URL) so novelty embeddings can use any OpenAI-compatible endpoint, such as OpenRouter or a local server, with whatever model name that endpoint serves. Closes algorithmicsuperintelligence#426 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Some OpenAI-compatible endpoints stream Server-Sent Events unless told otherwise, which surfaced as "'str' object has no attribute 'choices'". Send stream=False and raise a clear error if a raw string still comes back. Closes algorithmicsuperintelligence#436 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The visualizer always ran with debug=True, which enables the interactive Werkzeug debugger. Debug mode is now opt-in with --debug. Refs algorithmicsuperintelligence#481 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
codelion
force-pushed
the
openevolve-diff-format-validation
branch
from
September 28, 2026 02:39
9187975 to
2ac6b33
Compare
codelion
changed the title
Frame SAST flags assigning a string literal to model_cfg.api_key as a hardcoded secret (CWE-798). openai.OpenAI is patched in these tests, so the key is set to None instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reject malformed SEARCH/REPLACE delimiter sequences before mutation
Rationale
The standard diff extractor currently accepts a response with an extra standalone
=======line inside a replacement block. Its regex captures that line asreplacement text, and the worker writes it into the candidate source. A malformed
proposal can therefore become an evaluated program instead of a generation-format
failure. In one observed run, five SEARCH/REPLACE blocks contained seven separator
lines; the resulting Python source did not parse.
Changes
standard SEARCH/REPLACE block. Extra, nested, and unmatched marker lines cause
a format error.
validation fails.
diff_patternparsing behavior; the stricter grammar appliesonly to OpenEvolve's standard SEARCH/REPLACE format.
Reproduction
Use two valid SEARCH/REPLACE blocks and insert a second standalone
=======line in one replacement. Before this change,
extract_diffsreturned bothblocks and
apply_diffinserted the extra line into the output. After thischange,
extract_diffsraisesValueErrorand the worker does not evaluate acandidate.
Verification
The code was inspected against the observed malformed response and existing
standard diff grammar. Automated tests were not run for this revision.
Additional fixes folded into this PR (v0.4.0)
Rebased onto current
main, with tests added for the delimiter validation and a few related quick fixes for open issues:enforce_evolve_blocks(EVOLVE-BLOCK-START and EVOLVE-BLOCK-END markers do not work #422, Not respecting the EVOLVE-BLOCK-START / END markers #106): whentrue, everything outside theEVOLVE-BLOCK-START/ENDregions is restored to the parent code before evaluation, and responses that change or remove the markers are rejected. Defaultfalsekeeps current behaviour.#,//,/*,--,%), not only# EVOLVE-BLOCK-START.database.embedding_api_baseorOPENAI_EMBEDDING_BASE_URLlets novelty embeddings use any OpenAI-compatible endpoint.stream=False, and a raw SSE string now raises a clear error.exec_module operator-supplied code in-process;visualizer` serves checkpoint data with no auth #481): the Flask debugger is now opt-in with--debug. The in-process evaluator part of evaluatorexec_module operator-supplied code in-process;visualizer` serves checkpoint data with no auth #481 is not changed.token_usageis kept on the new delimiter error result, matching other worker errors onmain.Tests
All 568 tests pass locally (
python -m unittest discover tests). New tests cover delimiter validation edge cases, the worker's rejection paths (never evaluated, token usage kept), whitespace-tolerant matching, evolve-block parsing and enforcement in diff and full-rewrite modes, the embedding base URL,stream=False, and the visualizer flag. They were checked to fail with the source changes removed.Closes #346
Closes #422
Closes #106
Closes #426
Closes #436
🤖 Generated with Claude Code