scripts: Add explicit timeouts and record commit metadata in reports - #206
scripts: Add explicit timeouts and record commit metadata in reports#206Chand-ra wants to merge 2 commits into
Conversation
The IR programs take longer to execute than standard AFL++ inputs, so using the default AFL++ values causes false positive hangs and timeouts. Add explicit timeout durations to mitigate this.
For each configuration, record the latest Git commit metadata corresponding to that configuration and display it in the final report. This ensures evaluation data lineage and reproducibility.
|
Addresses #205. |
erickcestari
left a comment
erickcestari
left a comment
There was a problem hiding this comment.
Nice, really quick PR, and the commit hash is a great addition.
This should fix the issue I've recently opened #205.
| # Fixed power schedule for every standalone trial. | ||
| POWER_SCHEDULE = "explore" | ||
| # Fixed timeout for a single input. | ||
| EXEC_TIMEOUT = 2000 # 2 seconds |
There was a problem hiding this comment.
This value could be configured by a flag and also have a bigger default as 5000.
There was a problem hiding this comment.
The default value was decided upon after an offline discussion with @morehouse.
However, if you've actually encountered a non-hanging/non-timeout input take 5 seconds to execute on your machine, I agree that it is a good reason to increase the current value.
There was a problem hiding this comment.
I didn't find any hang with 5000 so far. But I agree that 2 seconds should already be enough.
There was a problem hiding this comment.
I agree it would be good to make this a flag.
morehouse
left a comment
morehouse
left a comment
There was a problem hiding this comment.
How was this tested? It broke for me on the first try.
| # Fixed power schedule for every standalone trial. | ||
| POWER_SCHEDULE = "explore" | ||
| # Fixed timeout for a single input. | ||
| EXEC_TIMEOUT = 2000 # 2 seconds |
There was a problem hiding this comment.
I agree it would be good to make this a flag.
| "AFL_NO_UI": "1", | ||
| "AFL_NO_COLOR": "1", | ||
| "AFL_FORKSRV_INIT_TMOUT": "1800000", | ||
| "AFL_HANG_TIMEOUT": "4000", # 4 seconds |
There was a problem hiding this comment.
Wrong env var name. We need to use AFL_HANG_TMOUT.
Would also be good to make this a flag.
| f"[yellow]Warning: Could not get git info for '{label}' in {smite_dir}[/]" | ||
| ) | ||
|
|
||
| (config_out / "latest_commit.txt").write_text(commit_info + "\n") |
There was a problem hiding this comment.
This breaks the evaluation script for me.
[*] Detected Targets: ['cln', 'latest_commit.txt', 'lnd']
- baseline/cln: found 3 valid trials
NotADirectoryError: [Errno 20] Not a directory: 'camp/baseline/latest_commit.txt'
3 participants
This PR introduces two improvements to the Smite coverage-based effectiveness evaluation pipeline:
Explicit fuzzer timeouts: Add explicit execution and hang timeout parameters to the fuzzer invocation in
smite-orchestrator.py. This prevents false-positive hangs and timeouts for IR programs.Evaluation data lineage: Record the latest Git commit hash and date for each configuration worktree during campaign startup, displaying this metadata at the top of the generated evaluation reports. This ensures strict data provenance for a rigorous evaluation.