pkg/aflow: expose reproducer test coverage to the LLM#7008
Merged
tarasmadan merged 3 commits intogoogle:masterfrom Apr 16, 2026
Merged
pkg/aflow: expose reproducer test coverage to the LLM#7008tarasmadan merged 3 commits intogoogle:masterfrom
tarasmadan merged 3 commits intogoogle:masterfrom
Conversation
ramosian-glider
force-pushed
the
hackathon-250326
branch
7 times, most recently
from
4d4ca67 to
c2a901c
Compare
Collaborator
|
@gemini-cli /review |
|
🤖 Hi @tarasmadan, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
This Pull Request introduces a powerful mechanism to expose code coverage from crash reproducer runs to the LLM agent. This is achieved by caching coverage artifacts and providing specialized tools for the agent to query and inspect covered files and source code snippets. The overall quality and architecture are consistent with the aflow package, but there are a few critical issues that need to be addressed before merging.
🔍 General Feedback
- Positive Highlights: The addition of coverage-aware tools is a significant improvement for the agent's ability to debug and refine reproducers. The caching strategy is well-integrated with the existing
aflowinfrastructure. - Security: The
get-file-coveragetool currently lacks validation for theFilenameargument, which could lead to path traversal if a model provides an absolute or relative path that escapes the kernel source root. - Regression: The
reproducetool now always enables coverage collection, which will cause it to fail for C-only reproducers that don't have a corresponding syzkaller program, as the underlyingRunTestrequires a syzkaller program for coverage collection.
ramosian-glider
force-pushed
the
hackathon-250326
branch
2 times, most recently
from
8da3bbb to
0acc6d9
Compare
tarasmadan
reviewed
Mar 27, 2026
dvyukov
reviewed
Mar 27, 2026
dvyukov
reviewed
Mar 27, 2026
ramosian-glider
force-pushed
the
hackathon-250326
branch
2 times, most recently
from
29db92c to
7c510bf
Compare
Member
Author
|
@gemini-cli /review |
|
🤖 Hi @ramosian-glider, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
There was a problem hiding this comment.
This PR successfully integrates coverage extraction and caching into the crash reproduction workflow, providing essential context for LLM agents to debug and refine reproducers. The implementation is clean, follows established aflow patterns, and includes robust security checks for file access.
🔍 General Feedback
- Caching: The use of
aflow.Cachewithdesc-based hashing correctly ensures that coverage data is logically linked to the specific test configuration. - Security:
filepath.IsLocalis correctly applied toCoverageIDandFilenameto prevent directory traversal. - Testing: New tools are well-tested with appropriate mock data.
A minor improvement is suggested to ensure that the CoverageID is consistently returned even when a crash isn't reproduced, which is vital for the agent's debugging capabilities.
tarasmadan
reviewed
Apr 14, 2026
ramosian-glider
force-pushed
the
hackathon-250326
branch
from
7c510bf to
3e2a771
Compare
Add cacheReadObject to allow retrieving cached objects by their ID (hash) alone, without needing the original description. This also updates the lastUsed timestamp to prevent active entries from being purged. Also add CacheReadObject wrapper in execute.go to expose it via Context. Also add unit tests in cache_test.go to verify the new functionality and LRU update.
ramosian-glider
force-pushed
the
hackathon-250326
branch
from
3e2a771 to
ad76ad9
Compare
Currently, the AI reproducer workflow simply reports whether a program crashed and what bug title was produced. This doesn't provide enough localized context for the LLM to refine and correctly debug a reproducer when it fails. This patch adds mechanisms to extract and cache coverage generated during the `crash.Reproduce` test cycle. The cached coverage arrays are decoupled from the immediate LLM context and are instead serialized to persistent directories. The unique hash ID (`CoverageID`) of this cache is then mapped and passed back to the LLM agent within `ReproduceResult`. Also introduces `get-coverage-files` and `get-file-coverage` LLM tools to `pkg/aflow/tool/syzlang`, enabling the `crash-repro-finder` agent to query and securely inspect these precise coverage traces file-by-file. To support sandboxed cache resolution, a new `CacheDir` helper is added to `aflow.Context`.
The ai coverage tools rely heavily on the internal aflow.Cache to read and parse reproducer test run footprints. In order to test them without requiring real LLM workflows or live syz-manager VMs, we need a way to mock the cache locally. This patch adds a `NewTestContext` helper in `pkg/aflow/test_util.go` to easily initialize an `aflow.Context` with an explicit local dummy disk cache. It then introduces `pkg/aflow/tool/syzlang/coverage_test.go` to thoroughly mock and test `getCoverageFiles` and `getFileCoverage`. The tests manufacture a fake source tree along with `symbolizer.Frame` coverage payloads to assert that the AI tool coverage extraction algorithm, including its snippet padding, right-aligned line spacing, and binary-search deduplication logic, all format correctly for the agent context.
ramosian-glider
force-pushed
the
hackathon-250326
branch
from
ad76ad9 to
2f9ebdf
Compare
Member
Author
|
PTAL |
Member
Author
|
oops, accidentally clicked auto-merge |
tarasmadan
approved these changes
Apr 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently, the AI reproducer workflow simply reports whether a program crashed and what bug title was produced. This doesn't provide enough localized context for the LLM to refine and correctly debug a reproducer when it fails.
This patch adds mechanisms to extract and cache coverage generated during the
crash.Reproducetest cycle. The cached coverage arrays are decoupled from the immediate LLM context and are instead serialized to persistent directories. The unique hash ID (CoverageID) of this cache is then mapped and passed back to the LLM agent withinReproduceResult.Also introduces
get-coverage-filesandget-file-coverageLLM tools topkg/aflow/tool/syzlang, enabling thecrash-repro-finderagent to query and securely inspect these precise coverage traces file-by-file.Before sending a pull request, please review Contribution Guidelines:
https://github.com/google/syzkaller/blob/master/docs/contributing.md