Add OrcaReplay to LLM & MCP Testing - #117
Merged
Merged
Conversation
|
Review changes with |
Reviewer's guide (collapsed on small PRs)Reviewer's GuideAdds an alphabetically placed OrcaReplay entry to the LLM & MCP Testing section, describing its process-level recording and provider-independent replay workflow for turning failed agent sessions into regression tests. Sequence diagram for OrcaReplay recording and replaysequenceDiagram
participant Orca as OrcaReplay CLI
participant Script as Python agent script
participant Provider as Model provider
participant Replay as Recorded responses
Orca->>Script: Launch under orca record
Script->>Provider: Model API call
Provider-->>Script: Model response
Orca->>Orca: Record conversation
Orca->>Script: Launch under orca replay
Script->>Replay: Model API call
Replay-->>Script: Recorded response
Script-->>Orca: Reproduce recorded run
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Code Review by Qodo🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)
Great, no issues found!Qodo reviewed your code and found no material issues that require reviewTip of the day💡 Did you know, you can reply 'qodo' on any finding to push back, ask questions, or dig deeper |
PR Summary by QodoAdd OrcaReplay to LLM and MCP testing resources
AI Description
High-Level Assessment
Files changed (1)
|
Contributor
|
Tick the box to add this pull request to the merge queue (same as
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="README.md" line_range="84" />
<code_context>
+- [OrcaReplay](https://github.com/Continuum-AI-Corp/OrcaReplay) - Records a Python agent's calls to its model provider from outside the process, then serves the recording back so the same run happens again with the provider unreachable and no key. Turns a failed session into a regression test. Apache-2.0.
</code_context>
<issue_to_address>
**🚨 issue (security):** The entry describes replay as making the same run happen again without warning that recorded tool calls execute for real; a user treating replay as an offline or safe regression run will still trigger side effects such as a recorded `curl` reaching the network.
**Triggers:** When a recorded agent run contains tool calls with external side effects.
**Suggested fix:** State that replay only blocks model-provider access and that recorded tools, including network calls, still execute normally.
```suggestion
- [OrcaReplay](https://github.com/Continuum-AI-Corp/OrcaReplay) - Records a Python agent's calls to its model provider from outside the process, then serves the recording back so the same run happens again with only model-provider access blocked; recorded tools, including network calls, still execute normally. Turns a failed session into a regression test. Apache-2.0.
```
</issue_to_address>Sourcery assessment
Approval pending. 1 finding to address first.
Blocking findings: README.md:84
| ## LLM & MCP Testing | ||
|
|
||
| - [mcp-server-fuzzer](https://github.com/Agent-Hellboy/mcp-server-fuzzer) - A comprehensive fuzzing tool designed specifically for testing Model Context Protocol (MCP) servers. It supports both tool argument fuzzing and protocol type fuzzing across multiple transport protocols. | ||
| - [OrcaReplay](https://github.com/Continuum-AI-Corp/OrcaReplay) - Records a Python agent's calls to its model provider from outside the process, then serves the recording back so the same run happens again with the provider unreachable and no key. Turns a failed session into a regression test. Apache-2.0. |
There was a problem hiding this comment.
🚨 issue (security): The entry describes replay as making the same run happen again without warning that recorded tool calls execute for real; a user treating replay as an offline or safe regression run will still trigger side effects such as a recorded curl reaching the network.
Triggers: When a recorded agent run contains tool calls with external side effects.
Suggested fix: State that replay only blocks model-provider access and that recorded tools, including network calls, still execute normally.
Suggested change
| - [OrcaReplay](https://github.com/Continuum-AI-Corp/OrcaReplay) - Records a Python agent's calls to its model provider from outside the process, then serves the recording back so the same run happens again with the provider unreachable and no key. Turns a failed session into a regression test. Apache-2.0. | |
| - [OrcaReplay](https://github.com/Continuum-AI-Corp/OrcaReplay) - Records a Python agent's calls to its model provider from outside the process, then serves the recording back so the same run happens again with only model-provider access blocked; recorded tools, including network calls, still execute normally. Turns a failed session into a regression test. Apache-2.0. |
![llamapreview[bot]](https://avatars.githubusercontent.com/in/1023541?s=60&v=4){kind=small}
There was a problem hiding this comment.
LlamaPReview — No blocking issues found
Read bounded PR-head context from README.md.
Review details and evidence
Material unknowns
- Whether a second OrcaReplay entry already exists elsewhere in README.md outside the observed exact-head slices (lines 80–88, 105–111), which would make this PR add a duplicate listing. A duplicate would be a post-merge cleanup item, not a correctness failure; it does not change the clear merge posture or the P3 documentation finding.
- Check: No pre-merge action required; if desired, grep the full README for the literal token
OrcaReplayat PR head to confirm a single occurrence.
- Check: No pre-merge action required; if desired, grep the full README for the literal token
LlamaPReview checks
- Read bounded PR-head context from
README.md. - Inspected matching PR-head repository snippets in
README.md. - Reviewed changed regions in
README.md.
LlamaPReview is an open-source pull request reviewer. Read the exact code that produced this review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Following up on #116, where you said "I stay with the zen of Python: practicality beats purity, and say things you use to test Python. ;-) Your PR is welcome."
Disclosure: I maintain OrcaReplay. Apache-2.0, free, no paid tier, no account.
Placement
## LLM & MCP Testing, alphabetically betweenmcp-server-fuzzerandSabot. One line, no other change.What it does, in the terms this list cares about
You have a Python test or agent script that calls an LLM. Run it once under
orca record; afterwardsorca replayruns the same script again with the provider unreachable, serving the recorded responses back. No key, no network to the vendor, no token spend, and the conversation is byte-identical. A session that failed in CI becomes something you can re-run locally.Nothing is installed into your code — no fixture, no decorator, no patching. The recorder moves a base-URL variable for the child process it launches, which is why it works the same for a bare
openaiscript, LangChain, CrewAI, LlamaIndex or LiteLLM without per-framework support.On the Python question from #116
The recorder itself is a Node CLI (
npm i -g orcareplay), which is what I flagged in the issue before sending this. Since then the Python side is also on PyPI:pip install orcareplay-openai-agentsadds an OpenAI Agents SDK tracing processor. So the entry is a tool you use to test Python, and there is now a Python package in it too — but I would rather you judged it on the first half, since that is the honest framing.Two limits, stated here rather than discovered later
curlreaches the network.Nearest existing entry
Cornellunder Mock and Stub is described as "record & replay mock server", which is the closest thing already listed. The difference worth knowing: Cornell is a server you point your code at, so the code under test must be configured for it; this wraps the process instead and needs no change to the code. Happy for you to decide the LLM section is the wrong home and move it there, or to decline — it is your list.Summary by Sourcery
New Features: