Skip to content

git: clarify when NOT to call status/diff/log — cuts agent extra-call rate (measured +21.7 pts) - #4520

Closed
sujugithub wants to merge 1 commit into
modelcontextprotocol:mainfrom
sujugithub:hitrate/git-descriptions
Closed

sujugithub wants to merge 1 commit into
modelcontextprotocol:mainfrom
sujugithub:hitrate/git-descriptions

Conversation

@sujugithub

Copy link
Copy Markdown

What

Rewrites 7 tool description strings in src/git/src/mcp_server_git/server.py. No code or behavior changes — only the text agents read when choosing tools.

Why

I measured how a mid-tier agent actually uses this server (12 realistic scenarios × 5 runs each, live server, every tool call recorded). Tool selection is already excellent — 100% hit rate, the diff-trio naming works. The failure mode is ritual verification calls: after git_add, agents call git_status or git_diff_staged to check it worked; before git_show HEAD, they call git_log to find a hash they don't need. A quarter of all runs were padded this way.

Measured effect of this diff (nothing else changed)

metric before after
tool hit rate 100% 100%
argument correctness 100% 100%
strict success (right tool + args + no extra calls) 75.0% 96.7%

Method: agent = gpt-oss-120b (deliberately mid-tier — weaker agents are hurt most by underspecified descriptions), N=5 runs per scenario, descriptions applied via in-memory override so before/after used the identical server build. Happy to share the full per-run data and scenario suite, or re-run against any revision of the wording.

Notes

  • The pattern in each rewrite: state that the operation is complete on success ("does not require calling git_status to verify"), and say which sibling tool covers the adjacent intent.
  • One deviation from the measured text: the tested git_show wording implied revision was optional; since GitShow.revision is required, this PR says Pass revision=\"HEAD\" instead — same intent, schema-accurate.
  • The measurement harness (scenarios + runner) is being open-sourced; I'll link it here once public.

🤖 Generated with Claude Code

Measured with an agent harness (12 scenarios x 5 runs each, live server):
tool selection was already perfect (100% hit rate) but agents padded runs
with ritual git_status/git_diff_staged/git_log verification calls. With only
these description changes: strict success (right tool + right args + no
extra calls) 75.0% -> 96.7%. No behavior changes.
@sujugithub

Copy link
Copy Markdown
Author

The measurement harness is now public: https://github.com/sujugithub/hitrate — the git scenario suite is scenarios/git/git.yaml and the per-run data behind the table above is in results/ (files dated 2026-07-13, git-*). Happy to re-run against any revised wording.

@cliffhall cliffhall added the closed-out-of-scope Closed: out of scope here; listings and new servers go to the MCP Registry (#4875) label Oct 20, 2026
@cliffhall

Copy link
Copy Markdown
Member

Thank you for this pull request, and apologies for the long wait for a response.

These servers are reference implementations: they exist to show how each part of MCP is used, not to be general-purpose products, so new features are taken selectively, and this change isn't one the maintainers plan to take. This repository also now accepts issues, not pull requests, from anyone but its maintainers; the policy is in CONTRIBUTING.md.

This is not a judgment on the idea or on you. A server in the MCP Registry may well offer it, and anyone is welcome to build on these servers to do so. If you think we've misread it, please open an issue describing the problem and the expected behavior.

This is part of the backlog triage in #4875.

@cliffhall cliffhall closed this Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

closed-out-of-scope Closed: out of scope here; listings and new servers go to the MCP Registry (#4875)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants