Skip to content

fix(adaptive): judge blocker taxonomy + tool argument facts - #59

Merged
sanil-23 merged 1 commit into
tinyhumansai:mainfrom
sanil-23:fix/adaptive-judge-taxonomy-toolfacts
Aug 17, 2026
Merged

fix(adaptive): judge blocker taxonomy + tool argument facts#59
sanil-23 merged 1 commit into
tinyhumansai:mainfrom
sanil-23:fix/adaptive-judge-taxonomy-toolfacts

Conversation

@sanil-23

@sanil-23 sanil-23 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Two independent field findings (see commit message): (1) the judge classified a retryable command error as terminal missing_evidence, ending an episode with budget remaining — the taxonomy now states that mechanical failures are goal_not_met (the graph can change), missing_evidence means another attempt meets the same nothing, and terminal labels are for when retrying genuinely cannot help; (2) HostFacts advertised tools as bare slugs, so an author invented args.command for a shell tool twice — ToolFact { slug, args } now carries each tool's argument shape, rendered per tool into the authoring prompt, covered by is_unknown and the configured-alone test.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added documentation for available tools and their expected arguments.
    • Tool information is now included in generated prompts and host-state reporting.
  • Bug Fixes

    • Improved handling of unknown states involving unavailable or undocumented tools.
    • Clarified failure classifications, including when retries are possible and when an episode should end.
    • Refined evidence and input requirements for more accurate outcome assessment.

Two field findings from driving the loop on a live host:

- The judge classified a retryable command error (a bad CLI flag) as
  terminal missing_evidence, ending an episode with budget remaining.
  The taxonomy now states it: mechanical failures — a miswired binding,
  a bad flag, a refused tool call — are goal_not_met, because the graph
  can be changed; missing_evidence is for goals where another attempt
  meets the same nothing; and the terminal labels are chosen only when
  retrying genuinely cannot help.

- HostFacts advertised tools as bare slugs, so an author invented
  `args.command` for a shell tool — twice, spending the episode on a
  key name it was never shown. ToolFact { slug, args } carries each
  tool's argument shape in prose, rendered one line per tool into the
  authoring prompt, covered by is_unknown and the configured-alone test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Aug 17, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

Failed to post review comments.

GitHub was unavailable or timed out while CodeRabbit was posting the review. Please request a new review later if the pull request still needs one. This happened while posting 2 inline comments. Use @coderabbitai full review to retry the review.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 9e499a20-03da-4f07-ac81-b2d4b66b0834

📥 Commits

Reviewing files that changed from the base of the PR and between 5387659 and a786309.

📒 Files selected for processing (2)
  • crates/adaptive/src/closing/judge.rs
  • crates/adaptive/src/host.rs

Included review availability: Your plan includes up to 1 review per rolling hour; 0 remain after this review.

🧰 Additional context used
📓 Path-based instructions (1)
**/*.rs

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.rs: Target Rust 2024 with MSRV 1.85; keep the crate compatible with that toolchain.
Use #![forbid(unsafe_code)]; do not introduce unsafe Rust code.
Keep #![warn(missing_docs)] satisfied: every public item must have a doc comment.
Preserve the host-agnostic design: never hard-code LLM, tool, HTTP, or persistence vendors; route new outside-world effects through a caps capability trait instead of direct dependencies.
Keep the workflow model declarative: do not add arbitrary embedded scripting to the workflow model; treat code execution only as a sandboxed capability, not as model logic.

Files:

  • crates/adaptive/src/closing/judge.rs
  • crates/adaptive/src/host.rs
🧠 Learnings (1)
📚 Learning: 2026-08-17T06:47:19.808Z
Learnt from: sanil-23
Repo: tinyhumansai/tinyflows PR: 55
File: crates/adaptive/src/closing/repair.rs:250-257
Timestamp: 2026-08-17T06:47:19.808Z
Learning: When generating durable adaptive identifiers such as shape_id, authored-graph fingerprints, or variant_id, use the shared crates/adaptive/src/reuse.rs::digest_hex implementation rather than introducing a separate hash format. Preserve its FNV-1a 64-bit behavior and 16-character hexadecimal output, and add or maintain published test-vector coverage for any changes.

Applied to files:

  • crates/adaptive/src/closing/judge.rs
  • crates/adaptive/src/host.rs
📝 Walkthrough

Walkthrough

The change adds documented tool argument shapes to HostFacts and includes them in rendered prompts. It also updates judge guidance for mechanical failures, missing evidence, input requirements, retries, and episode termination.

Changes

Adaptive context and closing

Layer / File(s) Summary
Document tool facts in host context
crates/adaptive/src/host.rs
Adds public ToolFact data and the HostFacts::tools collection. Unknown-state detection and host rendering now include configured tool facts. Tests cover state handling and prompt output.
Clarify blocker classification and retries
crates/adaptive/src/closing/judge.rs
Updates blocker guidance for mechanical failures, missing evidence, graph value-supply failures, retries, and episode termination.

Estimated code review effort: 2 (Simple) | ~15 minutes

Merge Risk: 🟡 Moderate · up to a7863

The PR improves retry classification and tool argument guidance, but it can still end retryable episodes too early and may break downstream users that construct the public host-facts struct directly. These issues should be fixed or explicitly accepted before merging.

Possibly related PRs

Poem

A rabbit reads each tool’s shape,
And marks the host facts clear.
Mechanical blocks now retry right,
While terminal paths stay near.
Evidence rules grow crisp and bright—
Hop, review, and cheer!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes both the judge blocker taxonomy changes and the addition of tool argument facts.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Warning

Review ran into problems

🔥 Problems

Git: Failed to clone repository. Please run the @coderabbitai full review command to re-trigger a full review. If the issue persists, set path_filters to include or exclude specific files.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sanil-23
sanil-23 merged commit 769af9a into tinyhumansai:main Aug 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant