Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions crates/adaptive/src/closing/judge.rs
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,24 @@ Return JSON: {\"satisfied\": bool, \"blocker\": str, \"gap\": str,
- satisfied: did the run achieve the goal. Not \"did it finish\" — a run can
complete every node and achieve nothing.
- blocker: when not satisfied, one of
goal_not_met it tried and fell short. The ordinary case.
goal_not_met it tried and fell short. The ordinary case — INCLUDING a
run that died on its own mechanics: a miswired binding, a
bad command flag, a refused tool call. The graph can be
changed, so the episode can continue; say what broke in
the gap.
unverified something was produced but the evidence does not show it
working.
missing_evidence nothing was produced and there is nothing to judge.
needs_input a person has to answer something first.
missing_evidence nothing was produced AND another attempt would meet the
same nothing — the goal itself offers no evidence to
collect. NOT for mechanical failures; those are
goal_not_met.
needs_input a person has to answer something first — not \"the graph
failed to supply a value\", which is goal_not_met.
external_wait waiting on something outside this system.

goal_not_met and unverified let the loop try again with a changed approach;
the other three end the episode. Choose the terminal ones only when another
attempt genuinely cannot help.
- gap: one line on what is still missing. It is read by whoever plans the next
attempt, so name the missing thing, not the feeling.
- attributed_to: the node id that fell short, when the evidence says which.
Expand Down
51 changes: 51 additions & 0 deletions crates/adaptive/src/host.rs
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,22 @@ use tinyflows::model::{NodeKind, WorkflowGraph};
/// What a host permits. Read from that host's configuration, never guessed.
///
/// Construct it with [`HostFacts::unknown`] and fill in what is actually known:
/// One callable tool, with the argument shape a `tool_call` node must send.
///
/// The engine's tool capability takes `args` as an opaque value, so the only
/// place an author can learn a tool's argument names is here. A slug listed
/// without a fact is a tool the model can only misuse — observed in the
/// field as an author inventing `args.command` for a shell tool, twice,
/// spending the whole episode on a key name it was never shown.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct ToolFact {
/// The slug a `tool_call` node's config names.
pub slug: String,
/// The arguments it takes, in prose an author can follow: key names,
/// which are required, and what each means.
pub args: String,
}

/// every collection left empty and every `Option` left `None` disables its own
/// check rather than failing it.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
Expand All @@ -47,6 +63,13 @@ pub struct HostFacts {
/// Slugs permitted beyond the native ones. Empty *and* `native_tools`
/// empty means slugs are unchecked.
pub tool_allowlist: Vec<String>,
/// Argument documentation for the tools worth documenting.
///
/// Additive: a slug may appear in `native_tools` without a fact here —
/// that is "callable, shape unknown", which is what every host said
/// before this field existed.
#[serde(default)]
pub tools: Vec<ToolFact>,
/// Hosts `http_request` may reach. Empty means unchecked.
pub http_allowlist: Vec<String>,
/// Whether `code` nodes run at all. `None` means unknown.
Expand Down Expand Up @@ -93,6 +116,7 @@ impl HostFacts {
&& self.run_timeout_secs.is_none()
&& self.native_tools.is_empty()
&& self.tool_allowlist.is_empty()
&& self.tools.is_empty()
&& self.http_allowlist.is_empty()
&& self.allow_code.is_none()
&& self.shell_available.is_none()
Expand Down Expand Up @@ -293,6 +317,9 @@ impl HostFacts {
);
say("tool slugs that resolve", render_list(&self.native_tools));
say("tool slugs also allowed", render_list(&self.tool_allowlist));
for tool in &self.tools {
say(&format!("tool `{}` args", tool.slug), tool.args.clone());
}
say("http hosts reachable", render_list(&self.http_allowlist));
if let Some(allowed) = self.allow_code {
say(
Expand Down Expand Up @@ -376,6 +403,13 @@ mod tests {
run_timeout_secs: Some(600),
..HostFacts::unknown()
},
HostFacts {
tools: vec![ToolFact {
slug: "host:shell".into(),
args: "`script` (inline) or `script_path`".into(),
}],
..HostFacts::unknown()
},
] {
assert!(!facts.is_unknown(), "{facts:?}");
assert!(!facts.render().is_empty(), "and it renders");
Expand Down Expand Up @@ -600,6 +634,23 @@ mod tests {
assert!(problems[0].contains("never dispatched"), "{problems:?}");
}

#[test]
fn a_tool_fact_renders_its_argument_shape_into_the_prompt() {
let facts = HostFacts {
native_tools: vec!["host:shell".into()],
tools: vec![ToolFact {
slug: "host:shell".into(),
args: "`script` (inline text) or `script_path` (a file); NOT `command`".into(),
}],
..HostFacts::unknown()
};
let rendered = facts.render();
assert!(
rendered.contains("tool `host:shell` args:") && rendered.contains("script_path"),
"{rendered}"
);
}

#[test]
fn the_rendering_states_consequences_not_just_values() {
let facts = HostFacts {
Expand Down