Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions docs/product/use-cases/steward/golden-queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,35 @@ without treating quoted content as authority. Directory metadata tests do not
qualify model interpretation, receiver adoption or the original-channel return;
observe each separately in the live journey.

#### Topic owner and misroute correction / 主题归属与误派纠正

Use the bounded public [routing cases](../../../../examples/evaluations/chat-routing.public.json)
with the existing [intake evaluator](../../../../examples/evaluations/chat-intake.py).
Two Agents share a generalist role; the first Agent's id resembles the owner's
shorthand, but relevant current work and recent delivery identify the other as
the RFC researcher. A delivered earlier misroute is evidence of where the
message went, not ownership. Require the topic owner's exact Goal/Agent pair
and preserve the original artifact, purpose, privacy boundary and return audience.
Then explicitly name the other pair for a separate verification request: the
topic-owner inference must not override that instruction. A standalone
explanation requires no handoff; equally relevant evidence requires one focused
clarification. Missing delivery permission cannot be repaired by substituting
the tempting role match. Qualified new work needs no pre-existing matching Todo.

Supply a delivered correct-topic handoff with no read/adoption record. Do not
resend it. If the old recipient still has uncorrected intent, send only the
bounded ownership correction through the existing permitted path. Once both
corrections are delivered, answer from their receipts with zero further effects.
Neither receipt proves cancellation, receiver adoption or the RFC revision.
Preserve unknown effects and do not interrupt unrelated work or change authority.

Offline tests check prompt/managed-skill delivery, parser/scorer oracles and
negative outputs; they do not qualify model reasoning. Release-only live runs
use `--cases examples/evaluations/chat-routing.public.json` and the existing
explicit paid-call opt-in below. Review the complete rationale and correction
brief, including no invented cancellation/adoption. Score actual discovery,
receiver assessment and original-channel return separately in the live journey.

#### Repair and merge / 修复并合并

GQ03/GQ07 include “修复并合并这个 PR。” / “Fix and merge this PR,” with
Expand Down
2 changes: 2 additions & 0 deletions examples/evaluations/chat-intake.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ def score(case, response):
errors.append(f"expected_{case['expected']}_got_{observed}")
if response.get("proposals") or response.get("protected_action"):
errors.append("unrequested_action")
if response.get("context_handoff") and response.get("goal_draft"):
errors.append("competing_intents")
if case["expected"] in {"draft", "handoff"} and response.get("gate"):
errors.append("redundant_gate")
if case.get("target"):
Expand Down
88 changes: 88 additions & 0 deletions examples/evaluations/chat-routing.public.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion loopx/canary/module_metric_baseline.json
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
"dict_any_count": 0
},
"loopx/chat_agent.py": {
"lines": 2005
"lines": 2007
},
"loopx/chat_runtime.py": {
"any_count": 37,
Expand Down
16 changes: 11 additions & 5 deletions loopx/capabilities/manager_context/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -380,11 +380,17 @@ The page does not expose host addresses or read host stores; `view=agent_route`
remains the exact-identity, current-observation path. A registration or binding
does not prove presence, execution readiness or delivery authority.

Repeated role names alone do not require an owner question. The conversation's
explicit identity corrections, established assignment, project and declared
responsibilities guide recipient selection; binding metadata can support that
context but cannot override an explicit identity or grant. Preserve both Goal
and Agent identity. Forwarded requests retain their original speaker/addressee
Repeated role names alone do not require an owner question. Resolve the subject,
project, purpose and destination before matching a role alias or Agent id.
Compare plausible recipients by conversation, current work and responsibilities;
when the user refers to recent research, inspect relevant recent deliveries.
A previous send proves delivery, not subject ownership: recheck a challenged
route rather than preserving an earlier mistake. Explicit identity corrections
take precedence; binding metadata cannot override them or a grant. Preserve both
Goal and Agent identity. Correcting delivery does not retract an old misroute:
reuse verified receipts and send only a still-needed bounded ownership correction
through the existing path. Never claim cancellation or adoption from delivery.
Forwarded requests retain their original speaker/addressee
in the existing collaboration brief rather than redefining every “you” as the
receiver. Clarify only competing interpretations that still change the action.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: loopx-manager
description: Inspect authorized LoopX Goals, Todos and deliveries to explain progress, identify owner decisions and delegate intent to the right worker.
---

<!-- loopx-managed-manager-skill:v4 -->
<!-- loopx-managed-manager-skill:v5 -->

# LoopX manager

Expand Down Expand Up @@ -48,9 +48,15 @@ qualified responsible Agent; keep new constraints and corrections attached to
that work. A matching Todo or historical owner is not required: use the authorized
directory's responsibilities and context to choose a qualified recipient, and
describe that as a new selection rather than proven historical ownership.
Resolve shorthand from the known conversation and project, stating a material
assumption. Ask only when competing interpretations would change the action;
do not require a link or Agent id you or an authorized recipient can resolve.
Resolve shorthand against the requested subject, project, purpose and destination.
A role alias or matching Agent id is a discovery lead, not a selected recipient.
Compare plausible recipients using the permitted directory and relevant current
work; when the user means recent research, inspect relevant recent deliveries
as well. Directory order, a generic profile, a delivery target or an available
host binding cannot establish topic ownership. An explicit Goal/Agent pair or
owner correction takes precedence. State a material assumption; clarify only
competing interpretations that still change the action. Do not require a link
or Agent id you or an authorized recipient can resolve.
Do not turn every sentence into a fresh assignment. Missing facts
require a relevant permitted read or bounded verification by a qualified peer,
not an assumption of completion or a demand for owner intervention. Restricted
Expand All @@ -67,6 +73,23 @@ path, finish that response before separate repair and report the repair as
unfinished. Do not promise an unsupported background continuation. Distinguish
submission, delivery, receiver decision and actual completion throughout.

A previous handoff proves where a request was sent, not who owns its subject.
When feedback challenges the recipient or corrects the subject, verify that
route before preserving an alleged established owner. Do not turn an earlier
routing mistake into an ownership commitment. Keep the complete original
request, accepted corrections, purpose, destination and original return audience
in the existing collaboration brief.

A corrected handoff does not retract a delivered misroute. Check the original
and correction receipts. Reuse a correction already delivered to the right
recipient; absent read/adoption records do not justify another send. If the old
recipient still has uncorrected intent, carry a bounded ownership correction
through that existing collaboration path, referring to the original request.
Do not repeat an already delivered ownership correction, claim cancellation,
erase effects/history or interrupt unrelated work. Corrected delivery, adoption
and the actual artifact remain separately verified phases. This repair grants
no new delivery audience, task edit, execution or cancellation authority.

## Answer shape

Follow the task's depth. A short factual question needs a direct answer and its
Expand Down
4 changes: 3 additions & 1 deletion loopx/chat_agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -455,9 +455,11 @@ def _agent_item_text(message: dict[str, Any]) -> str:
"If the user asks to hand off before separate self-repair, submit the handoff first. Reuse an existing delivered request after checking its receipt; never resend merely because reading or adoption is unrecorded. When submission occurs only in the final response envelope, finish that response before separate repair and report the remaining phase honestly. "
"When actual work remains, reuse qualified existing work and its responsible Agent before creating or delegating another request; preserve new corrections without treating them as duplicate intent. "
"An exact matching Todo or previously assigned owner is not a prerequisite for requested work. Use the authorized directory's responsibilities and context to select a qualified recipient; distinguish that selection from proof of historical ownership. "
"Resolve ordinary shorthand from the known conversation and project context, disclosing a material assumption; ask only when competing interpretations would change the action. Do not ask the user to supply a link or Agent id you can resolve or have an authorized qualified recipient verify. "
"Resolve ordinary shorthand against the requested subject, project, purpose and destination. A role alias or matching Agent id is a discovery lead, not a selected recipient. Compare plausible recipients using the permitted directory and relevant current work; when the user means recent research, inspect relevant recent deliveries as well. Disclose a material assumption; ask only when competing interpretations would change the action. Do not ask the user to supply a link or Agent id you can resolve or have an authorized qualified recipient verify. "
"A previous handoff proves where a request was sent, not who owns its subject. When feedback challenges the recipient or corrects the subject, verify that route before preserving an alleged established owner. "
"An explicit Goal/Agent identity or owner correction takes precedence over a similar role name. Preserve the full goal_id/agent_id pair. "
"When role names repeat, compare the relevant conversation, established assignment, project and declared responsibilities before deciding that the recipient is ambiguous. Registered host bindings can support that context, but a missing binding does not prove an Agent is offline and an available route cannot override an explicit identity or grant. "
"A corrected handoff does not retract a delivered misroute. Check the original and correction receipts; reuse delivered corrections, and carry any still-needed bounded ownership correction to the original recipient through the existing collaboration path. Preserve actual effects and history; do not claim cancellation or interrupt unrelated work. "
"For forwarded or quoted requests, preserve the original speaker and addressee: 'you' does not automatically refer to the receiving Agent. Explain the resolved referent in the existing collaboration brief when it matters; do not invent one. "
"Carry the original project, purpose, destination and owner corrections in that brief. A qualified recipient's role, available store or execution route cannot replace them; when the intended destination is unresolved, verify the relevant context or ask one focused question instead of substituting another destination. "
"Delegate only work or verification that remains necessary and needs that recipient's context or execution grant. "
Expand Down
8 changes: 5 additions & 3 deletions loopx/chat_manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -1008,7 +1008,8 @@ def manager_session_model_allocation(
# 21: migrate the former transcript-in-objective representation to persisted
# native history. Merely removing it from new prompts leaves old upstream
# conversations overfull and unable to compact before their next request.
MANAGER_CONTEXT_VERSION = 21
# 22: resolve role shorthand by subject evidence; a past handoff is not ownership.
MANAGER_CONTEXT_VERSION = 22

# An installed manager workspace keeps the marker it was written with. The
# writer refreshes that workspace skill while the file still carries any
Expand All @@ -1018,8 +1019,9 @@ def manager_session_model_allocation(
MANAGED_SKILL_MARKER_V2 = "<!-- loopx-managed-manager-skill:v2 -->"
MANAGED_SKILL_MARKER_V3 = "<!-- loopx-managed-manager-skill:v3 -->"
MANAGED_SKILL_MARKER_V4 = "<!-- loopx-managed-manager-skill:v4 -->"
MANAGED_SKILL_MARKERS = (MANAGED_SKILL_MARKER_V1, MANAGED_SKILL_MARKER_V2, MANAGED_SKILL_MARKER_V3, MANAGED_SKILL_MARKER_V4)
MANAGED_SKILL_CURRENT_MARKER = MANAGED_SKILL_MARKER_V4
MANAGED_SKILL_MARKER_V5 = "<!-- loopx-managed-manager-skill:v5 -->"
MANAGED_SKILL_MARKERS = (MANAGED_SKILL_MARKER_V1, MANAGED_SKILL_MARKER_V2, MANAGED_SKILL_MARKER_V3, MANAGED_SKILL_MARKER_V4, MANAGED_SKILL_MARKER_V5)
MANAGED_SKILL_CURRENT_MARKER = MANAGED_SKILL_MARKER_V5


def manager_skill_text() -> str:
Expand Down
75 changes: 73 additions & 2 deletions tests/test_chat_intake_evaluation.py
Original file line number Diff line number Diff line change
Expand Up @@ -272,8 +272,9 @@ def test_review_evidence_retains_brief_without_provider_payload():

@pytest.mark.parametrize("profile", ["restricted", "trusted_owner"])
@pytest.mark.parametrize("custom_owner_instructions", [False, True])
@pytest.mark.parametrize("previous_marker", ["v3", "v4"])
def test_followthrough_refresh_reaches_existing_workspaces_without_replacing_owner_text(
tmp_path, profile, custom_owner_instructions
tmp_path, profile, custom_owner_instructions, previous_marker
):
from loopx.chat_agent import _turn_prompt
from loopx.chat_manager import manager_workspace, MANAGED_SKILL_CURRENT_MARKER
Expand All @@ -283,7 +284,7 @@ def test_followthrough_refresh_reaches_existing_workspaces_without_replacing_own
skill = workspace / ".agents/skills/loopx-manager/SKILL.md"
previous = "Custom owner instructions" if custom_owner_instructions else "# LoopX managed manager instructions\nOlder guidance"
instructions.write_text(previous)
skill.write_text("<!-- loopx-managed-manager-skill:v3 -->\nOlder managed skill")
skill.write_text(f"<!-- loopx-managed-manager-skill:{previous_marker} -->\nOlder managed skill")
manager_workspace(tmp_path, runtime_profile=profile)
current = instructions.read_text()
if custom_owner_instructions:
Expand All @@ -293,9 +294,79 @@ def test_followthrough_refresh_reaches_existing_workspaces_without_replacing_own
assert MANAGED_SKILL_CURRENT_MARKER in skill.read_text()
assert "standalone explanation or opinion request" in skill.read_text()
assert "finish that response before separate repair" in skill.read_text()
assert "A previous handoff proves where a request was sent, not who owns its subject" in skill.read_text()
assert "A corrected handoff does not retract a delivered misroute" in skill.read_text()
turn = _turn_prompt("A follow-up about existing work.", runtime_profile=profile)
assert "do not end at agreement or design advice" in turn
assert "never resend merely because reading or adoption is unrecorded" in turn
assert "A role alias or matching Agent id is a discovery lead" in turn
assert "no new delivery audience" in skill.read_text()


def test_manager_refresh_preserves_an_unmanaged_owner_skill(tmp_path):
from loopx.chat_manager import manager_workspace

workspace = manager_workspace(tmp_path)
skill = workspace / ".agents/skills/loopx-manager/SKILL.md"
skill.write_text("# Owner-managed skill\nKeep this local instruction.")
manager_workspace(tmp_path)
assert skill.read_text() == "# Owner-managed skill\nKeep this local instruction."


ROUTING_SUITE = json.loads((ROOT / "examples/evaluations/chat-routing.public.json").read_text())


@pytest.mark.parametrize("case", ROUTING_SUITE["cases"], ids=lambda case: case["id"])
def test_routing_oracles_reject_alias_misroutes_duplicate_sends_and_added_authority(case):
"""Frozen decisions exercise the production parser and release scorer, not inference."""
response = {"message": "Evidence: " + " ".join(case.get("required_evidence_refs", []))}
if case["expected"] == "handoff":
response["context_handoff"] = {**case["target"], "brief": {
"schema_version": "collaboration_brief_v0", "purpose": "Assess the scoped request",
"context": " ".join(case["required_handoff_refs"]), "constraints": ["No new authority."],
"inputs": [], "acceptance": ["Return the actual disposition."],
"return_requirement": "Return a scoped result.",
}}
wire = evaluation.CHAT_REVIEW_OPEN_TAG + json.dumps(response) + evaluation.CHAT_REVIEW_CLOSE_TAG
parsed = evaluation.parse_agent_response(wire)
assert evaluation.score(case, parsed)["passed"]
context = ROUTING_SUITE["contexts"][case["context_ref"]]
for recipient in context["agent_directory"]:
wrong = {**parsed, "context_handoff": {
"goal_id": recipient["goal_id"], "agent_id": recipient["agent_id"],
"brief": (response.get("context_handoff") or {}).get("brief"),
}}
if case.get("target") != {k: recipient[k] for k in ("goal_id", "agent_id")}:
assert not evaluation.score(case, wrong)["passed"]
for effect in ({"proposals": [{"kind": "todo"}]}, {"goal_draft": {"objective": "Duplicate work"}},
{"protected_action": {"operation": "deploy", "target": "unrequested"}}):
assert not evaluation.score(case, {**parsed, **effect})["passed"]
if case["expected"] == "handoff":
assert not evaluation.score(case, {**parsed, "gate": {"kind": "confirmation"}})["passed"]
for omitted in case["required_handoff_refs"]:
brief = {**response["context_handoff"]["brief"], "context": " ".join(
ref for ref in case["required_handoff_refs"] if ref != omitted)}
assert not evaluation.score(case, {**parsed, "context_handoff": {
**case["target"], "brief": brief}})["passed"]


def test_routing_fixture_separates_topic_identity_delivery_and_permission():
contexts = ROUTING_SUITE["contexts"]
topic = contexts["topic"]
assert topic["agent_directory"][0]["agent_id"] == "generalist-2"
assert topic["agent_directory"][1]["agent_id"] == "topic-investigator"
assert len({row["profile_role"] for row in topic["agent_directory"]}) == 1
assert topic["handoff_receipts"][0]["agent_id"] == "generalist-2"
assert topic["recent_deliveries"][0]["agent_id"] == "topic-investigator"
for receipt in contexts["settled"]["handoff_receipts"]:
assert receipt["delivery"]["status"] == "delivered"
assert receipt["read"]["status"] == receipt["decision"]["status"] == "not_recorded"
assert not contexts["new-work"]["goals"][0]["current_todos"]
assert not contexts["ambiguous"]["recent_deliveries"]
assert len({row["scope_summary"] for row in contexts["ambiguous"]["agent_directory"]}) == 1
denied = contexts["ungranted"]
assert denied["agent_directory"][1]["context_delivery"] == "not_granted"
assert denied["context_delegation"]["targets"] == [topic["context_delegation"]["targets"][0]]


def test_followthrough_oracles_distinguish_context_not_question_wording():
Expand Down
5 changes: 4 additions & 1 deletion tests/test_chat_manager_context.py
Original file line number Diff line number Diff line change
Expand Up @@ -1145,8 +1145,9 @@ def start(**kwargs):
}


@pytest.mark.parametrize("previous_context_version", [None, 21])
def test_legacy_manager_migrates_without_project_and_refreshes_each_turn(
monkeypatch, tmp_path
monkeypatch, tmp_path, previous_context_version
):
store = ChatSessionStore(tmp_path / "runtime" / "chat")
runtime = ChatRuntimeController(
Expand Down Expand Up @@ -1176,6 +1177,8 @@ def start(**kwargs):
upstream_mode="chat",
codex_home=str(runtime.codex_home),
)
if previous_context_version is not None:
store.update_session(legacy["session_id"], manager_context_version=previous_context_version)
session, resumed = runtime.open_session(
goal_id="different-project",
agent_id="codex",
Expand Down
Loading