Skip to content

feat(gateway): pace trigger replies in virtual time, add deadlines, and attribute tool calls - #196

Merged
earakely-scale merged 4 commits into
mainfrom
edgararakelyan/trigger-latency-deadlines-attribution
Oct 14, 2026
Merged

earakely-scale merged 4 commits into
mainfrom
edgararakelyan/trigger-latency-deadlines-attribution

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Every trigger reaction fired within seconds of the call that provoked it. Nothing could react to a call that never came. The env trajectory did not record who made a call, or which trigger it answered. Personas who answer on a human schedule need more from the engine. So do deadlines ("if nobody replies within an hour, escalate") and trajectories you can read causally.

What

Pacing in virtual time

  • An action's delay waits before the action runs. It is either an ISO-8601 duration ("PT2M") or a seeded exponential ({"dist": "exp", "mean": "PT10M", "seed": 7}), like every's.
    • It counts virtual time as the clock runs. PT2M at 60× is two real seconds, and it is real seconds while the clock is unarmed. It follows a rate change and holds while the clock is frozen. A re-arm forward counts its jump, so a harness that fast-forwards the clock brings pending replies due; a re-arm back counts nothing.
    • The delay runs from its cause. A queued call's delay counts from that call, and a due mark's from the mark, so a burst or a backlog doesn't push each reply a full delay later.
    • A delay under a barrier is a 400, since the barrier would hold the solver's call for the whole wait.
  • An executor's hours defer its nl actions until they open. The shape is {"start": "09:00", "end": "17:00", "days": ["mon", …], "utc_offset": "-07:00"}, and the deferral emits a deferred event with until.
    • Hours apply only while the clock is armed. They are worked out again after a re-arm, and a clear lifts them.
    • register_env_triggers takes executor_hours: {agent_name: hours}. It rejects names that no action uses.
  • GET /triggers/state marks a fire that is waiting out a delay or hours (waiting). prompt_agent's settle poll no longer waits one out.

Deadlines

  • A one-shot time trigger's when.unless: {tool, where?, roles?} makes it a deadline. A matching call before it is due cancels it: the new status is cancelled, with a cancelled event naming the call.
    • For an after trigger, the window opens when its anchor starts firing. A reply to the anchor's own action counts; calls before the anchor don't.
    • A reply after the deadline is due comes too late.
    • Cancelling a deadline also cancels the time triggers anchored on it, since they can no longer fire.
  • unless on a non-time trigger or an every trigger is a 400.

Attribution

  • Every gateway tool_call event records the caller's role.
  • Calls an executor makes while realizing a fire also record caused_by: [{trigger_id, action_index, fire}]. This lists every fire in flight for that executor. fire counts the trigger's fires, failed ones included.
  • This covers the MCP proxy, /step, and tools the gateway serves itself.

Lifecycle

  • /triggers/remove now cancels the trigger's fire, so a reply waiting out its delay never lands. A removed or cleared trigger's fire logs nothing more.

Compatibility

  • The triggers/v1 card declares action_delay, executor_hours and unless.
  • The step refuses a gateway that doesn't declare what its triggers use. An older gateway accepts these fields and ignores them: the delay would not wait, and the deadline would fire after the reply.

Examples

One small example per new capability. Every example was run through this branch's engine, and the times are what it logged. Times are virtual, at −05:00.

1. Delay a reply: delay

{"id": "john-replies",
 "when": {"type": "action", "tool": "slack_send_message", "where": {"channel_id": {"equals": "D_JOHN"}}},
 "actions": [{"type": "tool", "tool": "slack_send_message", "as": "john@example.com", "delay": "PT5M",
              "args": {"channel_id": "D_JOHN", "text": "Got it, I'll take a look."}}]}

The solver DMs John at 10:00. The trigger logs delayed (300 s), and John's reply goes out at 10:05. Any action type can carry a delay.

2. A human-paced, repeatable delay: a seeded delay

{"id": "denise-answers",
 "when": {"type": "action", "tool": "slack_send_message", "repeat": true,
          "where": {"channel_id": {"equals": "D_DENISE"}}},
 "actions": [{"type": "nl", "executor": "denise", "delay": {"dist": "exp", "mean": "PT4M", "seed": 7},
              "instruction": "Read the solver's latest message in D_DENISE and reply to it."}]}

The solver writes at 10:00, 10:10 and 10:20. Denise answers after 94 s, 39 s and 253 s. Those delays average 4 minutes, and the seed gives the same delays on every run.

3. Working hours: executor_hours

{"type": "register_env_triggers", "env_id": "workspace", "watch_roles": ["default"],
 "executor_hours": {"bob": {"start": "09:00", "end": "17:00", "days": ["mon", "tue", "wed", "thu", "fri"],
                            "utc_offset": "-05:00"}},
 "triggers": [{"id": "bob-reviews", "when": {"type": "action", "tool": "gdocs_create_document"},
               "actions": [{"type": "nl", "executor": "bob",
                            "instruction": "Review the solver's new doc and leave comments."}]}]}

The solver creates a doc at 18:30 on a Wednesday. Bob's action logs deferred with until Thursday 09:00, and runs then. Hours apply only while the clock is armed. Registering on the gateway directly, they go on executors.bob.hours.

4. A deadline at a fixed time: at + unless

{"id": "ap-chases",
 "when": {"type": "time", "at": "2026-07-01T12:00:00-05:00",
          "unless": {"tool": "gmail_send_message", "where": {"to": {"regex": "ap@example.com"}}}},
 "actions": [{"type": "tool", "tool": "gmail_send_message", "as": "ap@example.com",
              "args": {"to": "solver@example.com", "subject": "Still need your approvals"}}]}

If the solver emails AP before noon (11:40 here), the trigger becomes cancelled and nothing is sent. Otherwise AP's chaser goes out at noon. A reply after noon is too late.

5. A deadline that starts from another trigger: after + unless

{"id": "denise-dm", "when": {"type": "time", "at": "PT10M"},
 "actions": [{"type": "nl", "executor": "denise", "instruction": "DM the solver in D_DENISE asking for the Q3 numbers."}]},
{"id": "denise-nudges",
 "when": {"type": "time", "after": "denise-dm", "offset": "PT30M",
          "unless": {"tool": "slack_send_message", "where": {"channel_id": {"equals": "D_DENISE"}}}},
 "actions": [{"type": "nl", "executor": "denise", "instruction": "Nudge the solver: you still need the Q3 numbers."}]}

Denise's DM at 10:10 opens a 30-minute window. A solver message in D_DENISE at 10:25 cancels the nudge, and one sent before her DM doesn't count. With no reply, she nudges at 10:40. The window opens as soon as the DM starts firing, so a reply that beats Denise's executor reporting back still counts.

6. Cancelling a chain

{"id": "bob-escalates", "when": {"type": "time", "after": "denise-nudges", "offset": "PT1H"},
 "actions": [{"type": "nl", "executor": "bob", "instruction": "Escalate in #finance: the Q3 numbers are overdue."}]}

Add this to example 5. The 10:25 reply now cancels both triggers, and bob-escalates logs cancelled with anchor: denise-nudges, because it can no longer fire. Without that reply, Bob escalates an hour after the nudge.

7. A deadline only certain roles can answer: unless.roles

{"id": "chase-bob",
 "when": {"type": "time", "at": "PT2H", "unless": {"tool": "approve_bill", "roles": ["bob@example.com"]}},
 "actions": [{"type": "nl", "executor": "denise", "instruction": "Ping Bob: the bill still needs his approval."}]}

The solver's own approve_bill at 10:30 doesn't count; Bob's at 11:00 cancels the chaser. Without roles, the watched roles (the solver) answer a deadline.

8. Who made each call: role and caused_by

Nothing to configure. Every tool_call in the env trajectory now records its caller. Calls an executor makes while it is acting on a trigger also record which trigger fire they serve:

{"event_type": "tool_call", "role": "default", "tool_call": {"function_name": "slack_send_message"}}
{"event_type": "tool_call", "role": "denise@example.com", "tool_call": {"function_name": "slack_send_message"},
 "caused_by": [{"trigger_id": "denise-answers", "action_index": 0, "fire": 2}]}
All together: a DM, a nudge and an escalation
{
  "type": "register_env_triggers",
  "env_id": "workspace",
  "watch_roles": ["default"],
  "executor_hours": {"bob": {"start": "09:00", "end": "17:00", "days": ["mon", "tue", "wed", "thu", "fri"], "utc_offset": "-05:00"}},
  "triggers": [
    {"id": "denise-dm", "when": {"type": "time", "at": "PT10M"},
     "actions": [{"type": "nl", "executor": "denise",
                  "instruction": "DM the solver in D_DENISE asking for the Q3 numbers by end of day."}]},
    {"id": "denise-answers",
     "when": {"type": "action", "tool": "slack_send_message", "repeat": true,
              "where": {"channel_id": {"equals": "D_DENISE"}}},
     "actions": [{"type": "nl", "executor": "denise", "delay": {"dist": "exp", "mean": "PT4M", "seed": 7},
                  "instruction": "Read the solver's latest message in D_DENISE and reply to it."}]},
    {"id": "denise-nudges",
     "when": {"type": "time", "after": "denise-dm", "offset": "PT30M",
              "unless": {"tool": "slack_send_message", "where": {"channel_id": {"equals": "D_DENISE"}}}},
     "actions": [{"type": "nl", "executor": "denise", "delay": "PT2M",
                  "instruction": "Nudge the solver in D_DENISE: you still need the Q3 numbers."}]},
    {"id": "bob-escalates",
     "when": {"type": "time", "after": "denise-nudges", "offset": "PT1H",
              "unless": {"tool": "slack_send_message", "where": {"channel_id": {"equals": "D_DENISE"}}}},
     "actions": [{"type": "nl", "executor": "bob",
                  "instruction": "Post in #finance that the Q3 numbers are overdue and ask the solver for an ETA."}]}
  ]
}

denise and bob are deployed agents (agent_name). When you register through the gateway directly, the same hours go on executors.bob.hours. The walk-through below assumes the clock is armed on a Wednesday at 16:00 −05:00. It was checked against the engine.

Virtual time The solver answers at 16:25 The solver stays quiet
16:10 Denise DMs; the nudge is anchored for 16:41 same
16:25 the reply cancels denise-nudges, and the cancel cascades to bob-escalates (anchor: denise-nudges) —
16:26 Denise answers after a seeded delay of about 94 s —
16:41 — the nudge comes due, waits its PT2M, and goes out at 16:43; Bob's escalation is anchored for 17:43
17:43 — Bob is off, so it is deferred until Thursday 09:00
Thu 09:00 — Bob escalates

A reply that lands after a window has closed comes too late. A reply during the hour after the nudge cancels only Bob's escalation.

The firing log carries the new events, and the trajectory names who made each call and which fires it answers:

{"kind": "delayed", "trigger_id": "denise-answers", "action_index": 0, "virtual_seconds": 93.916, "executor": "denise"}
{"kind": "deferred", "trigger_id": "bob-escalates", "action_index": 0, "until": "2026-10-15T14:00:00Z", "executor": "bob"}
{"kind": "cancelled", "trigger_id": "denise-nudges", "provoking": {"tool": "slack_send_message", "role": "default"}}
{"kind": "cancelled", "trigger_id": "bob-escalates", "anchor": "denise-nudges"}

{"event_type": "tool_call", "role": "denise@example.com", "tool_call": {"function_name": "slack_send_message"},
 "caused_by": [{"trigger_id": "denise-answers", "action_index": 0, "fire": 1}]}

Tests

Unit

  • Validation.
  • Delays: fixed and seeded. The action runs at the delayed virtual instant. Queued calls and due marks are each timed from their own cause.
  • Hours: deferral, the unarmed case, and being worked out again after a re-arm or a clear.
  • A table of _next_opening cases.
  • The virtual wait against the real clock: rate, frozen, a re-arm forward and back, and ending when the clock changes generation.
  • Deadlines:
    • cancel and ignore cases;
    • a reply during the anchor's action;
    • a reply after the deadline is due;
    • the cascade;
    • unless: null.
  • Lifecycle: remove or clear mid-delay, and fire numbering across a failed fire.
  • Attribution through the MCP proxy and a gateway tool.
  • The step's executor_hours and feature checks.

Integration (local gateway)

  • A reply waits out its delay and then its executor's opening hours, checked on the virtual timestamps, and its writes name the fire.
  • A deadline fires on silence, and a reply cancels it.

Chaos (Linux VM, local gateways built from this branch)

Stub A2A executors on the host, the in-repo items server behind a gateway built from this branch, faults injected per scenario.

Scenario Fault Result
Delayed burst 10 provoking calls on a repeat trigger, PT30S delay at 60× 10 fires, one after another, none failed
Re-arm / clear mid-delay PT10M at 1×, re-armed to 600× after 2 s; PT20S with the clock cleared after 2 s fired 5 s in, then 20 s in, as counted
Deadline vs reply race 20 one-minute deadlines at 60×, replies spread across the due instant 10 cancelled, 10 fired; every deadline exactly one of the two
Executor down when its delay ends bob stopped during a PT2M delay action_failed under bob, bob's trigger re-armed, alice's fired
Two fires to one executor at once both write over the gateway's MCP endpoint each write caused_by both fires; a later call by that role carries none
Gateway from main the step against a gateway without these features refuses ['action_delay'], ['unless'], ['executor_hours'], registers nothing; sent raw, that gateway runs a PT10M delay at once and fires a deadline after its reply
Frozen clock mid-wait rate 0 during a delay and an hours deferral both held 8 s (firing, waiting), then acted 2.7 s after the clock ran again; no failures
Hours across a re-arm and a clear deferred Friday 18:00 → Monday 09:00 re-arm to Monday 10:00 released it in 3 s; a clear released another in 3 s
Remove mid-delay /triggers/remove during an 8 s delay no reply; log is added, detected, delayed, removed
Reply during the anchor's DM the solver answers before the DM's A2A task reports back deadline cancelled, its escalation cancelled (anchor: deadline), nothing anchored or fired
Regression: named executors and when.roles kill mid-fire, hang past timeout, network partition, 25-call burst, retried registration, failed state, ping-pong, concurrent chains, barrier, state trigger, kill mid-chain all pass
Regression: version skew a gateway from before named executors refuses ['named_executors'] and ['provoking_roles']; sent raw, it misroutes and misreads them, as before

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no actionable issue remains from this review.

Summary

Adds virtual-time delays, executor working hours, cancellable deadlines, and tool-call attribution.

  • Trigger actions wait for their delay and executor hours before they run.
  • One-shot time triggers can cancel on a matching call or stop when removed.
  • Tool-call events show who called and which trigger fires led to executor calls.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Provoking call or due mark] --> B[Wait for action delay]
  B --> C{Executor hours apply?}
  C -->|Yes| D[Wait for opening]
  C -->|No| E[Run action]
  D --> E
  F[Forward clock jump] --> B
  G[Clock re-arm or clear] --> D
  E --> H[Record caller and active fires]
  I[Matching reply before deadline] --> J[Cancel deadline and dependent time triggers]
Loading

Reviews (3) · Last reviewed commit: "fix(gateway): count a forward re-arm tow..." · Reviewed by Greptile

earakely-scale and others added 2 commits October 13, 2026 21:53
…nd attribute tool calls

Every trigger reaction fired within seconds, nothing could react to a call
that never came, and the env trajectory did not say who made a call.

- An action's `delay` (an ISO-8601 duration, or a seeded exponential like
  `every`) waits that much virtual time before it runs, and an executor's
  `hours` defer its nl actions until its working hours open. Both follow the
  gateway clock's rate; while the clock is unarmed a delay is real seconds
  and hours do not apply. register_env_triggers takes `executor_hours`.
- A one-shot time trigger's `when.unless` makes it a deadline: a matching
  call before it is due cancels it (new status `cancelled`). For an `after`
  trigger the window opens when its anchor fires.
- Every gateway tool_call event records the caller's `role`, and calls an
  executor makes while realizing a fire record `caused_by`.

The card declares `action_delay`, `executor_hours` and `unless`, and the step
refuses an older gateway, which would ignore them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…le deadlines cleanly

- A delay now counts virtual time on a clock stopwatch (Clock.elapsed), so
  it follows a rate change exactly, holds while the clock is frozen instead
  of dividing by zero, and doesn't count a re-arm's jump. It runs from its
  cause: a queued call's delay from that call, a due mark's from the mark,
  so a burst or a backlog no longer pushes each reply a full delay later.
- Working hours are reckoned again after a re-arm and lifted by a clear,
  and release once the opening is reached.
- An `after` deadline's window opens when its anchor starts firing, so a
  reply to the anchor's own action cancels it; a reply after the deadline
  is due comes too late. Cancelling a deadline also retires the time
  triggers anchored on it.
- remove() cancels the trigger's fire, and a removed or cleared trigger's
  fire logs nothing more. caused_by numbers every fire, failed ones too.
- state() marks a fire that is waiting, and prompt_agent's settle poll no
  longer waits one out. `unless: null` means none; a barrier is refused for
  an executor with hours, as for a delay; malformed hours.days is a 400.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner October 14, 2026 06:04
Comment thread src/agent_env/env/gateway/triggers.py Outdated
Comment thread src/agent_env/env/gateway/triggers.py
Comment thread src/agent_env/env/gateway/triggers.py Outdated
earakely-scale and others added 2 commits October 13, 2026 23:28
…e day in hours

A batch of due marks now reckons every mark's delay start up front, so a
re-arm during one mark's wait doesn't count toward the next. A working
window's length is taken modulo a timedelta day instead of 24 * 60.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A harness that fast-forwards the clock by re-arming it, as a staged warm-up
does, now brings pending delays due: Clock.elapsed() adds the jump of each
re-arm forward. A re-arm back still counts nothing, so a reset never
shortens a wait.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit e9d2e60 into main Oct 14, 2026
15 of 17 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/trigger-latency-deadlines-attribution branch October 14, 2026 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant