ateapi,atenet: fail fast when resuming a CRASHED or DELETING actor by ygao-g · Pull Request #3402 · agent-substrate/substrate

Yuan Gao (ygao-g) · 2026-10-28T20:12:26Z

A routed request to a CRASHED actor parks for the router's whole 5 s budget and answers 503 after about 19 ResumeActor calls, holding one parking slot throughout. The state never clears on its own: only RevertActor moves the actor on. The request should fail at once, with one ResumeActor call and no slot.

Fixes #3269

One routed request before the fix, as the repro recorded it:

sequenceDiagram
    autonumber
    participant C as client
    participant E as Envoy ext_proc
    participant R as router ActorResumer
    participant A as ateapi ResumeActor
    C->>E: POST / for demo/x (ACTOR_STATE_CRASHED)
    E->>R: RequestHeaders
    R->>A: ResumeActor
    A-->>R: FailedPrecondition (AssignWorker prerequisite not met)
    Note over R: retryable: parks, takes one lot slot
    loop 18 more times, 100 ms x 1.1^n, until the 5 s budget ends
        R->>A: ResumeActor
        A-->>R: FailedPrecondition, unchanged
    end
    Note over R: budgetExhaustedError wraps the last error
    R-->>E: 503 actor demo/x unavailable
    E-->>C: 503 after 5.0 s

Design. ateapi marks the two states that do clear on their own. ensureWorkerAssigned attaches an ErrorInfo detail (domain ate.dev, reason ACTOR_SUSPENDING or ACTOR_REVERTING) to its FailedPrecondition for those states; the message does not change. The router parks a FailedPrecondition only when it carries one of those reasons and fails fast on every other one with the unchanged 503 body, actor <ref> unavailable: <description>. The router still knows no actor states, only the two reasons.

The same request after the fix:

sequenceDiagram
    autonumber
    participant C as client
    participant E as Envoy ext_proc
    participant R as router ActorResumer
    participant A as ateapi ResumeActor
    C->>E: POST / for demo/x (ACTOR_STATE_CRASHED)
    E->>R: RequestHeaders
    R->>A: ResumeActor
    A-->>R: FailedPrecondition, no ErrorInfo reason
    Note over R: isTransientResumeErr false, no retry, no lot slot
    R-->>E: 503 actor demo/x unavailable, AssignWorker prerequisite not met
    E-->>C: 503 after 14 ms

Behavior change: besides a CRASHED, DELETING or UNSPECIFIED actor, a missing template, a volume without a storage ID, a worker without a node name and a failed DetachActor no longer park. Each fails at once with the 503 it used to get after the budget.

Unchanged: the status code and the message text. The AgentGateway dataplane resumes through ate-api-server itself and still parks. A resume that crashes the actor inside validateAssignedWorker returns Aborted, so that one request still holds a slot for one backoff before the 503. Whether the fail-fast status should be a 4xx, 409 or 400, is open on the errors.go thread.

Review in commit order: apierror adds the Reason type, the two reasons and the detail; ateapi attaches them in ensureWorkerAssigned; atenet/router parks only on them; e2e adds the test and the dataplane's fails-fast flag.

Tests. The reason chain is proven link by link: TestEnsureWorkerAssigned (SUSPENDING and REVERTING carry a reason, CRASHED and DELETING do not), TestFromErrorReason, the interceptor row, and the router rows (a plain FailedPrecondition makes one attempt and is not shed by a full lot; a reasoned one parks until the budget). The e2e TestRoutedRequestToCrashedActorFailsFast crashes an actor by deleting its worker pod and routes one request: 503 in about 14 ms on Envoy, where the time bound is the proof that nothing parked; on main the same request takes 5.0 s. Each new test fails with its production hunk removed. The parking suite, go test -race -count=3 -shuffle=on on the touched trees and hack/verify-all.sh pass.

AI use: AI-assisted under my direction; I reviewed and tested the change.

Tests pass
Docs updated

Bowei Du (bowei) · 2026-10-29T05:25:36Z

 }

-// loadActorForResume fetches the current actor record and its template.
+// loadActorForResume fetches the current actor record and its template. It


This is AI comment, can you remove this diff?

Bowei Du (bowei) · 2026-10-29T05:29:24Z

 	// for. Short-circuit immediately to avoid an unnecessary template read on
-	// the hot resume path.
-	if actor.GetStatus().GetState() == ateapi.ActorState_ACTOR_STATE_RUNNING {
+	// the hot resume path. CRASHED and DELETING are refused here as well, before


Delete this AI written comment.

Put the behavior comments inline in the switch

switch actor.GetStatus().GetState() { // If the actor is already running, there is no pending restore to prepare // for. Short-circuit immediately to avoid an unnecessary template read on // the hot resume path. case ateapi.ActorState_ACTOR_STATE_RUNNING: return actor, nil, nil // CRASHED Actors should also short-circuit as this state is permanent. case ateapi.ActorState_ACTOR_STATE_CRASHED: return nil, nil, apierror.FailedPreconditionWithReason(apierror.ReasonActorCrashed, "actor %s is CRASHED; revert it with RevertActor", actorRef) // DELETING Actors should also short-circuit as this state is permanent. case ateapi.ActorState_ACTOR_STATE_DELETING: return nil, nil, apierror.FailedPreconditionWithReason(apierror.ReasonActorDeleting, "actor %s is being deleted", actorRef) }

Bowei Du (bowei) · 2026-10-29T05:29:37Z

 // ensureVolumesCreated provisions any actor volumes that have no storage
 // volume ID yet, persisting the resulting volume state (even when creation
 // partially failed, so progress is not lost) and returning the stored copy.
-// An actor being deleted gets no new volumes.


Why is this removed

Bowei Du (bowei) · 2026-10-29T05:29:53Z

 		markSkipped(ctx, "no volumes awaiting creation")
 		return actor, nil
 	}
-	if st := actor.GetStatus().GetState(); st == ateapi.ActorState_ACTOR_STATE_DELETING {


Why is this removed?

Bowei Du (bowei) · 2026-10-29T05:30:07Z

 		wantCreates int
 	}{
 		{"suspended actor gets its volume", ateapi.ActorState_ACTOR_STATE_SUSPENDED, codes.OK, 1},
-		{"deleting actor gets no volume", ateapi.ActorState_ACTOR_STATE_DELETING, codes.FailedPrecondition, 0},


Why is this test changed?

Bowei Du (bowei) · 2026-10-29T05:34:13Z

 		// Preserve the gRPC description for FailedPrecondition and Aborted:
 		// they carry actionable client-facing context (e.g. "another operation is
 		// in progress for this actor") and are not security-sensitive.
+		if terminalResumeReason(err) {


Does it make sense to flip this and only allow list parking for a narrow set of errors that can result in Actors being able to be successful after parking?

Bowei Du (bowei) · 2026-10-29T05:34:40Z

+		if terminalResumeReason(err) {
+			// A retry cannot succeed until an operator reverts the actor or the
+			// delete finishes, so 409 rather than the 503 clients retry.
+			re.StatusCode = int(envoy_type.StatusCode_Conflict)


not sure about the error code here. 4xx means the request is invalid which is not the case here.

Will switch back to 503 for now, so the behavior change is only that the request fails at once instead of parking.

On the code itself, two data points for a later decision: google/rpc/code.proto maps FAILED_PRECONDITION to HTTP 400 and ABORTED to 409, so the canonical gRPC mapping is a 4xx; and RFC 9110 defines 409 as a conflict with the current state of the resource that the client can resolve and resubmit, which fits CRASHED (RevertActor, then retry).

Bowei Du (bowei) · 2026-10-29T05:35:34Z

-// fails with: FailedPrecondition once the crash is committed, since retrying
-// the operation is pointless, or crashActor's own error if the crash did not
-// commit, so the caller retries and the retry crashes the actor again.
+// fails with. Once the crash is committed that is FailedPrecondition with the


why did this comment get changed?

Bowei Du (bowei) · 2026-10-29T05:37:39Z

 	"go.ate.dev/substrate/internal/resources"
 )

+// terminalPrecondition builds the FailedPrecondition ateapi returns for an


I would prefer putting these as closures in test func if the symbol is not reused elsewhere:

func TestActorNotFound(...) { ... makeTerminalPreconditionErr := func(t*testing.T, ...) {} makeOtherPreconditionErr := func... }

Bowei Du (bowei) · 2026-10-29T05:38:44Z

-// FailedPrecondition stays retryable too: it no longer carries saturation, but
-// it does cover states a concurrent operation can move the actor out of.
+// FailedPrecondition is retried for states a concurrent operation moves the
+// actor out of (SUSPENDING, REVERTING), but not when ateapi marks it with an


I would rather have the code ONLY retry on SUSPENDING, REVERTING and don't retry on anything else rather than only fail on crash. Can we do this and does it simplify the code?

Bowei Du (bowei) · 2026-10-29T05:39:41Z

Can we gate return to only conditions that are retryable instead of the current approach.

In order to tell the actor state that clears on its own, SUSPENDING or REVERTING, from those that do not, such as CRASHED, introduce a Reason to the apiError, ACTOR_SUSPENDING or ACTOR_REVERTING. FromError sends it as an ErrorInfo detail under the ate.dev domain. First of four commits that make the router fail fast on a crashed actor. Signed-off-by: Yuan Gao <ypgao@google.com>

Before, ensureWorkerAssigned refuses every state but SUSPENDED and RESUMING with one plain FailedPrecondition. The SUSPENDING and REVERTING arms now attach a reason through assignWorkerPrerequisiteNotMetError so that they differentiate from other states like CRASHED. Signed-off-by: Yuan Gao <ypgao@google.com>

retryable now parks a FailedPrecondition only when it carries the ACTOR_SUSPENDING or ACTOR_REVERTING reason; every other FailedPrecondition fails at once with the same 503 and description as before. After the commit, note that besides a CRASHED, DELETING or UNSPECIFIED actor, this stops parking on a missing template, a volume without a storage ID, a worker without a node name and a failed DetachActor. Signed-off-by: Yuan Gao <ypgao@google.com>

The new demo test crashes an actor by deleting its worker pod, routes one request and expects 503 with ACTOR_STATE_CRASHED in the body. On Envoy, the answer must arrive in under 2 s to prove nothing parked. AgentGateway resumes through ate-api-server itself and still parks, so the dataplane's CrashedActorFailsFast flag skips the time bound there. Signed-off-by: Yuan Gao <ypgao@google.com>

Yuan Gao (ygao-g) added kind/bug Something isn't working / bugfixes area/network area/api User-facing API changes labels Oct 28, 2026

Yuan Gao (ygao-g) assigned Yuan Gao (ygao-g) and Bowei Du (bowei) Oct 28, 2026

Yuan Gao (ygao-g) requested review from Lior Lieberman (LiorLieberman) and Bowei Du (bowei) October 28, 2026 20:19

Yuan Gao (ygao-g) force-pushed the fix-3269-crashed-resume-fail-fast branch 2 times, most recently from 315957b to 552492b Compare October 29, 2026 00:00

Bowei Du (bowei) reviewed Oct 29, 2026

View reviewed changes

Yuan Gao (ygao-g) force-pushed the fix-3269-crashed-resume-fail-fast branch from 552492b to 0e2a813 Compare October 29, 2026 16:01

Yuan Gao (ygao-g) force-pushed the fix-3269-crashed-resume-fail-fast branch 3 times, most recently from 47d3a0f to ca2b111 Compare October 29, 2026 16:50

Yuan Gao (ygao-g) added 3 commits October 29, 2026 09:50

Yuan Gao (ygao-g) force-pushed the fix-3269-crashed-resume-fail-fast branch 3 times, most recently from 0e2a813 to 5c5aba3 Compare October 29, 2026 16:56

Navigation Menu

Repository navigation

ateapi,atenet: fail fast when resuming a CRASHED or DELETING actor - #3402

ateapi,atenet: fail fast when resuming a CRASHED or DELETING actor#3402
Yuan Gao (ygao-g) wants to merge 4 commits into
agent-substrate:mainfrom
ygao-g:fix-3269-crashed-resume-fail-fast

Yuan Gao (ygao-g) commented Oct 28, 2026 •

edited

Loading

Uh oh!

Bowei Du (bowei) Oct 29, 2026

Uh oh!

Bowei Du (bowei) Oct 29, 2026

Uh oh!

Bowei Du (bowei) Oct 29, 2026

Uh oh!

Bowei Du (bowei) Oct 29, 2026

Uh oh!

Bowei Du (bowei) Oct 29, 2026

Uh oh!

Bowei Du (bowei) Oct 29, 2026

Uh oh!

Bowei Du (bowei) Oct 29, 2026

Uh oh!

Yuan Gao (ygao-g) Oct 29, 2026 •

edited

Loading

Uh oh!

Bowei Du (bowei) Oct 29, 2026

Uh oh!

Bowei Du (bowei) Oct 29, 2026

Uh oh!

Bowei Du (bowei) Oct 29, 2026

Uh oh!

Bowei Du (bowei) commented Oct 29, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Yuan Gao (ygao-g) commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yuan Gao (ygao-g) Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Bowei Du (bowei) commented Oct 29, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Yuan Gao (ygao-g) commented Oct 28, 2026 •

edited

Loading

Yuan Gao (ygao-g) Oct 29, 2026 •

edited

Loading