[Yufan Su (yufan-su)]

Fixes #3331

Why

When an actor's resume takes longer than the parking budget plus 5s, Envoy's ext_proc timeout ends the request with a 504 and no body. gRPC clients see Unavailable with an empty message, and the router logs the request as canceled by the client. With the default 5s budget, this happens to the first request for an actor restored from a snapshot of about 1 GiB.

What changed:

The router now stops waiting 1s before Envoy's timeout and answers with 503 actor <ref> is still resuming, retry. The resume keeps running, and later requests join it. Envoy's timeout stays as a backstop for when the router cannot answer. The egress handlers already cap their lookups under the ext_proc timeout the same way.

The error wraps context.DeadlineExceeded, so the route and parking metrics count it as timeout.

AI use: AI-assisted. I reviewed and tested it.

• Tests pass
• Docs updated

[Yufan Su (yufan-su)]

Bowei Du (@bowei)

We can customize the ext_proc timeout error with config like:

local_reply_config:
  mappers:
  - filter:
      extension_filter:
        name: envoy.access_loggers.extension_filters.cel
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.access_loggers.filters.cel.v3.ExpressionFilter
          expression: response.code_details == "ext_proc_error_per-message_timeout_exceeded"
    status_code: 503
    body:
      inline_string: "timed out waiting for the router, retry"

But this won't fix the router log, it will still say request for actor X canceled by client

I think we should keep the router-side timeout as the fix

[Bowei Du (bowei)]

does this need to be exported?

extProcTimeout (no Margin)

can we have this

// extProcTimeout is how much longer than the park budget Envoy waits for
// the router's reply
const extProcTimeout = 5 * time.Second
// envoyExtProcTimeout is the timeout on the Envoy side. This always needs
// to be less than extProcTimeout. If extproc times out first, Envoy ends
// up sending a response that is not very helpful to the end user (empty 504)
const envoyExtProcTimeout = extProcTimeout - 1*time.Second

[Yufan Su (yufan-su)]

It's exported because dataplane.go sets Envoy's ext_proc timeout from it (parkCfg.Budget + ingress.ExtProcTimeoutMargin), so the router's wait and Envoy's timeout come from one constant.

I kept "Margin" in the name because the value is added to the budget and isn't the whole timeout.

Also I called it routerExtProcTimeout instead of envoyExtProcTimeout because it is how long we wait in router for the flight to complete.