Skip to content

馃挜 Fix unbounded timeout failure chain in local activity - #3006

Merged
Quinn-With-Two-Ns merged 2 commits into
temporalio:mainfrom
Quinn-With-Two-Ns:issue-3005
Aug 13, 2026
Merged

馃挜 Fix unbounded timeout failure chain in local activity #3006
Quinn-With-Two-Ns merged 2 commits into
temporalio:mainfrom
Quinn-With-Two-Ns:issue-3005

Conversation

@Quinn-With-Two-Ns

@Quinn-With-Two-Ns Quinn-With-Two-Ns commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

What was changed

Local activities retried across the local-retry threshold were carrying the entire ActivityFailure wrapper into each new attempt. Repeated timeouts therefore created an unbounded nested failure chain, cuasing oversized histories or serialization failures.

Why?

Was causing the failure to grow very large and even fail the workflow it got so large.

Checklist

  1. Closes Local activity retries build a Failure chain the SDK cannot deserialize聽#3005

  2. How was this tested:

  1. Any docs updates needed?

Note

Medium Risk
Touches core local-activity retry and failure propagation in the workflow sync path; behavior change is narrow but could affect failure shape seen by workflows and history markers.

Overview
Fixes local activity retries that cross the local retry threshold so they no longer grow an unbounded nested failure chain on repeated failures (e.g. timeouts).

When scheduling the next attempt after backoff, SyncWorkflowContext now passes laException.getFailure().getCause() as previousExecutionFailure instead of the full wrapper failure. That keeps only the underlying attempt failure (e.g. TimeoutFailure) linked across attempts, avoiding oversized workflow history and serialization blowups.

A new integration test repeatedTimeoutsDoNotBuildAnUnboundedFailureChain asserts that after max attempts the surfaced ActivityFailure has at most two chained TimeoutFailure nodes, with the oldest having no further cause.

Reviewed by Cursor Bugbot for commit df1786e. Bugbot is set up for automated code reviews on this repo. Configure here.

@Quinn-With-Two-Ns
Quinn-With-Two-Ns requested a review from a team as a code owner August 13, 2026 05:39
Comment on lines +103 to +104
@Test(timeout = 120_000)
public void repeatedTimeoutsDoNotBuildAnUnboundedFailureChain() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to redesign this test in a way that requires shorter execution time? Either by reducing timeouts/retry intervals or by using time skipping?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah we don't need to do 53 I was just copying what the customer reported

@Quinn-With-Two-Ns
Quinn-With-Two-Ns merged commit ade44bc into temporalio:main Aug 13, 2026
25 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Local activity retries build a Failure chain the SDK cannot deserialize

2 participants