JIT: Support general inlining in runtime async - #131538
Draft
jakobbotsch wants to merge 46 commits into
Draft
Conversation
We will need this to enable inlining.
Invariant nodes and LCL_VARs do not need to be lifted across async calls.
Consolidate the duplicated non-null assert in FinishContextHandlingAndSuspensionWithHelper into a single assert covering all three well-known args. Update the stale comment in InsertFinishContextHandlingCall that still described the resumed placeholder as being replaced by a 'continuation != null' computation; it is now replaced by the already-computed resumed value. No functional change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d86cad55-abc0-42d6-8369-bc6bc619326f
Value numbering can fold the AsyncResumedUse argument to a constant when the resumed indicator is provably zero at the call, for example when two awaits sit on mutually exclusive paths. IsReusableSuspension only accepted locals, so it bailed out with 'argument is too complex' and gave up on merging the suspension points. That produced a duplicated CORINFO_HELP_ALLOC_CONTINUATION sequence, an extra resumption block and an extra readonly data slot. In smoke_tests.nativeaot this showed up as +59 bytes (17.15%) on Generics+TestAsyncGVMScenarios:AsyncGvm1[System.__Canon] and +57 bytes (16.52%) on RunAsync. Accept invariant nodes as well, matching the predicate already used for invariantResumed in CreateResumptionsAndSuspensions. Invariant nodes are constants, LCL_ADDR or FTN_ADDR, so comparing them and removing them from the call is safe. Both methods now improve by 15 bytes relative to the base instead of regressing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d86cad55-abc0-42d6-8369-bc6bc619326f
…ning # Conflicts: # src/coreclr/inc/jiteeversionguid.h # src/coreclr/jit/async.cpp # src/coreclr/jit/async.h # src/coreclr/tools/superpmi/superpmi-shared/methodcontext.h # src/tests/async/pgo/pgo.csproj
StoreAsyncAwaiter removed the awaiter node from the call block after it had already been spliced into the suspension block's LIR range, corrupting the range and tripping 'prev == m_lastNode'. Remove the node from the call block before building the store instead, and remove the FIELD_LIST node itself in the multi-field case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
Folds in dotnet#127800. Renames GT_CONTINUATION_FIELD_OFFSET to GT_CONTINUATION_MEMBER_OFFSET to match the node actually implemented. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
…ontext to the JIT General runtime async inlining needs to check, when an inlined callee logically returns to its caller after having been resumed, whether it is already running in the continuation context the caller's continuation captured, and to restore that continuation's ExecutionContext. IsOnRightContext mirrors the 'can inline' conditions in RuntimeAsyncTaskContinuation.QueueIfNecessary; the two must be kept in sync. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
Suspends and resumes in a specified continuation context, rather than one captured at the suspension point. General runtime async inlining uses this when an inlined callee logically returns to its caller after having been resumed and IsOnRightContext reports the current context does not match the one the caller's continuation captured. It suspends on an already completed task so that the dispatcher re-dispatches immediately. That dispatch runs with canInline: false, so it always posts or schedules onto the requested context instead of running inline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
Extends ContinuationMember with slots for the ExecutionContext, continuation context and continuation flags that an inlined async frame captures when it logically returns to its caller. These are keyed by inline depth rather than by individual inline frame. Two frames at the same depth can share storage because their live ranges cannot overlap: the value is written at a frame's first suspension and consumed by that same frame's post-inline IR, and one frame at a given depth must be left before another can be entered. This bounds the extra continuation members by the maximum inline depth instead of by the number of inlined async frames. No functional change yet; nothing requests these members. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
…exts Adds the JitAsyncInlining config knob, off by default, which lifts the CALLEE_AWAIT restriction and allows inlining async callees that may suspend. When an await in an inlinee may suspend it no longer inherits context handling from the inlining call. Instead it is treated exactly like an await in a non-inlined method and picks up the inlinee's own resumed/ExecutionContext/ SynchronizationContext locals, which the inlinee's own SaveAsyncContexts phase already creates. The cheap paths (async versions of synchronous methods and tail awaits) keep inheriting from the caller and are unchanged. With the knob off behavior is unchanged: the restructured gate reports the same CALLEE_AWAIT observation, and inlinees do not get their own context args. The knob is not yet functional when enabled; the post-inline IR and the nested capture chain at suspensions are still missing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
…al async inlining SaveAsyncContexts now records whether the body contains any async call. For an inlinee without one the resumed indicator is provably always false, so the post-inline async frame IR can be skipped entirely, keeping the cheap shape for what is by far the common case. Also bails out of general async inlining in OSR methods. An OSR method is entered with the tier0 method's continuation, whose layout differs from the one the OSR method computes, so an inlined frame's members cannot be read from it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
…Context RestoreExecutionContext takes the Thread whose contexts were captured on method entry. The restore an inlined async frame needs runs only after a resumption, so it must target the thread we were resumed on instead. Adds RestoreInlinedFrameExecutionContext, which reads the current thread itself, and points the JIT-EE handle at it. This also keeps the JIT side to a single argument. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
SaveAsyncContexts runs for inlinees too, and JIT_FLAG_OSR is not cleared for an inlinee, so opts.IsOSR() is true while importing an inlinee of an OSR method. The OSR-specific handling was therefore applied to inlinees, where all three parts of it are wrong: - the resumed indicator was initialized from the continuation argument, but an inlinee has not resumed just because the OSR method was entered via a resumption - CaptureContexts was skipped, leaving the inlinee's context locals unassigned for its own RestoreContexts - the inlinee's fresh temps were marked lvIsOSRLocal, though they have no home in the tier0 frame An inlinee inside an OSR method starts a fresh logical frame, so it needs the same treatment as in a non-OSR method. With this, general async inlining works in OSR methods: an inlinee's resumed indicator can only become true at a resumption point belonging to that inlinee's own async calls, which are resumption points of the OSR method's continuation layout, so the continuation is never the tier0 one where it is read. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
…y region Suspending inside a protected region additionally requires catching the exception and rethrowing it around the post-inline handling, per the design document. That is not implemented, so bail out for now. Inlinees with EH are only inlined when compInlineMethodsWithEH is enabled, so this is reachable but not the common case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
…caller
When an inlined async callee is resumed inside its own body, the async
infrastructure only restores the context of the suspension point itself. The
work it would otherwise have done when the callee's frame returned to its caller
now has to be performed by the JIT:
if (resumed_F)
{
RestoreInlinedFrameExecutionContext(continuation.ExecutionContextFor<caller>);
if (!IsOnRightContext(continuation.ContinuationContextFor<caller>,
continuation.FlagsFor<caller>))
{
await SwitchContext(...);
}
resumed_caller = true;
}
This is emitted as a real control flow diamond at the join point all of the
inlinee's returns converge on, alongside the other post-inline IR. IsOnRightContext
stays visible in the IR and is an inline candidate, so it needs to be a statement
root with its value consumed through a GT_RET_EXPR.
Two bugs this exposed are fixed here as well:
- Continuation members were registered on whichever Compiler instance was
importing, so an awaiter inside an inlinee got an index into the inlinee's
vector while its offset node was resolved against the root's layout. Members
describe the root method's continuation, so they are now always registered
there.
- GT_CONTINUATION_MEMBER_OFFSET was always bashed to a TYP_INT constant. Offsets
used in address arithmetic are TYP_I_IMPL, so the node's type is now preserved.
Also records each inlined frame's async locals on its InlineContext, since the
inlinee's Compiler is discarded once it has been spliced in.
The nested capture chain at suspensions is still missing, so enabling
JitAsyncInlining is not yet correct.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
When an async call inside an inlined frame suspends, every enclosing frame that has not yet resumed needs its contexts captured into the continuation and restored onto the thread, as if its physical frame had returned. The IR for that is created by the async transformation, long after the optimizer has run, so those values have to be kept live and GC reported at the call until then. That is exactly what the existing per-frame context args do, so the enclosing frames just add more of them: one set of AsyncResumedUse/AsyncExecutionContext/ AsyncSynchronizationContext per frame, appended outward when the inlinee is spliced in. These are pseudo-args, so duplicates take no registers and are expanded out later. The innermost frame stays first. Only uses are needed for the enclosing frames. AsyncResumedDef stays innermost-only, since the caller's indicator is set by the post-inline IR rather than by a resumption point. Adjusts the two places that assumed these args were unique: IsReusableSuspension now requires all of them to agree, and HandleReusedSuspension removes all of them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
Completes general runtime async inlining. When an async call inside an inlined frame suspends, every enclosing frame that has not yet resumed must capture the contexts it would hand to its caller and restore its caller's contexts onto the thread, as if its physical frame had returned. A frame having resumed implies its caller has too, so the frames can be walked outward as a straight line rather than as the nested conditionals of the design document: once one frame has resumed, the helpers for it and every frame outside it no-op. Each suspension jumps to a tail block that does this, shared by all suspensions in the same frame. Adds AsyncHelpers.CaptureInlinedFrameTransition for the capture, which no-ops when the frame has already resumed. The chain of frames is recorded on AsyncCallInfo rather than being recovered from the call's context args, since optimizations may rewrite the arg nodes; the args remain purely to keep the values live and GC reported until here. Also fixes impInheritAsyncContextsFromInliner, which propagated only the innermost frame's contexts to an inlined callee's async calls. An async call carrying a frame chain can itself be an inline candidate, and dropping the rest of the chain silently lost the handling for every frame outside the immediate one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
The worked examples from the design document. Inlining removes a call site and a callee, and with them the context handling the async infrastructure would otherwise perform at each frame boundary, so these pin down the observable results: each frame resumes on the synchronization context its own continuation captured, and sees the AsyncLocal values from the ExecutionContext it captured. The callees are marked AggressiveInlining because the profitability heuristic otherwise rejects them, and the test runs with tiering disabled, since at tier 0 nothing is inlined and the tests would not exercise the feature at all. Verified to exercise a three frame chain, and to fail if the frame transition IR is removed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
Comment on lines
+2640
to
+2649
| // TODO-Async: when this call suspends, the frames enclosing the caller need their | ||
| // context handling run too. Those come from the enclosing frame chain, which is added | ||
| // to async calls as additional pseudo-args. | ||
| call->gtArgs.PushFront(this, NewCallArg::Primitive(gtNewLclVarNode(syncCtx, TYP_REF)) | ||
| .WellKnown(WellKnownArg::AsyncSynchronizationContext)); | ||
| call->gtArgs.PushFront(this, NewCallArg::Primitive(gtNewLclVarNode(execCtx, TYP_REF)) | ||
| .WellKnown(WellKnownArg::AsyncExecutionContext)); | ||
| call->gtArgs.PushFront(this, NewCallArg::Primitive(gtNewLclVarNode(resumed, TYP_INT)) | ||
| .WellKnown(WellKnownArg::AsyncResumedUse)); | ||
| } |
This was referenced Jul 29, 2026
# Conflicts: # src/coreclr/jit/async.cpp # src/coreclr/jit/async.h # src/coreclr/jit/compiler.h # src/coreclr/jit/importercalls.cpp
The context pseudo-args on an async call are the source of truth for what the suspension stores. The whole optimizer runs between adding them and the async transformation, so those nodes can be rewritten: 'resumed' indicators in particular get const folded, and contexts can be copy propagated to other locals. Reconstructing the values from the locals recorded at inline time ignored all of that and stored whatever the original locals happened to hold. Take the actual nodes off the call instead, spilling to a temp only when a node is neither invariant nor a local, exactly as the existing shared finish-context handling does when it needs to reference an arg from another block. The nodes also become the identity used for sharing a tail, so suspensions share one only when they agree on every value. That is now observable: with two awaits in the same inlined frame, the first has its resumed indicator folded to the constant zero while the second still reads the local, and they correctly get separate tails. Sharing them would have stored the wrong value at one of them. This removes the need for the side table, so AsyncCallInfo::InlineFrameLocals and the AsyncFrameLocals type go away. InlineContext keeps its frame locals, which are still the right source when creating the args during inlining. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
Each suspension now gets its own tail. Sharing required the suspensions to agree on every value the tail stores, which in practice they do not: the first await in a frame typically has its resumed indicator folded to a constant while later ones still read the local, so the tails were never actually shared. Left as a TODO; it is an optimization that saves cold code and can come back separately. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
JulieLeeMSFT
added
the
Priority:1
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 84 out of 84 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
src/coreclr/jit/fginline.cpp:2853
SwitchContextis an async helper that always suspends. In the resumption path,resumedCalleris only set indoneBlock, which executes after theSwitchContextcall. IfSwitchContextsuspends (the expected behavior), suspension processing will observeresumedCaller == 0, which can cause the switch-call suspension to be treated like a first-time (non-resumed) suspension for the caller frame.
// switchBlock: get back onto the continuation context the caller's continuation
// captured. This is an await, so it may suspend.
{
GenTreeCall* const switchCall = gtNewUserCallNode(asyncInfo->switchContextMethHnd, TYP_VOID);
switchCall->gtArgs.PushFront(this,
The post-inline IR for an inlined async frame returning to its caller was a full diamond: an ExecutionContext restore, an IsOnRightContext call with its own conditional, and a SwitchContext await, spread over four blocks and relying on inlining the two helpers back in. Only the resumed check is worth having as IR. It is what guards the synchronous path, which is the one to optimize for; anything past it has already suspended and resumed at least once, so a call is cheap there. Fold IsOnRightContext and SwitchContext into a single AsyncHelpers.RestoreInlinedFrameContinuationContext await that does the check internally, leaving two blocks and no inline candidates. The ExecutionContext restore stays a separate non-async call. It cannot move into the same helper: a runtime async method restores the contexts it captured on entry when it returns, which would undo it on the non-suspending path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
RestoreInlinedFrameExecutionContext and RestoreInlinedFrameContinuationContext become one RestoreInlinedFrameContexts await. The ExecutionContext restore can live in the async helper after all: it is a manually marked async method rather than the async variant of a task returning one, so CORINFO_ASYNC_SAVE_CONTEXTS is not set for it and there is no save/restore around its body to undo the restore on the way out. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
Split the suspension out into a tail awaited SwitchToContinuationContext so that what is left is just the ExecutionContext restore and the continuation context check. Take the thread from the cached one in the await state rather than a second TLS lookup, and inline IsOnRightContext into its only caller. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
The check for whether we are already on the right continuation context is what guards the suspension, so it is worth having in the caller. Mark the call so the inliner can pull it in, and force it with AggressiveInlining since the helper is over the IL size budget. This also needs the call to carry an AsyncResumedDef like every other async call. Without it, inlining a body containing an async call trips the use/def parity assert in impInheritAsyncContextsFromInliner, which propagates both to the inlinee's own async calls. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
Whether to inline it should be the JIT's call based on PGO, not forced here. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
Those args model what a frame captures and restores when a suspension unwinds through it. None of that applies here: we only run at all because the frame we are returning out of was resumed, so every frame in the chain already has a continuation holding what it captured when it first suspended, and a suspension here goes straight back to the dispatcher, which restores the thread's contexts itself. This also drops the AsyncResumedDef added along with the inline candidate marking; with no args at all the use/def parity in impInheritAsyncContextsFromInliner holds trivially and the inlinee's own async calls correctly get no context args either. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
The comparisons are reference comparisons either way, so the castclass just costs IL. That matters here: it brings the method from 130 to 120 IL bytes, under the 128 byte inlining limit, so the JIT can now actually inline it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
The chain was built from a side table of context locals recorded on InlineContext. That table is populated when a frame is spliced into its caller, which happens after the frame's own inlinees have already been processed, so an inner frame never saw the frames between it and the root and silently dropped their transitions. Copy the context args off the call being inlined instead. They describe exactly the chain the inlinee ends up in, and they compose: a frame picks up its caller's chain when it is spliced in, and every call in it, including ones that came from its own inlinees, is extended again when that caller is inlined. The first set on the call belongs to the frame it suspends in and is consumed by that frame's own handling, so it is skipped unless it is all there is. This also drops the assumption that the root method is always a logical async frame. It is not when it does no context handling itself, for example when it is the async version of a synchronous method; there is then nothing to hand back and no transitions to run. InlineContext keeps only a bit saying whether the frame does context handling, set while the frame is compiled rather than when it is spliced in, so that depths are right for its own inlinees. Fixes an assert compiling System.IO.ConnectedStreams.BidirectionalStreamBufferStream.ReadAsync. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed3e552-b907-4815-9683-56a7fed2cdf3
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 84 out of 84 changed files in this pull request and generated no new comments.
Suppressed comments (4)
src/coreclr/jit/jitconfigvalues.h:627
- PR description suggests this may need to ship disabled-by-default during validation, but the new config switch enables general runtime async inlining by default (value
1). If this is meant to be an opt-in feature initially, consider defaulting to0and enabling viaDOTNET_JitAsyncInlining=1during testing/validation.
// Enable general inlining of runtime async calls, i.e. inlining of async
// callees that may suspend. When zero, only the restricted cases are inlined:
// callees without any awaits, async versions of synchronous methods, and tail
// awaits.
RELEASE_CONFIG_INTEGER(JitAsyncInlining, "JitAsyncInlining", 1)
src/tests/async/custom-struct-awaiters/custom-struct-awaiters.cs:19
- Using
Task.Wait()in a test entrypoint tends to wrap failures inAggregateExceptionand can introduce deadlock hazards if aSynchronizationContextever gets involved. PreferGetAwaiter().GetResult()for synchronous blocking in tests.
src/tests/async/inlined-frame-contexts/inlined-frame-contexts.cs:52 ThreadPool.QueueUserWorkItemflowsExecutionContext, which can mask bugs in runtime-asyncExecutionContextcapture/restore (the callback may already run under the captured context). Suppressing flow here makes the test more directly validate the JIT/runtime context handling it is intended to pin down.
src/tests/async/osr-inlined-contexts/osr-inlined-contexts.cs:8BumpAsyncis markedasyncbut contains noawait, which triggers CS1998. Several existing runtime-async tests suppress this warning explicitly; doing the same here avoids build breaks when warnings are treated as errors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Built on #128152 and #131342.
Currently just validating. I am hoping to get this in .NET 11, perhaps disabled by default, depending on how validation goes.
Fix #127865