Is your feature request related to a problem or challenge?
External sort can keep already-consumed input batches alive after producing a partial output batch. That can hold memory longer than necessary for sorts that emit output in multiple chunks.
Describe the solution you'd like
After partial output, release input batches that are no longer referenced by pending row indices or live cursors. Batches that are still referenced should remain available for later output.
Describe alternatives you've considered
The current implementation keeps cleanup cheaper by dropping consumed batches only after all buffered row indices are drained. Keeping that behavior is simpler, but it can delay returning memory that is no longer needed.
The tradeoff in releasing batches earlier is that cleanup has to mark live batches and remap remaining row indices after partial output. I do not have an independent performance measurement for this first change yet.
Additional context
This request is limited to earlier release of unreferenced sort input batches. It does not propose sort output byte-targeting, spill admission changes, or user-visible API/configuration changes.
Is your feature request related to a problem or challenge?
External sort can keep already-consumed input batches alive after producing a partial output batch. That can hold memory longer than necessary for sorts that emit output in multiple chunks.
Describe the solution you'd like
After partial output, release input batches that are no longer referenced by pending row indices or live cursors. Batches that are still referenced should remain available for later output.
Describe alternatives you've considered
The current implementation keeps cleanup cheaper by dropping consumed batches only after all buffered row indices are drained. Keeping that behavior is simpler, but it can delay returning memory that is no longer needed.
The tradeoff in releasing batches earlier is that cleanup has to mark live batches and remap remaining row indices after partial output. I do not have an independent performance measurement for this first change yet.
Additional context
This request is limited to earlier release of unreferenced sort input batches. It does not propose sort output byte-targeting, spill admission changes, or user-visible API/configuration changes.