Skip to content

[core] Reuse projected entries in manifest run merge - #9241

Draft
leaves12138 wants to merge 6 commits into
apache:masterfrom
leaves12138:codex/manifest-radix-sort
Draft

[core] Reuse projected entries in manifest run merge#9241
leaves12138 wants to merge 6 commits into
apache:masterfrom
leaves12138:codex/manifest-radix-sort

Conversation

@leaves12138

@leaves12138 leaves12138 commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #9135 and #9213.

What changed

  • Replace the hand-written manifest field indexes and custom projected-row materialization in run merge with ProjectedManifestEntry.ENTRY_LAYOUT_PROJECTION.
  • Keep the shared PartitionDictionary unchanged and reuse it from the nested ManifestEntryRunMerge.SortPartitionDictionary, which only adds concurrent lookup and comparator ranks.
  • Collect minor-compaction DELETE state per discovery task and combine it afterward instead of synchronizing every DELETE entry.
  • Keep Avro file-name views borrowed for the current entry and copy only keys that outlive the row into reusable owned buffers.
  • Add MAX_SEQUENCE_NUMBER to the shared entry layout because it is a row-id manifest sorting key.

Why

The run-merge implementation duplicated schema layout knowledge with numeric field constants and manual GenericRow projection. That made it easy for the optimized sort path to drift from the shared manifest representation and made schema changes require edits in multiple places. This follow-up routes the sort path through the same projected manifest abstraction used by block-aware ordinary merging while keeping sort-specific concurrency and ordering out of the common partition dictionary.

Validation

  • Formatting and repository checks passed with ~/bin/check.
  • Targeted manifest tests: 136 tests, 0 failures, 1 existing skip.
  • Existing run-merge tests cover parallel discovery, partition ordering, secondary sorting keys, DELETE filtering, and external-sort equivalence.
  • dev2 large-table metadata benchmark without an Xmx limit:
Snapshot Before After Result
8795 5.26 s 4.96 s no regression
8816 19.03 s 16.33 s no regression

The output manifest counts and bytes remained unchanged in both benchmark cases.

@leaves12138
leaves12138 marked this pull request as ready for review August 15, 2026 18:29
@leaves12138
leaves12138 marked this pull request as draft August 15, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant