Skip to content

[docs] Document Paimon dynamic read options - #4021

Open
Gabriel39 wants to merge 1 commit into
apache:masterfrom
Gabriel39:agent/document-paimon-reader-options
Open

[docs] Document Paimon dynamic read options#4021
Gabriel39 wants to merge 1 commit into
apache:masterfrom
Gabriel39:agent/document-paimon-reader-options

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Document the dynamic Paimon batch-read options supported by Doris master.

  • Lists the seven supported options, their Paimon defaults, Doris validation bounds, and runtime effects.
  • Shows both Catalog defaults (paimon.table-option.*) and relation-scoped @options(...) overrides.
  • Documents the actual precedence: relation options, Catalog properties, physical table options, then Paimon defaults.
  • Explains validation and legacy-catalog compatibility behavior.
  • Clarifies that snapshot/startup selectors use dedicated query options and that Flink source, streaming, layout, write, and compaction options are not dynamic Doris reader settings.
  • Corrects the IOManager section so sort-spill-threshold is configured on the physical Paimon table.

Related code change: apache/doris#66247.

Version scope

This change updates only current English and Chinese documentation because the expanded allowlist and precedence behavior are being introduced on Doris master. Released 4.x, 3.x, and 2.1 documentation must continue to describe their released behavior.

Validation

  • yarn docs:links:changed
  • yarn docs:i18n-sync:changed
  • yarn docs:lint:changed
  • yarn build (stopped after changed MDX parsing; see note below)

The changed-file checks pass. Their reported warnings are pre-existing repository-wide Markdown, SEO, external-link, and version-sync notices; the intentional current-only version scope is explained above. The local full build parsed the changed English and Chinese MDX without an error, but was stopped after 32 minutes while it was still processing the repository's other locales and historical versions. The PR Build Check completed successfully and is the authoritative full-site validation.

Versions

  • dev
  • 4.x
  • 3.x
  • 2.1 or older (not covered by version/language sync gate)

Languages

  • Chinese
  • English
  • Japanese candidate translation needed

Docs Checklist

  • Checked by AI
  • Test Cases Built
  • Updated required version and language counterparts, or explained why not
  • If only one language changed, confirmed whether source/translation counterparts need sync

Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Jul 30, 2026
@Gabriel39
[fix](paimon) Expand dynamic reader option support
1ffd61f 
### What problem does this PR solve?

Issue Number: None

Related PR: apache#66247

Problem Summary: Doris allowed only `read.batch-size` and
`file-reader-async-threshold` through Paimon Catalog properties and relation
`@options`, so safe batch-read planning controls such as split sizing, file-index
reads, and manifest planning were rejected. The missing support also made it
unclear which official Paimon option names Doris consumes; these names must remain
the Paimon names and do not all use a `scan.` prefix.

Expand the allowlist to the seven batch-read options consumed by Doris, validate
their types and safety bounds, and keep manifest/partition-order settings on the
effective metadata projection path. Catalog defaults and relation overrides now
accept the same reader tuning while context selectors, Flink enumerator settings,
streaming, layout, and write options remain excluded.

The previous BE unit-test commit also omitted the new `JniColumn.output_type`
member from two designated initializers. Clang treats that omission as an error
under the BE UT warning policy, so initialize the field explicitly in both tests.

### Release note

Paimon Catalog properties and relation `@options` now support
`read.batch-size`, `file-reader-async-threshold`, `file-index.read.enabled`,
`source.split.target-size`, `source.split.open-file-cost`,
`scan.manifest.parallelism`, and `scan.plan-sort-partition`.

### Check List (For Author)

- Test:
    - Unit and P0 regression coverage added.
    - Isolated Paimon 1.3.1 Java compile/runtime validation passed.
    - Regression framework compilation passed (466 Groovy files).
    - Doris clang-format 16 check passed.
    - Full FE/BE unit execution was not run locally because the installed
      third-party toolchain, including `protoc`, is absent; CI is requested.
- Behavior changed: Yes. Doris accepts five additional safe Paimon batch-read
  options from Catalog defaults and relation-scoped overrides.
- Does this need documentation: Yes.
  apache/doris-website#4021
@Gabriel39 Gabriel39 mentioned this pull request Jul 30, 2026
Open
@Gabriel39
Gabriel39 marked this pull request as ready for review July 30, 2026 07:43
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Jul 30, 2026
@Gabriel39
[fix](paimon) Expand dynamic reader option support
fb2ea10 
Issue Number: None

Related PR: apache#66247

Problem Summary: Doris allowed only `read.batch-size` and
`file-reader-async-threshold` through Paimon Catalog properties and relation
`@options`, so safe batch-read planning controls such as split sizing, file-index
reads, and manifest planning were rejected. The missing support also made it
unclear which official Paimon option names Doris consumes; these names must remain
the Paimon names and do not all use a `scan.` prefix.

Expand the allowlist to the seven batch-read options consumed by Doris, validate
their types and safety bounds, and keep manifest/partition-order settings on the
effective metadata projection path. Catalog defaults and relation overrides now
accept the same reader tuning while context selectors, Flink enumerator settings,
streaming, layout, and write options remain excluded.

The previous BE unit-test commit also omitted the new `JniColumn.output_type`
member from two designated initializers. Clang treats that omission as an error
under the BE UT warning policy, so initialize the field explicitly in both tests.

Paimon Catalog properties and relation `@options` now support
`read.batch-size`, `file-reader-async-threshold`, `file-index.read.enabled`,
`source.split.target-size`, `source.split.open-file-cost`,
`scan.manifest.parallelism`, and `scan.plan-sort-partition`.

- Test:
    - Unit and P0 regression coverage added.
    - Isolated Paimon 1.3.1 Java compile/runtime validation passed.
    - Regression framework compilation passed (466 Groovy files).
    - Doris clang-format 16 check passed.
    - Full FE/BE unit execution was not run locally because the installed
      third-party toolchain, including `protoc`, is absent; CI is requested.
- Behavior changed: Yes. Doris accepts five additional safe Paimon batch-read
  options from Catalog defaults and relation-scoped overrides.
- Does this need documentation: Yes.
  apache/doris-website#4021
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Jul 31, 2026
@Gabriel39
[fix](paimon) Expand dynamic reader option support
b473593 
### What problem does this PR solve?

Issue Number: None

Related PR: apache#66247

Problem Summary: Doris allowed only `read.batch-size` and
`file-reader-async-threshold` through Paimon Catalog properties and relation
`@options`, so safe batch-read planning controls such as split sizing, file-index
reads, and manifest planning were rejected. The missing support also made it
unclear which official Paimon option names Doris consumes; these names must remain
the Paimon names and do not all use a `scan.` prefix.

Expand the allowlist to the seven batch-read options consumed by Doris, validate
their types and safety bounds, and keep manifest/partition-order settings on the
effective metadata projection path. Catalog defaults and relation overrides now
accept the same reader tuning while context selectors, Flink enumerator settings,
streaming, layout, and write options remain excluded.

The previous BE unit-test commit also omitted the new `JniColumn.output_type`
member from two designated initializers. Clang treats that omission as an error
under the BE UT warning policy, so initialize the field explicitly in both tests.

### Release note

Paimon Catalog properties and relation `@options` now support
`read.batch-size`, `file-reader-async-threshold`, `file-index.read.enabled`,
`source.split.target-size`, `source.split.open-file-cost`,
`scan.manifest.parallelism`, and `scan.plan-sort-partition`.

### Check List (For Author)

- Test:
    - Unit and P0 regression coverage added.
    - Isolated Paimon 1.3.1 Java compile/runtime validation passed.
    - Regression framework compilation passed (466 Groovy files).
    - Doris clang-format 16 check passed.
    - Full FE/BE unit execution was not run locally because the installed
      third-party toolchain, including `protoc`, is absent; CI is requested.
- Behavior changed: Yes. Doris accepts five additional safe Paimon batch-read
  options from Catalog defaults and relation-scoped overrides.
- Does this need documentation: Yes.
  apache/doris-website#4021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Gabriel39