Skip to content

[FLINK-40424][table][python] Support windowing TVFs in the DataFrame API - #29070

Open
Timm0 wants to merge 1 commit into
apache:masterfrom
Timm0:FLINK-40424
Open

[FLINK-40424][table][python] Support windowing TVFs in the DataFrame API#29070
Timm0 wants to merge 1 commit into
apache:masterfrom
Timm0:FLINK-40424

Conversation

@Timm0

@Timm0 Timm0 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

What is the purpose of the change

This change adds tumble, hop, cumulate, and session window methods to the DataFrame API and the internal Table API machinery they need. The window operation is built programmatically as a LogicalTableFunctionScan
rather than by generating SQL strings, so the DataFrame layer does not have to track Flink's SQL dialect. No new public Table API is introduced, since all added Table API types are @Internal.

Brief change log

  • Add tumble/hop/cumulate/session window methods to the PyFlink DataFrame API, with time-column resolution and day-time interval handling
  • Introduce internal WindowTableFunctionQueryOperation
  • Dispatch it through QueryOperationVisitor and QueryOperationDefaultVisitor and construct it via OperationTreeBuilder.windowTableFunction
  • Convert the operation to a LogicalTableFunctionScan programmatically in QueryOperationConverter
  • Add tests: api-java operation test, planner WindowTableFunctionRelNodeTest, QueryOperation serialization + semantic programs, and PyFlink unit + IT tests
  • Expose createTable/getOperationTreeBuilder on the Table API test steps and refactor _materialize into test_case_utils for stream test cases
  • Document the window methods in dataframe.rst

Verifying this change

  • Added api-java operation test
  • Added planner WindowTableFunctionRelNodeTest
  • Added QueryOperation serialization + semantic programs
  • Added PyFlink unit + IT tests

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): yes
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs flink-python/docs/reference/pyflink.dataframe/dataframe.rst + Python docstrings

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Opus 4.8 (1M context)

- Add `tumble`/`hop`/`cumulate`/`session` window methods to the PyFlink `DataFrame` API, with time-column resolution and day-time interval handling
- Introduce internal `WindowTableFunctionQueryOperation`
- Dispatch it through `QueryOperationVisitor` and `QueryOperationDefaultVisitor` and construct it via `OperationTreeBuilder.windowTableFunction`
- Convert the operation to a `LogicalTableFunctionScan` programmatically in `QueryOperationConverter`
- Add tests: api-java operation test, planner `WindowTableFunctionRelNodeTest`, `QueryOperation` serialization + semantic programs, and PyFlink unit + IT tests
- Expose `createTable`/`getOperationTreeBuilder` on the Table API test steps and refactor `_materialize` into `test_case_utils` for stream test cases
- Document the window methods in `dataframe.rst`
@Timm0
Timm0 marked this pull request as ready for review September 2, 2026 12:41
@flinkbot

flinkbot commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants