[flink] Fix TableNotExistException when listing partitions on changelog / binlog virtual tables by naivedogger · Pull Request #3641 · apache/fluss

naivedogger · 2026-07-13T08:31:33Z

Purpose

Linked issue: close #3640

When the Flink optimizer calls listPartitions() on a $changelog or $binlog virtual table, it throws TableNotExistException because the virtual table suffix is not stripped before querying the Fluss server.

getTable() correctly dispatches virtual table names to getVirtualChangelogTable()/getVirtualBinlogTable(), but tableExists(), listPartitions(), createPartition(), and dropPartition() all pass the unstripped virtual table name directly to the admin API, which fails because no physical table named xxx$changelog exists.

Brief change log

Added toPhysicalTablePath() private helper method to FlinkCatalog that strips $changelog, $binlog, and $lake suffixes from the table name, returning the base physical table path
Updated tableExists() to use toPhysicalTablePath() — virtual tables now correctly report existence based on the base table
Updated listPartitions(ObjectPath, CatalogPartitionSpec) to use toPhysicalTablePath() — virtual tables now list partitions of the base table
Updated createPartition() and dropPartition() to use toPhysicalTablePath() — partition management through virtual table names delegates to the base table

Tests

Added FlinkCatalogTest#testVirtualTablePartitionsAndExistence which verifies:
- tableExists() returns true for $changelog and $binlog virtual table names
- listPartitions() returns the same partitions as the base table for both virtual table suffixes
- listPartitions() with a CatalogPartitionSpec works correctly on virtual table names
- createPartition() and dropPartition() work through virtual table names
Existing FlinkCatalogTest#testOperatePartitions continues to pass (no regression)

API and Format

No API or storage format changes. The fix only affects internal catalog method behavior for virtual table names.

Documentation

No documentation changes needed — this is a bug fix that makes virtual table behavior consistent with getTable().

…log/$binlog virtual tables

Copilot

Pull request overview

This PR fixes Flink optimizer failures when it calls partition-related catalog methods (listPartitions, createPartition, dropPartition) on $changelog / $binlog virtual tables by mapping those virtual table names back to their underlying physical table before calling the Fluss admin API.

Changes:

Added toPhysicalTablePath() in FlinkCatalog and switched tableExists(), listPartitions(...), createPartition(), and dropPartition() to use it for virtual table names.
Added unit/integration test coverage to ensure partition listing and partition operations work through $changelog / $binlog names.
Added an end-to-end SHOW PARTITIONS regression test for $changelog virtual tables.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/catalog/FlinkCatalog.java	Strips virtual-table suffixes for existence/partition operations so `$changelog`/`$binlog` map to the base physical table.
fluss-flink/fluss-flink-common/src/test/java/org/apache/fluss/flink/source/ChangelogVirtualTableITCase.java	Adds IT coverage for `SHOW PARTITIONS` on `$changelog` virtual tables.
fluss-flink/fluss-flink-common/src/test/java/org/apache/fluss/flink/catalog/FlinkCatalogTest.java	Adds unit test verifying `tableExists` + partition operations via `$changelog`/`$binlog` paths.
fluss-flink/fluss-flink-common/src/test/java/org/apache/fluss/flink/catalog/FlinkCatalogITCase.java	Adds IT coverage for planner-driven `listPartitions()` behavior on `$changelog` tables.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    private TablePath toPhysicalTablePath(ObjectPath objectPath) {
+        String tableName = objectPath.getObjectName();
+        // Strip virtual table suffixes to get the base physical table name
+        if (tableName.endsWith(CHANGELOG_TABLE_SUFFIX)) {
+            tableName =
+                    tableName.substring(0, tableName.length() - CHANGELOG_TABLE_SUFFIX.length());
+        } else if (tableName.endsWith(BINLOG_TABLE_SUFFIX)) {
+            tableName = tableName.substring(0, tableName.length() - BINLOG_TABLE_SUFFIX.length());
+        } else if (tableName.contains(LAKE_TABLE_SPLITTER)) {
+            tableName = tableName.split("\\" + LAKE_TABLE_SPLITTER)[0];
+        }
+        return TablePath.of(objectPath.getDatabaseName(), tableName);
+    }


MehulBatra · 2026-07-22T19:20:09Z

Thanks for tackling #3640. The root cause is correct, and the listPartitions() fix is the right approach.
A couple of comments:

listPartitions() stripping the virtual suffix is exactly what we want. A partitioned $changelog/$binlog read should resolve partitions from the base table.

tableExists() returning true for virtual tables also makes sense since they are queryable.

One thing I think should change is createPartition() and dropPartition(). $changelog and $binlog are read-only virtual tables, so partition operations should be rejected with a clear error instead of being forwarded to the base table. Because of that, the corresponding tests should expect these operations to fail rather than succeed.

One small nit: I would also keep the $lake handling out of toPhysicalTablePath(). This PR is about $changelog and $binlog, and adding $lake changes the behavior of tableExists() while introducing unrelated scope. It would be better to keep the helper limited to the virtual tables this PR is fixing.

Overall, the read path changes look good to me. I just think we should reject mutating DDL on virtual tables and keep the $lake changes for a separate PR.

MehulBatra

Left one small comment

naivedogger · 2026-07-23T08:21:29Z

Thanks @MehulBatra for your review! I have pushed an update to address the comments:

Reject createPartition / dropPartition on read-only $changelog / $binlog virtual tables with a clear error (read path unchanged).
Reverted the out-of-scope $lake handling in toPhysicalTablePath().
Updated tests to expect these operations to fail.

PTAL, thanks!

jim added 2 commits July 13, 2026 16:15

[flink] Fix TableNotExistException when listing partitions on $change…

36d6902

…log/$binlog virtual tables

[flink] Add E2E tests for SHOW PARTITIONS on virtual tables

9fa8a51

MehulBatra requested a review from Copilot July 22, 2026 18:40

Copilot started reviewing on behalf of MehulBatra July 22, 2026 18:40 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

MehulBatra reviewed Jul 22, 2026

View reviewed changes

naivedogger added 2 commits July 23, 2026 16:10

address comment

1b3b4ca

nit

6f64b75

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[flink] Fix TableNotExistException when listing partitions on changelog / binlog virtual tables#3641

[flink] Fix TableNotExistException when listing partitions on changelog / binlog virtual tables#3641
naivedogger wants to merge 4 commits into
apache:mainfrom
naivedogger:fix/changelog-virtual-table-listPartitions

naivedogger commented Jul 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

MehulBatra commented Jul 22, 2026 •

edited

Loading

Uh oh!

MehulBatra left a comment

Uh oh!

naivedogger commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

naivedogger commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Brief change log

Tests

API and Format

Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

MehulBatra commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MehulBatra left a comment

Choose a reason for hiding this comment

Uh oh!

naivedogger commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

naivedogger commented Jul 13, 2026 •

edited

Loading

MehulBatra commented Jul 22, 2026 •

edited

Loading