[FLINK-40496][table-planner] Apply append-only column rules consistently when altering a materialized table

[AHeise]

What is the purpose of the change

CREATE OR ALTER MATERIALIZED TABLE derived its column changes differently from ALTER ... AS and did not surface a query-driven column reorder to the append-only validation, so reordering existing columns behaved inconsistently: silently applied when the query text was unchanged (rewriting the stored column order with no error), rejected otherwise. The silent path left the stored schema disagreeing with the query and miscompiled the positional refresh INSERT.

Given users_shops = (user_id, shop_id, ds, order_cnt):

CREATE OR ALTER MATERIALIZED TABLE users_shops (shop_id, user_id, ds, order_cnt)
  AS SELECT user_id, shop_id, ds, COUNT(order_id) AS order_cnt FROM ...

The bare column list reorders the columns but the query text is unchanged, so no ModifyDefinitionQuery is produced and the append-only check is skipped — before this change the reorder was applied silently.

Brief change log

• CREATE OR ALTER computes its column diff from the query (via validateAndExtractColumnChanges), positioning old columns by their rank among the columns that survive into the new schema so retained non-persisted columns do not skew the diff.
• The append-only column rules apply to every query-carrying alter (CREATE OR ALTER and ALTER ... AS): existing columns may not be reordered or retyped even when the query text is unchanged. Metadata-only DDL alters are unaffected.
• Second commit: ALTER ... AS and CREATE OR ALTER now share one diff implementation; buildSchemaTableChanges is removed.

Verifying this change

Added and updated tests:

• MaterializedTableStatementITCase — CREATE OR ALTER and ALTER ... AS reject reordering existing columns end-to-end, including a bare-list reorder with an unchanged query.
• SqlMaterializedTableNodeToOperationConverterTest, SqlNodeToOperationSqlCreateOrAlterMaterializedTableConverterTest, AlterMaterializedTableAsQueryOperationValidationTest, ValidateAndExtractColumnChangesTest — operation- and guard-level coverage of the reject/accept cases.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Generated-by: Claude Code (Claude Opus 4.8)

[flinkbot]

CI report:

• 7604fa9 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[AHeise]

Unifies both schema changing parts: ALTER ... AS and CoA.

[raminqaf]

Thanks for handling this @AHeise! I left some comments!

[raminqaf]

A bit confused here why this case was removed. If I understood correctly, we are replacing a nullable column with a non-nullable column. Did you removed it because it is a valid case now?

[AHeise]

Yes — tolerated now, not rejected. That case tightened nullability (nullable -> NOT NULL), which the unified diff treats as a query-inference artifact: CAST('123' AS STRING) AS d yields a NOT NULL type though the user changed nothing about d's declared type, so rejecting it would block legitimate query edits. The stored (nullable) type is kept — no change emitted, hence no error. The unsafe direction (NOT NULL -> nullable) is still rejected; see typeChanged and the loosen/tighten specs in ValidateAndExtractColumnChangesTest.

[raminqaf]

LGTM!