[FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast by raminqaf · Pull Request #28758 · apache/flink

raminqaf · 2026-07-16T09:55:59Z

What is the purpose of the change

Follow-up to the initial VARIANT-to-primitive cast support (FLINK-37925). It tightens the cast semantics, turns the string cast into a real value cast, adds a decodability check, and documents the resulting behavior.

Numeric casts from a VARIANT now reject values that do not fit the target instead of silently wrapping. An out-of-range integer or an overflowing DECIMAL fails CAST and returns NULL for TRY_CAST. FLOAT and DOUBLE keep lenient IEEE conversion, where overflow becomes infinity. This follows the behavior of Spark's variant casts.
CAST(VARIANT AS CHAR/VARCHAR) now extracts the scalar value, so a stored string is returned unquoted (foo), while objects and arrays return their JSON representation. JSON_STRING remains the way to get the JSON text, where a string stays quoted ("foo"). Previously the cast reused the JSON serialization and was indistinguishable from JSON_STRING.
Adds BinaryVariant.checkFullySupported(), which walks a variant in place and throws on the first value whose type-code Flink cannot decode (the extended Parquet codes for TIME, nanosecond timestamps, and UUID). It reads only headers and offsets, extracting no leaf values.

Brief change log

Add VariantCastUtils (flink-table-runtime): range-checked numeric narrowing (truncate toward zero, throw on overflow) and scalar-to-string extraction.
VariantToPrimitiveCastRule: route integer and DECIMAL targets through the checked helpers; keep FLOAT/DOUBLE lenient.
VariantToStringCastRule: value extraction instead of JSON serialization; JSON_STRING (JsonStringCallGen) is left unchanged.
LogicalTypeCasts: express VARIANT cast validation in the per-target rules and drop the JSON_STRING cast hint.
Add BinaryVariant.checkFullySupported(); fix the Variant.getInstant javadoc to reference Type.TIMESTAMP_LTZ and document microsecond timestamp precision.
Docs: VARIANT value encoding, the PARSE_JSON type mapping, the string-cast vs JSON_STRING distinction, and that NaN/infinity are not valid JSON (witha store-as-string workaround).

Verifying this change

This change added and updated tests:

CastFunctionITCase: overflow cases (CAST fails, TRY_CAST returns NULL), lenient FLOAT/DOUBLE, and string-cast value extraction (a string returns unquoted, objects return JSON).
LogicalTypeCastsTest: the VARIANT cast-support matrix.
BinaryVariantInternalBuilderTest: PARSE_JSON rejects NaN, Infinity, and -Infinity.
BinaryVariantTest: checkFullySupported accepts supported types and rejects every primitive type-code above 16.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): yes, javadoc-only change to the @PublicEvolving Variant interface (no
signature change)
The serializers: no
The runtime per-record code paths (performance sensitive): yes, only on the VARIANT-to-numeric cast path
Anything that affects deployment or recovery: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no (it refines existing FLINK-37925 behavior)
If yes, how is the feature documented? docs and JavaDocs

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Opus 4.8

flinkbot · 2026-07-16T10:01:19Z

CI report:

f9bd3a0 UNKNOWN
02f5ffc Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

… fit the target type and allow string cast Follow-up to the initial VARIANT-to-primitive cast support. Numeric casts from a VARIANT now reject values that do not fit the target type instead of silently wrapping: an out-of-range integer or an overflowing DECIMAL fails CAST and returns NULL for TRY_CAST, matching Spark's variant cast behavior. FLOAT and DOUBLE keep lenient IEEE conversion, where overflow becomes infinity. The new VariantCastUtils performs the checked narrowing by truncating toward zero and range-checking the value. CAST(VARIANT AS CHAR/VARCHAR) is now allowed and returns the JSON string representation, so the previous JSON_STRING-only restriction and its cast hint are removed. VARIANT cast validation is expressed directly in the per-target rules of LogicalTypeCasts. The VARIANT section of the data types reference documents the overflow behavior and the double-cast pattern for wrap-around narrowing.

…pe mapping Describe which value kinds a VARIANT can hold, including that TIMESTAMP and TIMESTAMP_LTZ use microsecond precision and DATE a day count, and that there is no TIME kind. Add a table showing how PARSE_JSON maps JSON values to variant kinds, and explain that PARSE_JSON cannot produce FLOAT, DATE, TIMESTAMP, TIMESTAMP_LTZ, or BYTES because JSON has no literal for them; those kinds come from other producers such as VariantBuilder or format conversions. Point the PARSE_JSON function reference to the VARIANT data type section for the mapping details.

…nt interface Note that getDateTime and getInstant return values with microsecond precision, which the LocalDateTime and Instant return types do not convey on their own. Fix the getInstant javadoc to reference Type.TIMESTAMP_LTZ, the type it actually accepts, instead of Type.TIMESTAMP.

NaN, Infinity, and -Infinity are not valid JSON, so parseJson throws and TRY_PARSE_JSON returns NULL. Lock the behavior in with a unit test.

…nknown type-codes checkFullySupported walks the variant binary in place and throws VariantTypeException on the first value whose type-code Flink cannot decode, such as the extended Parquet/Spark codes for TIME, TIMESTAMP_NANOS, or UUID. It reads only headers and offsets, extracts no leaf values, and allocates no child variants.

snuyanzin · 2026-07-17T17:24:45Z

+        // NaN and the infinities are not valid JSON, so parsing fails: PARSE_JSON surfaces the
+        // error and TRY_PARSE_JSON returns NULL.


why do we assume that
IIRC Jackson has a property allowing to read them

NaN and Infinity are not part of the JSON specification: https://www.rfc-editor.org/info/rfc8259/#section-6

Numeric values that cannot be represented in the grammar below (such as Infinity and NaN) are not permitted.

Jackson does not allow it by default and you need to switch a flag to allow it. To represent them, the user has to define them as string and cast them as float. The workaround is documented in the variant type

then the question: why do we support these values in some parts of the code and do not support in others?
Do we have a plan to be consistent for all parts?

In Variant we don't and I don't think we should because it follows the JSON spec. Vendors like Snowflake and Databricks also don't allow it.
For cast we do CAST('Nan' AS FLOAT) and it is a valid case

snuyanzin · 2026-07-20T09:27:32Z

+```sql
+CAST(CAST(PARSE_JSON('1000') AS INT) AS TINYINT) -- returns -24 (wrap-around)
+```
+{{< /hint >}}


what is difference here from non VARIANT case?

snuyanzin · 2026-07-20T09:34:38Z

+                        // Spark-style overflow: an integer value outside the target range fails
+                        // CAST and returns NULL for TRY_CAST, instead of wrapping around.
+                        .testTableApiRuntimeError(
+                                call("PARSE_JSON", "40000").cast(SMALLINT()), "overflowed")
+                        .testSqlRuntimeError("CAST(PARSE_JSON('40000') AS SMALLINT)", "overflowed")
                        .testResult(
-                                call("PARSE_JSON", "40000").cast(SMALLINT()),
-                                "CAST(PARSE_JSON('40000') AS SMALLINT)",
-                                (short) -25536,
-                                SMALLINT().notNull())
-                        .testResult(
-                                call("PARSE_JSON", "128").cast(TINYINT()),
-                                "CAST(PARSE_JSON('128') AS TINYINT)",
-                                (byte) -128,
-                                TINYINT().notNull())
+                                call("PARSE_JSON", "40000").tryCast(SMALLINT()),
+                                "TRY_CAST(PARSE_JSON('40000') AS SMALLINT)",
+                                null,
+                                SMALLINT())
+                        .testTableApiRuntimeError(
+                                call("PARSE_JSON", "128").cast(TINYINT()), "overflowed")


why do we do this this way?

Better to have same way as regular cast

Then there is SqlConformance#checkedArithmetic which we can make use to switch between different behavior

The idea here is that the user first casts (converts) the value from variant to the actual type it has and then performs an explicit cast if they want too. In this case the 128 is an INT so the user first needs to cast it INT and then if they want to explicitly cast to TINYINT.

yes, however it does not answer why we expect NULL after that

davidradl · 2026-07-20T15:41:13Z

+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.


why microsecond not nanoseconds?

davidradl · 2026-07-20T15:43:17Z

+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+


I am curious why Date does not go to Flink date which would seem more natural

raminqaf force-pushed the FLINK-37925-followup branch 2 times, most recently from beec0e0 to e03672b Compare July 16, 2026 10:07

raminqaf commented Jul 16, 2026

View reviewed changes

Comment thread docs/content/docs/sql/reference/data-types.md Outdated

raminqaf force-pushed the FLINK-37925-followup branch 2 times, most recently from f02dbdc to 49d4bca Compare July 16, 2026 10:45

github-actions Bot added the community-reviewed PR has been reviewed by the community. label Jul 16, 2026

raminqaf force-pushed the FLINK-37925-followup branch 5 times, most recently from bff4701 to 3750b25 Compare July 17, 2026 11:37

raminqaf added 4 commits July 17, 2026 13:38

[FLINK-37925][core] Test that PARSE_JSON rejects NaN and infinities

f9bd3a0

NaN, Infinity, and -Infinity are not valid JSON, so parseJson throws and TRY_PARSE_JSON returns NULL. Lock the behavior in with a unit test.

raminqaf force-pushed the FLINK-37925-followup branch from 3750b25 to f9bd3a0 Compare July 17, 2026 11:38

raminqaf changed the title ~~[FLINK-37925][table] Range-check VARIANT numeric casts and allow casting VARIANT to string~~ [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast Jul 17, 2026

snuyanzin reviewed Jul 17, 2026

View reviewed changes

Comment thread ...k-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java Outdated

snuyanzin reviewed Jul 17, 2026

View reviewed changes

[FLINK-37925] Add convertToBigDecimal function

02f5ffc

snuyanzin reviewed Jul 20, 2026

View reviewed changes

davidradl reviewed Jul 20, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast#28758

[FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast#28758
raminqaf wants to merge 6 commits into
apache:masterfrom
raminqaf:FLINK-37925-followup

raminqaf commented Jul 16, 2026 •

edited

Loading

Uh oh!

flinkbot commented Jul 16, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

snuyanzin Jul 17, 2026

Uh oh!

raminqaf Jul 19, 2026 •

edited

Loading

Uh oh!

snuyanzin Jul 19, 2026

Uh oh!

raminqaf Jul 20, 2026

Uh oh!

snuyanzin Jul 20, 2026

Uh oh!

snuyanzin Jul 20, 2026

Uh oh!

raminqaf Jul 20, 2026

Uh oh!

snuyanzin Jul 20, 2026 •

edited

Loading

Uh oh!

davidradl Jul 20, 2026

Uh oh!

davidradl Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		// NaN and the infinities are not valid JSON, so parsing fails: PARSE_JSON surfaces the
		// error and TRY_PARSE_JSON returns NULL.

Uh oh!

Conversation

raminqaf commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Was generative AI tooling used to co-author this PR?

Uh oh!

flinkbot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raminqaf Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snuyanzin Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

raminqaf commented Jul 16, 2026 •

edited

Loading

flinkbot commented Jul 16, 2026 •

edited

Loading

raminqaf Jul 19, 2026 •

edited

Loading

snuyanzin Jul 20, 2026 •

edited

Loading