[SPARK-58218][SQL] Replace generic RuntimeException with typed exceptions in AvroDeserializer#57376
Open
ganeshashree wants to merge 1 commit into
Open
[SPARK-58218][SQL] Replace generic RuntimeException with typed exceptions in AvroDeserializer#57376ganeshashree wants to merge 1 commit into
ganeshashree wants to merge 1 commit into
Conversation
…ions in AvroDeserializer ### What changes were proposed in this pull request? `AvroDeserializer` threw three generic `java.lang.RuntimeException`s. This replaces them with typed exceptions: - The `BYTES -> BinaryType` type-mismatch case now throws `IncompatibleSchemaException` (via `errorPrefix`), matching the five sibling type-mismatch cases in the same `newWriter` method that already do so. - The non-nullable array-element and map-value null-read cases now throw a `SparkRuntimeException` with a new error condition `AVRO_CANNOT_READ_NULL_FIELD` (SQLSTATE 22004), the read-side analog of the existing `AVRO_CANNOT_WRITE_NULL_FIELD`. ### Why are the changes needed? Generic `RuntimeException`s are not catchable by error class, carry no SQLSTATE, and (for the binary case) were inconsistent with the sibling cases in the same method. Spark has largely migrated runtime failures to the error-class framework; these sites were left behind. The write path already had a typed, catalogued error for the analogous non-null-field situation (`AVRO_CANNOT_WRITE_NULL_FIELD`); the read path now has its counterpart. ### Does this PR introduce _any_ user-facing change? Yes. When reading Avro data where a null appears in a non-nullable array element or map value, or where a `BYTES` value is not a valid Avro binary, the exception type and message change from a generic `RuntimeException` to a typed `SparkRuntimeException` / `IncompatibleSchemaException` carrying an error condition and SQLSTATE. ### How was this patch tested? Added unit tests in `AvroCatalystDataConversionSuite` that deserialize Avro data containing a null into a non-nullable Catalyst array element and map value, asserting the `AVRO_CANNOT_READ_NULL_FIELD` error condition and field-path parameter via `checkError`. Also ran `SparkThrowableSuite` (error-conditions.json validation), `AvroCatalystDataConversionSuite`, and `AvroSerdeSuite`. Co-authored-by: Isaac
uros-b
approved these changes
Jul 21, 2026
uros-b
left a comment
uros-b
left a comment
Member
There was a problem hiding this comment.
Thank you @ganeshashree! cc @MaxGekk @HyukjinKwon for Avro errors
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
AvroDeserializerthrew three genericjava.lang.RuntimeExceptions. This replaces them with typed exceptions:BYTES -> BinaryTypetype-mismatch case now throwsIncompatibleSchemaException(viaerrorPrefix), matching the five sibling type-mismatch cases in the samenewWritermethod that already do so.SparkRuntimeExceptionwith a new error conditionAVRO_CANNOT_READ_NULL_FIELD(SQLSTATE 22004), the read-side analog of the existingAVRO_CANNOT_WRITE_NULL_FIELD.Why are the changes needed?
Generic
RuntimeExceptions are not catchable by error class, carry no SQLSTATE, and (for the binary case) were inconsistent with the sibling cases in the same method. Spark has largely migrated runtime failures to the error-class framework; these sites were left behind. The write path already had a typed, catalogued error for the analogous non-null-field situation (AVRO_CANNOT_WRITE_NULL_FIELD); the read path now has its counterpart.Does this PR introduce any user-facing change?
Yes. When reading Avro data where a null appears in a non-nullable array element or map value, or where a
BYTESvalue is not a valid Avro binary, the exception type and message change from a genericRuntimeExceptionto a typedSparkRuntimeException/IncompatibleSchemaExceptioncarrying an error condition and SQLSTATE.How was this patch tested?
Added unit tests in
AvroCatalystDataConversionSuitethat deserialize Avro data containing a null into a non-nullable Catalyst array element and map value, asserting theAVRO_CANNOT_READ_NULL_FIELDerror condition and field-path parameter viacheckError. Also ranSparkThrowableSuite(error-conditions.json validation),AvroCatalystDataConversionSuite, andAvroSerdeSuite.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)