Skip to content

[core] Support geospatial data types - #9251

Open
JingsongLi wants to merge 5 commits into
apache:masterfrom
JingsongLi:codex/support-geospatial-types
Open

[core] Support geospatial data types#9251
JingsongLi wants to merge 5 commits into
apache:masterfrom
JingsongLi:codex/support-geospatial-types

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

What changed

  • Add GEOMETRY and GEOGRAPHY data types aligned with Iceberg v3, including CRS and geography edge algorithms.
  • Store geospatial values as WKB and integrate them with row data, serializers, schema JSON, statistics, Arrow/GeoArrow, and native Parquet logical annotations.
  • Preserve geospatial types in Iceberg v3 metadata, including the default geography algorithm when Iceberg omits it.
  • Preserve GeoArrow extension metadata together with nested Parquet field IDs.
  • Reject Iceberg-incompatible geography CRS values and mark generated Parquet geospatial primitive types with undefined column order.
  • Add native Spark 4.1 SQL support. Spark 3.x, Spark 4.0, and Flink SQL fail explicitly instead of silently exposing geospatial columns as binary.
  • Reject geospatial publication through Iceberg REST Catalog while the bundled Iceberg REST client cannot parse Iceberg v3 geospatial types.
  • Document the type syntax, defaults, Parquet mapping, Java WKB representation, engine compatibility, key restrictions, and Iceberg REST limitation.

Why

Iceberg supports geospatial logical types in format v3. Paimon needs compatible JVM schema semantics and native Parquet annotations so CRS and edge-algorithm metadata survive across Java, Iceberg, Parquet, Spark, Flink, and Arrow boundaries. Engines without native geospatial types must reject the schema rather than perform a lossy conversion to binary.

Compatibility

Entry point Behavior
Java API Native GEOMETRY and GEOGRAPHY support with WKB values
Spark 4.1 Native SQL read/write for CRSs recognized by Spark; geography supports only spherical; mixed SRIDs are unsupported
Spark 3.x and Spark 4.0 Explicit unsupported error
Flink SQL Explicit unsupported error
Iceberg metadata format v3 Supported for table-location, Hadoop catalog, and Hive catalog storage
Iceberg REST Catalog Explicit unsupported error

Geospatial columns require Parquet for data, per-level, and changelog files. They cannot be used as primary, partition, bucket, or sequence keys. When Iceberg metadata is enabled, a GEOGRAPHY CRS cannot contain a comma, including in nested columns. Paimon records null counts but does not expose byte-wise WKB values as lower or upper bounds.

Validation

  • Core Iceberg conversion, schema validation, and field aggregation regression tests passed.
  • Arrow Java nested array/map GeoArrow metadata compatibility test passed.
  • Parquet schema conversion and real-file WKB/geospatial-statistics tests passed.
  • Spark SQL boundary tests passed on Spark 3.2, 3.5, and 4.0; Spark 4.1 native Java and SQL tests passed.
  • Flink SQL geospatial ITCase passed with Flink 1.x and Flink 2.x profiles.
  • Non-fast Maven compile passed with Checkstyle, Spotless, and Enforcer enabled.
  • Docusaurus production build passed; it reports only the existing repository-wide /concepts/overview broken-link warning.
  • git diff --check passed.

@JingsongLi
JingsongLi marked this pull request as ready for review August 16, 2026 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant