diff --git a/docs/docs/concepts/data-types.md b/docs/docs/concepts/data-types.md index d8149e0c078b..a72ae3fe01fa 100644 --- a/docs/docs/concepts/data-types.md +++ b/docs/docs/concepts/data-types.md @@ -76,6 +76,23 @@ All data types supported by Paimon are as follows: BYTES is a synonym for VARBINARY(2147483647). + + GEOMETRY
+ GEOMETRY(crs) + + Data type of a planar geometry encoded as OGC Well-Known Binary (WKB).

+ The optional crs identifies the coordinate reference system. The default is OGC:CRS84. + + + + GEOGRAPHY
+ GEOGRAPHY(crs)
+ GEOGRAPHY(crs, algorithm) + + Data type of a geography whose edges are interpolated on the surface of the coordinate reference system, encoded as OGC Well-Known Binary (WKB).

+ The default crs is OGC:CRS84 and the default edge interpolation algorithm is spherical. Supported algorithms are spherical, vincenty, thomas, andoyer, and karney. + + DECIMAL
DECIMAL(p)
@@ -189,3 +206,11 @@ All data types supported by Paimon are as follows: + +:::note Geospatial type availability + +`GEOMETRY` and `GEOGRAPHY` columns require Parquet for data, per-level, and changelog files. They cannot be used as primary, partition, bucket, sequence, or clustering keys. + +The Paimon Java API supports geospatial columns. Spark 4.1 requires `spark.sql.geospatial.enabled=true`, supports CRSs recognized by Spark, and supports only the `spherical` geography edge algorithm. Flink SQL, Spark 3.x, and Spark 4.0 reject geospatial columns instead of exposing them as binary and losing the CRS or edge algorithm. + +::: diff --git a/docs/docs/concepts/spec/fileformat.md b/docs/docs/concepts/spec/fileformat.md index 252be7f8de85..65a4d5af0c44 100644 --- a/docs/docs/concepts/spec/fileformat.md +++ b/docs/docs/concepts/spec/fileformat.md @@ -63,6 +63,16 @@ The following table lists the type mapping from Paimon type to Parquet type. BINARY + + GEOMETRY(crs) + BINARY + GEOMETRY(crs) + + + GEOGRAPHY(crs, algorithm) + BINARY + GEOGRAPHY(crs, algorithm) + DECIMAL(P, S) P <= 9: INT32, P <= 18: INT64, P > 18: FIXED_LEN_BYTE_ARRAY @@ -142,8 +152,10 @@ The following table lists the type mapping from Paimon type to Parquet type. Limitations: + 1. [Parquet does not support nullable map keys](https://github.com/apache/parquet-format/blob/master/LogicalTypes#maps). 2. Parquet TIMESTAMP type with precision 9 will use INT96, but this int96 is a time zone converted value and requires additional adjustments. +3. Tables containing `GEOMETRY` or `GEOGRAPHY` columns must use Parquet for `file.format`, every entry in `file.format.per.level`, and `changelog-file.format` when configured. ## AVRO diff --git a/docs/docs/flink/quick-start.mdx b/docs/docs/flink/quick-start.mdx index a9054a06738e..e39426c11811 100644 --- a/docs/docs/flink/quick-start.mdx +++ b/docs/docs/flink/quick-start.mdx @@ -31,6 +31,12 @@ under the License. This documentation is a guide for using Paimon in Flink. +:::warning + +Flink SQL does not currently support Paimon `GEOMETRY` or `GEOGRAPHY` columns. Reading, writing, or copying a table whose schema contains either type fails explicitly instead of exposing the column as `VARBINARY` and losing its CRS or edge algorithm. Use the Paimon Java API or Spark 4.1 for geospatial columns. + +::: + ## Jars Paimon currently supports Flink 2.2, 2.1, 2.0, 1.20, 1.19, 1.18, 1.17, 1.16. We recommend the latest Flink version for a better experience. diff --git a/docs/docs/iceberg/index.md b/docs/docs/iceberg/index.md index 6c4b30751773..a3bc42631213 100644 --- a/docs/docs/iceberg/index.md +++ b/docs/docs/iceberg/index.md @@ -98,6 +98,8 @@ Paimon Iceberg compatibility currently supports the following data types. | `TIMESTAMP_LTZ` (precision 3-6) | `timestamptz` | | `TIMESTAMP` (precision 7-9) | `timestamp_ns` | | `TIMESTAMP_LTZ` (precision 7-9) | `timestamptz_ns` | +| `GEOMETRY(crs)` | `geometry(crs)` | +| `GEOGRAPHY(crs, algorithm)` | `geography(crs, algorithm)` | | `ARRAY` | `list` | | `MAP` | `map` | | `ROW` | `struct` | @@ -108,4 +110,12 @@ Paimon Iceberg compatibility currently supports the following data types. - `TIMESTAMP` and `TIMESTAMP_LTZ` types with precision from 3 to 6 are mapped to standard Iceberg timestamp types - `TIMESTAMP` and `TIMESTAMP_LTZ` types with precision from 7 to 9 use nanosecond precision and require Iceberg v3 format +**Note on Geospatial Types:** +- `GEOMETRY` and `GEOGRAPHY` values use OGC Well-Known Binary (WKB). The default CRS is `OGC:CRS84`, and the default geography edge algorithm is `spherical`. +- Geospatial columns require Parquet for data, per-level, and changelog files. When Iceberg metadata is enabled, set `metadata.iceberg.format-version` to `3`. +- Spark SQL supports geospatial columns in Spark 4.1 when `spark.sql.geospatial.enabled=true`, for CRSs recognized by Spark, with the `spherical` geography edge algorithm. Spark 3.x, Spark 4.0, and Flink SQL reject these columns instead of exposing them as binary and losing the CRS or edge algorithm. +- When Iceberg metadata is enabled, a `GEOGRAPHY` CRS cannot contain a comma, including in nested columns, because Iceberg's geospatial type grammar uses commas to separate parameters. +- Iceberg REST catalog publication does not yet support geospatial columns. Use `table-location`, `hadoop-catalog`, or `hive-catalog` metadata storage instead. +- Geospatial columns cannot be primary, partition, bucket, sequence, or clustering keys. Paimon records null counts but does not publish byte-wise lower or upper bounds for WKB values. + ::: diff --git a/docs/docs/iceberg/rest-catalog.mdx b/docs/docs/iceberg/rest-catalog.mdx index 457b138385e6..21d6fbc8fc7b 100644 --- a/docs/docs/iceberg/rest-catalog.mdx +++ b/docs/docs/iceberg/rest-catalog.mdx @@ -35,6 +35,12 @@ You need to provide information about Rest Catalog by setting options prefixed w `'metadata.iceberg.rest.uri' = 'https://localhost/'`. Paimon will try to use these options to initialize an iceberg rest catalog, and use this rest catalog to commit metadata. +:::warning + +Tables containing `GEOMETRY` or `GEOGRAPHY` columns cannot use `'metadata.iceberg.storage' = 'rest-catalog'` because the bundled Iceberg REST client cannot parse Iceberg v3 geospatial types. Use `table-location`, `hadoop-catalog`, or `hive-catalog` metadata storage instead. + +::: + **Dependency:** This feature needs dependency: @@ -128,4 +134,3 @@ There are some cases when committing to iceberg rest catalog: 1. table not exists in iceberg rest-catalog. It'll create the table in rest catalog first, and commit metadata. 2. table exists in iceberg rest-catalog and is compatible with the base metadata stored in the separate directory. It'll directly get the table and commit metadata. 3. table exists, and isn't compatible with the base metadata stored in the separate directory. It'll **drop the table and recreate the table**, then commit metadata. - diff --git a/docs/docs/primary-key-table/sequence-rowkind.mdx b/docs/docs/primary-key-table/sequence-rowkind.mdx index 8a0f49fb7307..bc18e4fcb3e3 100644 --- a/docs/docs/primary-key-table/sequence-rowkind.mdx +++ b/docs/docs/primary-key-table/sequence-rowkind.mdx @@ -55,7 +55,7 @@ CREATE TABLE my_table ( The record with the largest `sequence.field` value will be the last to merge, if the values are the same, the input -order will be used to determine which one is the last one. `sequence.field` supports fields of all data types. +order will be used to determine which one is the last one. `sequence.field` does not support `GEOMETRY`, `GEOGRAPHY`, or managed `BLOB` fields. You can define multiple fields for `sequence.field`, for example `'update_time,flag'`, multiple fields will be compared in order. diff --git a/docs/docs/program-api/java-api.mdx b/docs/docs/program-api/java-api.mdx index 887f36c1bbc1..60695034015d 100644 --- a/docs/docs/program-api/java-api.mdx +++ b/docs/docs/program-api/java-api.mdx @@ -459,6 +459,24 @@ public class StreamWriteTable { | map | org.apache.paimon.data.InternalMap | | InternalRow | org.apache.paimon.data.InternalRow | +### Geospatial Types + +Use the public type factories to declare geospatial columns: + +```java +import org.apache.paimon.types.DataType; +import org.apache.paimon.types.DataTypes; +import org.apache.paimon.types.EdgeAlgorithm; + +DataType defaultGeometry = DataTypes.GEOMETRY(); +DataType projectedGeometry = DataTypes.GEOMETRY("EPSG:3857"); +DataType defaultGeography = DataTypes.GEOGRAPHY(); +DataType karneyGeography = + DataTypes.GEOGRAPHY("OGC:CRS84", EdgeAlgorithm.KARNEY); +``` + +`GEOMETRY` and `GEOGRAPHY` values are represented as OGC Well-Known Binary (WKB) byte arrays in Paimon's internal row API. The default CRS is `OGC:CRS84`, and the default geography edge interpolation algorithm is `EdgeAlgorithm.SPHERICAL`. + ## Predicate Types | SQL Predicate | Paimon Predicate | diff --git a/docs/docs/spark/quick-start.mdx b/docs/docs/spark/quick-start.mdx index d62d3a2b4ee5..b552aa3b0817 100644 --- a/docs/docs/spark/quick-start.mdx +++ b/docs/docs/spark/quick-start.mdx @@ -374,6 +374,16 @@ All Spark's data types are available in package `org.apache.spark.sql.types`. VarBinaryType, BinaryType true + + GeometryType (Spark 4.1) + GeometryType + true + + + GeographyType (Spark 4.1) + GeographyType + true + VariantType(Spark4.0+) VariantType @@ -384,10 +394,16 @@ All Spark's data types are available in package `org.apache.spark.sql.types`. :::warning +Native `GeometryType` and `GeographyType` conversion is supported only in Spark 4.1 and only for CRSs recognized by Spark. Enable it explicitly in production with `--conf spark.sql.geospatial.enabled=true`; Spark enables it automatically only in its test environment. Spark 4.1 supports only the `spherical` geography edge algorithm, so Paimon geography types using `vincenty`, `thomas`, `andoyer`, or `karney` cannot be converted. Spark 3.x and Spark 4.0 reject Paimon geospatial columns instead of exposing them as `BinaryType`, which would lose the CRS or edge algorithm. Paimon does not support Spark geospatial types with mixed SRIDs. + +::: + +:::warning + Due to the previous design, in Spark3.3 and below, Paimon will map both Paimon's TimestampType and LocalZonedTimestamp to Spark's TimestampType, and only correctly handle with TimestampType. Therefore, when using Spark3.3 and below, reads Paimon table with LocalZonedTimestamp type written by other engines, such as Flink, the query result of LocalZonedTimestamp type will have time zone offset, which needs to be adjusted manually. When using Spark3.4 and above, all timestamp types can be parsed correctly. -::: \ No newline at end of file +::: diff --git a/paimon-api/src/main/java/org/apache/paimon/types/DataTypeCasts.java b/paimon-api/src/main/java/org/apache/paimon/types/DataTypeCasts.java index a49e7e4ff6f6..e92c35ec4585 100644 --- a/paimon-api/src/main/java/org/apache/paimon/types/DataTypeCasts.java +++ b/paimon-api/src/main/java/org/apache/paimon/types/DataTypeCasts.java @@ -211,6 +211,11 @@ public static boolean supportsCompatibleCast(DataType sourceType, DataType targe return true; } + if (sourceType.isAnyOf(DataTypeRoot.GEOMETRY, DataTypeRoot.GEOGRAPHY) + || targetType.isAnyOf(DataTypeRoot.GEOMETRY, DataTypeRoot.GEOGRAPHY)) { + return false; + } + return compatibleCastingRules .get(targetType.getTypeRoot()) .contains(sourceType.getTypeRoot()); @@ -230,6 +235,11 @@ private static boolean supportsCasting( return true; } + if (sourceType.isAnyOf(DataTypeRoot.GEOMETRY, DataTypeRoot.GEOGRAPHY) + || targetType.isAnyOf(DataTypeRoot.GEOMETRY, DataTypeRoot.GEOGRAPHY)) { + return false; + } + final DataTypeRoot sourceRoot = sourceType.getTypeRoot(); final DataTypeRoot targetRoot = targetType.getTypeRoot(); diff --git a/paimon-api/src/main/java/org/apache/paimon/types/DataTypeDefaultVisitor.java b/paimon-api/src/main/java/org/apache/paimon/types/DataTypeDefaultVisitor.java index af680ede62e2..bdd31d194752 100644 --- a/paimon-api/src/main/java/org/apache/paimon/types/DataTypeDefaultVisitor.java +++ b/paimon-api/src/main/java/org/apache/paimon/types/DataTypeDefaultVisitor.java @@ -119,6 +119,16 @@ public R visit(BlobType blobType) { return defaultMethod(blobType); } + @Override + public R visit(GeometryType geometryType) { + return defaultMethod(geometryType); + } + + @Override + public R visit(GeographyType geographyType) { + return defaultMethod(geographyType); + } + @Override public R visit(ArrayType arrayType) { return defaultMethod(arrayType); diff --git a/paimon-api/src/main/java/org/apache/paimon/types/DataTypeJsonParser.java b/paimon-api/src/main/java/org/apache/paimon/types/DataTypeJsonParser.java index 4079dd8c47c0..f808114666d5 100644 --- a/paimon-api/src/main/java/org/apache/paimon/types/DataTypeJsonParser.java +++ b/paimon-api/src/main/java/org/apache/paimon/types/DataTypeJsonParser.java @@ -331,6 +331,8 @@ private enum Keyword { LEGACY, VARIANT, BLOB, + GEOMETRY, + GEOGRAPHY, NOT } @@ -549,6 +551,10 @@ private DataType parseTypeByKeyword() { return new VariantType(); case BLOB: return new BlobType(); + case GEOMETRY: + return parseGeometryType(); + case GEOGRAPHY: + return parseGeographyType(); case VECTOR: return parseVectorType(); default: @@ -683,5 +689,40 @@ private DataType parseVectorType() { nextToken(TokenType.END_SUBTYPE); return DataTypes.VECTOR(length, elementType); } + + private DataType parseGeometryType() { + if (!hasNextToken(TokenType.BEGIN_PARAMETER)) { + return DataTypes.GEOMETRY(); + } + nextToken(TokenType.BEGIN_PARAMETER); + String crs = parseGeospatialParameter(); + nextToken(TokenType.END_PARAMETER); + return DataTypes.GEOMETRY(crs); + } + + private DataType parseGeographyType() { + if (!hasNextToken(TokenType.BEGIN_PARAMETER)) { + return DataTypes.GEOGRAPHY(); + } + nextToken(TokenType.BEGIN_PARAMETER); + String crs = parseGeospatialParameter(); + EdgeAlgorithm algorithm = GeographyType.DEFAULT_ALGORITHM; + if (hasNextToken(TokenType.LIST_SEPARATOR)) { + nextToken(TokenType.LIST_SEPARATOR); + algorithm = EdgeAlgorithm.fromName(parseGeospatialParameter()); + } + nextToken(TokenType.END_PARAMETER); + return DataTypes.GEOGRAPHY(crs, algorithm); + } + + private String parseGeospatialParameter() { + nextToken(); + if (token().type != TokenType.IDENTIFIER + && token().type != TokenType.LITERAL_STRING + && token().type != TokenType.KEYWORD) { + throw parsingError("Geospatial type parameter expected."); + } + return token().value; + } } } diff --git a/paimon-api/src/main/java/org/apache/paimon/types/DataTypeRoot.java b/paimon-api/src/main/java/org/apache/paimon/types/DataTypeRoot.java index f55da9c4706f..27deb50d93d6 100644 --- a/paimon-api/src/main/java/org/apache/paimon/types/DataTypeRoot.java +++ b/paimon-api/src/main/java/org/apache/paimon/types/DataTypeRoot.java @@ -104,6 +104,10 @@ public enum DataTypeRoot { BLOB(DataTypeFamily.PREDEFINED), + GEOMETRY(DataTypeFamily.PREDEFINED), + + GEOGRAPHY(DataTypeFamily.PREDEFINED), + ARRAY(DataTypeFamily.CONSTRUCTED, DataTypeFamily.COLLECTION), VECTOR(DataTypeFamily.CONSTRUCTED, DataTypeFamily.COLLECTION), diff --git a/paimon-api/src/main/java/org/apache/paimon/types/DataTypeVisitor.java b/paimon-api/src/main/java/org/apache/paimon/types/DataTypeVisitor.java index 6e377309f237..0b546e8a914f 100644 --- a/paimon-api/src/main/java/org/apache/paimon/types/DataTypeVisitor.java +++ b/paimon-api/src/main/java/org/apache/paimon/types/DataTypeVisitor.java @@ -66,6 +66,14 @@ public interface DataTypeVisitor { R visit(BlobType blobType); + default R visit(GeometryType geometryType) { + throw new UnsupportedOperationException("Unsupported type: " + geometryType); + } + + default R visit(GeographyType geographyType) { + throw new UnsupportedOperationException("Unsupported type: " + geographyType); + } + R visit(ArrayType arrayType); R visit(VectorType vectorType); diff --git a/paimon-api/src/main/java/org/apache/paimon/types/DataTypes.java b/paimon-api/src/main/java/org/apache/paimon/types/DataTypes.java index 39b180651ef5..b953a14d9bd0 100644 --- a/paimon-api/src/main/java/org/apache/paimon/types/DataTypes.java +++ b/paimon-api/src/main/java/org/apache/paimon/types/DataTypes.java @@ -163,6 +163,26 @@ public static BlobType BLOB() { return new BlobType(); } + public static GeometryType GEOMETRY() { + return new GeometryType(); + } + + public static GeometryType GEOMETRY(String crs) { + return new GeometryType(crs); + } + + public static GeographyType GEOGRAPHY() { + return new GeographyType(); + } + + public static GeographyType GEOGRAPHY(String crs) { + return new GeographyType(crs); + } + + public static GeographyType GEOGRAPHY(String crs, EdgeAlgorithm algorithm) { + return new GeographyType(crs, algorithm); + } + public static OptionalInt getPrecision(DataType dataType) { return dataType.accept(PRECISION_EXTRACTOR); } diff --git a/paimon-api/src/main/java/org/apache/paimon/types/EdgeAlgorithm.java b/paimon-api/src/main/java/org/apache/paimon/types/EdgeAlgorithm.java new file mode 100644 index 000000000000..b16675d8b076 --- /dev/null +++ b/paimon-api/src/main/java/org/apache/paimon/types/EdgeAlgorithm.java @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.types; + +import org.apache.paimon.annotation.Public; + +import java.util.Locale; + +/** Algorithm used to interpolate geography edges. */ +@Public +public enum EdgeAlgorithm { + SPHERICAL, + VINCENTY, + THOMAS, + ANDOYER, + KARNEY; + + public static EdgeAlgorithm fromName(String algorithmName) { + if (algorithmName == null) { + throw new IllegalArgumentException("Invalid edge interpolation algorithm: null"); + } + + try { + return valueOf(algorithmName.toUpperCase(Locale.ROOT)); + } catch (IllegalArgumentException e) { + throw new IllegalArgumentException( + "Invalid edge interpolation algorithm: " + algorithmName, e); + } + } + + @Override + public String toString() { + return name().toLowerCase(Locale.ROOT); + } +} diff --git a/paimon-api/src/main/java/org/apache/paimon/types/GeographyType.java b/paimon-api/src/main/java/org/apache/paimon/types/GeographyType.java new file mode 100644 index 000000000000..d4f115c51249 --- /dev/null +++ b/paimon-api/src/main/java/org/apache/paimon/types/GeographyType.java @@ -0,0 +1,104 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.types; + +import org.apache.paimon.annotation.Public; + +import java.util.Locale; +import java.util.Objects; + +/** Geography encoded as OGC Well-Known Binary. */ +@Public +public class GeographyType extends DataType { + + private static final long serialVersionUID = 1L; + + public static final String DEFAULT_CRS = "OGC:CRS84"; + + public static final EdgeAlgorithm DEFAULT_ALGORITHM = EdgeAlgorithm.SPHERICAL; + + private static final String FORMAT = "GEOGRAPHY(%s, %s)"; + + private final String crs; + + private final EdgeAlgorithm algorithm; + + public GeographyType(boolean isNullable, String crs, EdgeAlgorithm algorithm) { + super(isNullable, DataTypeRoot.GEOGRAPHY); + this.crs = GeometryType.validateCrs(crs == null ? DEFAULT_CRS : crs); + this.algorithm = algorithm == null ? DEFAULT_ALGORITHM : algorithm; + } + + public GeographyType(String crs, EdgeAlgorithm algorithm) { + this(true, crs, algorithm); + } + + public GeographyType(String crs) { + this(crs, DEFAULT_ALGORITHM); + } + + public GeographyType() { + this(DEFAULT_CRS); + } + + public String getCrs() { + return crs; + } + + public EdgeAlgorithm getAlgorithm() { + return algorithm; + } + + @Override + public int defaultSize() { + return 20; + } + + @Override + public DataType copy(boolean isNullable) { + return new GeographyType(isNullable, crs, algorithm); + } + + @Override + public String asSQLString() { + return withNullability(FORMAT, GeometryType.formatCrs(crs), algorithm); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass() || !super.equals(o)) { + return false; + } + GeographyType that = (GeographyType) o; + return crs.equalsIgnoreCase(that.crs) && algorithm == that.algorithm; + } + + @Override + public int hashCode() { + return Objects.hash(super.hashCode(), crs.toUpperCase(Locale.ROOT), algorithm); + } + + @Override + public R accept(DataTypeVisitor visitor) { + return visitor.visit(this); + } +} diff --git a/paimon-api/src/main/java/org/apache/paimon/types/GeometryType.java b/paimon-api/src/main/java/org/apache/paimon/types/GeometryType.java new file mode 100644 index 000000000000..2a98b4a2ac39 --- /dev/null +++ b/paimon-api/src/main/java/org/apache/paimon/types/GeometryType.java @@ -0,0 +1,121 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.types; + +import org.apache.paimon.annotation.Public; + +import java.util.Locale; +import java.util.Objects; + +import static org.apache.paimon.utils.EncodingUtils.escapeSingleQuotes; + +/** Planar geometry encoded as OGC Well-Known Binary. */ +@Public +public class GeometryType extends DataType { + + private static final long serialVersionUID = 1L; + + public static final String DEFAULT_CRS = "OGC:CRS84"; + + private static final String FORMAT = "GEOMETRY(%s)"; + + private final String crs; + + public GeometryType(boolean isNullable, String crs) { + super(isNullable, DataTypeRoot.GEOMETRY); + this.crs = validateCrs(crs == null ? DEFAULT_CRS : crs); + } + + public GeometryType(String crs) { + this(true, crs); + } + + public GeometryType() { + this(DEFAULT_CRS); + } + + public String getCrs() { + return crs; + } + + @Override + public int defaultSize() { + return 20; + } + + @Override + public DataType copy(boolean isNullable) { + return new GeometryType(isNullable, crs); + } + + @Override + public String asSQLString() { + return withNullability(FORMAT, formatCrs(crs)); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass() || !super.equals(o)) { + return false; + } + GeometryType that = (GeometryType) o; + return crs.equalsIgnoreCase(that.crs); + } + + @Override + public int hashCode() { + return Objects.hash(super.hashCode(), crs.toUpperCase(Locale.ROOT)); + } + + @Override + public R accept(DataTypeVisitor visitor) { + return visitor.visit(this); + } + + static String validateCrs(String crs) { + if (crs.isEmpty()) { + throw new IllegalArgumentException("Invalid CRS: " + crs); + } + return crs; + } + + static String formatCrs(String crs) { + if (Character.isDigit(crs.charAt(0))) { + return "'" + escapeSingleQuotes(crs) + "'"; + } + for (int i = 0; i < crs.length(); i++) { + char character = crs.charAt(i); + if (Character.isWhitespace(character) + || character == '<' + || character == '>' + || character == '(' + || character == ')' + || character == ',' + || character == '.' + || character == '\'' + || character == '`') { + return "'" + escapeSingleQuotes(crs) + "'"; + } + } + return crs; + } +} diff --git a/paimon-api/src/test/java/org/apache/paimon/types/GeospatialTypeTest.java b/paimon-api/src/test/java/org/apache/paimon/types/GeospatialTypeTest.java new file mode 100644 index 000000000000..d467b7b91ae1 --- /dev/null +++ b/paimon-api/src/test/java/org/apache/paimon/types/GeospatialTypeTest.java @@ -0,0 +1,78 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.types; + +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +class GeospatialTypeTest { + + @Test + void testIcebergCompatibleDefaultsAndFormatting() { + assertThat(new GeometryType().asSQLString()).isEqualTo("GEOMETRY(OGC:CRS84)"); + assertThat(new GeographyType().asSQLString()).isEqualTo("GEOGRAPHY(OGC:CRS84, spherical)"); + assertThat(new GeographyType().notNull().asSQLString()) + .isEqualTo("GEOGRAPHY(OGC:CRS84, spherical) NOT NULL"); + } + + @Test + void testCrsEqualityIsCaseInsensitive() { + assertThat(new GeometryType("OGC:CRS84")).isEqualTo(new GeometryType("ogc:crs84")); + assertThat(new GeometryType("OGC:CRS84").hashCode()) + .isEqualTo(new GeometryType("ogc:crs84").hashCode()); + assertThat(new GeographyType("OGC:CRS84", EdgeAlgorithm.KARNEY)) + .isEqualTo(new GeographyType("ogc:crs84", EdgeAlgorithm.KARNEY)); + assertThat(new GeographyType("OGC:CRS84", EdgeAlgorithm.KARNEY)) + .isNotEqualTo(new GeographyType("OGC:CRS84", EdgeAlgorithm.SPHERICAL)); + GeometryType custom = new GeometryType("custom, crs's definition"); + assertThat(DataTypeJsonParser.parseAtomicTypeSQLString(custom.asSQLString())) + .isEqualTo(custom); + } + + @Test + void testInvalidParameters() { + assertThatThrownBy(() -> new GeometryType("")) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("Invalid CRS"); + assertThatThrownBy(() -> EdgeAlgorithm.fromName("rhumb")) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("Invalid edge interpolation algorithm"); + } + + @Test + void testOnlyIdenticalGeospatialTypesCanBeCast() { + assertThat( + DataTypeCasts.supportsCast( + new GeometryType("OGC:CRS84"), + new GeometryType("ogc:crs84").notNull(), + true)) + .isTrue(); + assertThat( + DataTypeCasts.supportsCast( + new GeometryType("OGC:CRS84"), new GeometryType("EPSG:3857"), true)) + .isFalse(); + assertThat( + DataTypeCasts.supportsCompatibleCast( + new GeographyType("OGC:CRS84", EdgeAlgorithm.SPHERICAL), + new GeographyType("OGC:CRS84", EdgeAlgorithm.KARNEY))) + .isFalse(); + } +} diff --git a/paimon-arrow/src/main/java/org/apache/paimon/arrow/ArrowFieldTypeConversion.java b/paimon-arrow/src/main/java/org/apache/paimon/arrow/ArrowFieldTypeConversion.java index 80d9208053b8..a6a1328a3163 100644 --- a/paimon-arrow/src/main/java/org/apache/paimon/arrow/ArrowFieldTypeConversion.java +++ b/paimon-arrow/src/main/java/org/apache/paimon/arrow/ArrowFieldTypeConversion.java @@ -30,6 +30,8 @@ import org.apache.paimon.types.DecimalType; import org.apache.paimon.types.DoubleType; import org.apache.paimon.types.FloatType; +import org.apache.paimon.types.GeographyType; +import org.apache.paimon.types.GeometryType; import org.apache.paimon.types.IntType; import org.apache.paimon.types.LocalZonedTimestampType; import org.apache.paimon.types.MapType; @@ -85,6 +87,16 @@ public FieldType visit(VarBinaryType varBinaryType) { varBinaryType.isNullable(), Types.MinorType.VARBINARY.getType(), null); } + @Override + public FieldType visit(GeometryType geometryType) { + throw new UnsupportedOperationException("Unsupported primitive type: " + geometryType); + } + + @Override + public FieldType visit(GeographyType geographyType) { + throw new UnsupportedOperationException("Unsupported primitive type: " + geographyType); + } + @Override public FieldType visit(DecimalType decimalType) { return new FieldType( diff --git a/paimon-arrow/src/main/java/org/apache/paimon/arrow/ArrowUtils.java b/paimon-arrow/src/main/java/org/apache/paimon/arrow/ArrowUtils.java index 5ea4ac8aaac9..92812bbeab67 100644 --- a/paimon-arrow/src/main/java/org/apache/paimon/arrow/ArrowUtils.java +++ b/paimon-arrow/src/main/java/org/apache/paimon/arrow/ArrowUtils.java @@ -56,7 +56,9 @@ import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; +import java.util.LinkedHashMap; import java.util.List; +import java.util.Map; import java.util.stream.Collectors; import static org.apache.paimon.utils.StringUtils.toLowerCaseIfNeed; @@ -132,12 +134,7 @@ public static Field toArrowField( int depth, ArrowFieldTypeConversion.ArrowFieldTypeVisitor visitor) { FieldType fieldType = dataType.accept(visitor); - fieldType = - new FieldType( - fieldType.isNullable(), - fieldType.getType(), - fieldType.getDictionary(), - Collections.singletonMap(PARQUET_FIELD_ID, String.valueOf(fieldId))); + fieldType = withFieldId(fieldType, fieldId); List children = null; if (dataType instanceof ArrayType || dataType instanceof VectorType) { final DataType elementType; @@ -157,11 +154,10 @@ public static Field toArrowField( typeInner.isNullable(), typeInner.getType(), typeInner.getDictionary(), - Collections.singletonMap( - PARQUET_FIELD_ID, - String.valueOf( - SpecialFields.getArrayElementFieldId( - fieldId, depth + 1)))), + withFieldIdMetadata( + typeInner, + SpecialFields.getArrayElementFieldId( + fieldId, depth + 1))), field.getChildren()); children = Collections.singletonList(field); } else if (dataType instanceof MapType) { @@ -182,11 +178,9 @@ public static Field toArrowField( keyType.isNullable(), keyType.getType(), keyType.getDictionary(), - Collections.singletonMap( - PARQUET_FIELD_ID, - String.valueOf( - SpecialFields.getMapKeyFieldId( - fieldId, depth + 1)))), + withFieldIdMetadata( + keyType, + SpecialFields.getMapKeyFieldId(fieldId, depth + 1))), keyField.getChildren()); Field valueField = @@ -204,11 +198,9 @@ public static Field toArrowField( valueType.isNullable(), valueType.getType(), valueType.getDictionary(), - Collections.singletonMap( - PARQUET_FIELD_ID, - String.valueOf( - SpecialFields.getMapValueFieldId( - fieldId, depth + 1)))), + withFieldIdMetadata( + valueType, + SpecialFields.getMapValueFieldId(fieldId, depth + 1))), valueField.getChildren()); FieldType structType = @@ -246,6 +238,23 @@ public static Field toArrowField( return new Field(fieldName, fieldType, children); } + private static FieldType withFieldId(FieldType fieldType, int fieldId) { + return new FieldType( + fieldType.isNullable(), + fieldType.getType(), + fieldType.getDictionary(), + withFieldIdMetadata(fieldType, fieldId)); + } + + private static Map withFieldIdMetadata(FieldType fieldType, int fieldId) { + Map metadata = new LinkedHashMap<>(); + if (fieldType.getMetadata() != null) { + metadata.putAll(fieldType.getMetadata()); + } + metadata.put(PARQUET_FIELD_ID, String.valueOf(fieldId)); + return metadata; + } + public static ArrowFieldWriter[] createArrowFieldWriters( VectorSchemaRoot vectorSchemaRoot, RowType rowType) { ArrowFieldWriter[] fieldWriters = new ArrowFieldWriter[rowType.getFieldCount()]; diff --git a/paimon-arrow/src/main/java/org/apache/paimon/arrow/converter/Arrow2PaimonVectorConverter.java b/paimon-arrow/src/main/java/org/apache/paimon/arrow/converter/Arrow2PaimonVectorConverter.java index ce0d067e6392..e3a9a53d51bc 100644 --- a/paimon-arrow/src/main/java/org/apache/paimon/arrow/converter/Arrow2PaimonVectorConverter.java +++ b/paimon-arrow/src/main/java/org/apache/paimon/arrow/converter/Arrow2PaimonVectorConverter.java @@ -57,6 +57,8 @@ import org.apache.paimon.types.DecimalType; import org.apache.paimon.types.DoubleType; import org.apache.paimon.types.FloatType; +import org.apache.paimon.types.GeographyType; +import org.apache.paimon.types.GeometryType; import org.apache.paimon.types.IntType; import org.apache.paimon.types.LocalZonedTimestampType; import org.apache.paimon.types.MapType; @@ -237,6 +239,16 @@ public byte[] getBytes() { }; } + @Override + public Arrow2PaimonVectorConverter visit(GeometryType geometryType) { + return visit(new VarBinaryType(geometryType.isNullable(), VarBinaryType.MAX_LENGTH)); + } + + @Override + public Arrow2PaimonVectorConverter visit(GeographyType geographyType) { + return visit(new VarBinaryType(geographyType.isNullable(), VarBinaryType.MAX_LENGTH)); + } + @Override public Arrow2PaimonVectorConverter visit(DecimalType decimalType) { return vector -> diff --git a/paimon-arrow/src/main/java/org/apache/paimon/arrow/writer/ArrowFieldWriterFactoryVisitor.java b/paimon-arrow/src/main/java/org/apache/paimon/arrow/writer/ArrowFieldWriterFactoryVisitor.java index 419da16afdd3..972955955cc4 100644 --- a/paimon-arrow/src/main/java/org/apache/paimon/arrow/writer/ArrowFieldWriterFactoryVisitor.java +++ b/paimon-arrow/src/main/java/org/apache/paimon/arrow/writer/ArrowFieldWriterFactoryVisitor.java @@ -29,6 +29,8 @@ import org.apache.paimon.types.DecimalType; import org.apache.paimon.types.DoubleType; import org.apache.paimon.types.FloatType; +import org.apache.paimon.types.GeographyType; +import org.apache.paimon.types.GeometryType; import org.apache.paimon.types.IntType; import org.apache.paimon.types.LocalZonedTimestampType; import org.apache.paimon.types.MapType; @@ -81,6 +83,16 @@ public ArrowFieldWriterFactory visit(VarBinaryType varBinaryType) { return ArrowFieldWriters.BinaryWriter::new; } + @Override + public ArrowFieldWriterFactory visit(GeometryType geometryType) { + return ArrowFieldWriters.BinaryWriter::new; + } + + @Override + public ArrowFieldWriterFactory visit(GeographyType geographyType) { + return ArrowFieldWriters.BinaryWriter::new; + } + @Override public ArrowFieldWriterFactory visit(DecimalType decimalType) { return (fieldVector, isNullable) -> diff --git a/paimon-arrow/src/test/java/org/apache/paimon/arrow/ArrowSchemaMetadataCompatibilityTest.java b/paimon-arrow/src/test/java/org/apache/paimon/arrow/ArrowSchemaMetadataCompatibilityTest.java index 73b8a2b28a4d..53c6e4ee55be 100644 --- a/paimon-arrow/src/test/java/org/apache/paimon/arrow/ArrowSchemaMetadataCompatibilityTest.java +++ b/paimon-arrow/src/test/java/org/apache/paimon/arrow/ArrowSchemaMetadataCompatibilityTest.java @@ -21,8 +21,10 @@ import org.apache.paimon.format.FormatMetadataUtils; import org.apache.paimon.types.DataField; import org.apache.paimon.types.DataTypes; +import org.apache.paimon.types.EdgeAlgorithm; import org.apache.paimon.types.RowType; +import org.apache.arrow.vector.types.pojo.ArrowType; import org.apache.arrow.vector.types.pojo.Field; import org.apache.arrow.vector.types.pojo.FieldType; import org.apache.arrow.vector.types.pojo.Schema; @@ -30,6 +32,7 @@ import java.nio.ByteBuffer; import java.util.ArrayList; +import java.util.Collections; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; @@ -106,6 +109,67 @@ public void testArrowJavaSchemaCanBeReadByFormatMetadata() { .containsAllEntriesOf(tagsMetadata); } + @Test + public void testGeoArrowMetadataCanBeReadByArrowJava() { + RowType rowType = + DataTypes.ROW( + DataTypes.FIELD(0, "geom", DataTypes.GEOMETRY("EPSG:3857")), + DataTypes.FIELD( + 1, + "nested", + DataTypes.ROW( + DataTypes.FIELD( + 2, + "geog", + DataTypes.GEOGRAPHY( + "EPSG:4326", EdgeAlgorithm.KARNEY)))), + DataTypes.FIELD( + 3, "geometries", DataTypes.ARRAY(DataTypes.GEOMETRY("EPSG:32632"))), + DataTypes.FIELD( + 4, + "geo_map", + DataTypes.MAP( + DataTypes.GEOGRAPHY("EPSG:4326", EdgeAlgorithm.THOMAS), + DataTypes.GEOMETRY("EPSG:3857")))); + + byte[] schemaBytes = + FormatMetadataUtils.buildArrowSchemaMetadata( + rowType, Collections.emptyMap(), FormatMetadataUtils.PARQUET_FIELD_ID_KEY); + Schema schema = Schema.deserializeMessage(ByteBuffer.wrap(schemaBytes)); + + Field geometry = schema.findField("geom"); + assertThat(geometry.getType()).isEqualTo(ArrowType.Binary.INSTANCE); + assertThat(geometry.getMetadata()) + .containsEntry("ARROW:extension:name", "geoarrow.wkb") + .containsEntry("ARROW:extension:metadata", "{\"crs\":\"EPSG:3857\"}"); + + Field geography = schema.findField("nested").getChildren().get(0); + assertThat(geography.getType()).isEqualTo(ArrowType.Binary.INSTANCE); + assertThat(geography.getMetadata()) + .containsEntry("ARROW:extension:name", "geoarrow.wkb") + .containsEntry( + "ARROW:extension:metadata", "{\"edges\":\"karney\",\"crs\":\"EPSG:4326\"}"); + + Field arrayElement = schema.findField("geometries").getChildren().get(0); + assertThat(arrayElement.getMetadata()) + .containsKey(FormatMetadataUtils.PARQUET_FIELD_ID_KEY) + .containsEntry("ARROW:extension:name", "geoarrow.wkb") + .containsEntry("ARROW:extension:metadata", "{\"crs\":\"EPSG:32632\"}"); + + Field mapEntry = schema.findField("geo_map").getChildren().get(0); + Field mapKey = mapEntry.getChildren().get(0); + assertThat(mapKey.getMetadata()) + .containsKey(FormatMetadataUtils.PARQUET_FIELD_ID_KEY) + .containsEntry("ARROW:extension:name", "geoarrow.wkb") + .containsEntry( + "ARROW:extension:metadata", "{\"edges\":\"thomas\",\"crs\":\"EPSG:4326\"}"); + Field mapValue = mapEntry.getChildren().get(1); + assertThat(mapValue.getMetadata()) + .containsKey(FormatMetadataUtils.PARQUET_FIELD_ID_KEY) + .containsEntry("ARROW:extension:name", "geoarrow.wkb") + .containsEntry("ARROW:extension:metadata", "{\"crs\":\"EPSG:3857\"}"); + } + private static RowType rowType() { return DataTypes.ROW( DataTypes.FIELD(0, "id", DataTypes.INT()), diff --git a/paimon-arrow/src/test/java/org/apache/paimon/arrow/ArrowUtilsTest.java b/paimon-arrow/src/test/java/org/apache/paimon/arrow/ArrowUtilsTest.java index 75d70c849b9c..fd9c6eb0b916 100644 --- a/paimon-arrow/src/test/java/org/apache/paimon/arrow/ArrowUtilsTest.java +++ b/paimon-arrow/src/test/java/org/apache/paimon/arrow/ArrowUtilsTest.java @@ -125,6 +125,24 @@ public void testVectorType() { Assertions.assertThat(field.getChildren()).hasSize(1); } + @Test + public void testGeospatialTypesUnsupported() { + Assertions.assertThatThrownBy( + () -> ArrowUtils.toArrowField("geom", 7, DataTypes.GEOMETRY(), 0)) + .isInstanceOf(UnsupportedOperationException.class) + .hasMessage("Unsupported primitive type: GEOMETRY(OGC:CRS84)"); + + Assertions.assertThatThrownBy( + () -> + ArrowUtils.toArrowField( + "geographies", + 8, + DataTypes.ARRAY(DataTypes.GEOGRAPHY()), + 0)) + .isInstanceOf(UnsupportedOperationException.class) + .hasMessage("Unsupported primitive type: GEOGRAPHY(OGC:CRS84, spherical)"); + } + @Test public void testSameRootAllocatorIncludesNestedVectors() { try (RootAllocator allocator = new RootAllocator(); diff --git a/paimon-codegen/src/main/scala/org/apache/paimon/codegen/GenerateUtils.scala b/paimon-codegen/src/main/scala/org/apache/paimon/codegen/GenerateUtils.scala index 87e1bcca2897..f1043858ce6f 100644 --- a/paimon-codegen/src/main/scala/org/apache/paimon/codegen/GenerateUtils.scala +++ b/paimon-codegen/src/main/scala/org/apache/paimon/codegen/GenerateUtils.scala @@ -375,7 +375,7 @@ object GenerateUtils { // ordered by type root definition case CHAR | VARCHAR => BINARY_STRING case BOOLEAN => className[JBoolean] - case BINARY | VARBINARY => "byte[]" + case BINARY | VARBINARY | GEOMETRY | GEOGRAPHY => "byte[]" case DECIMAL => className[Decimal] case TINYINT => className[JByte] case SMALLINT => className[JShort] @@ -404,7 +404,7 @@ object GenerateUtils { s"(($BINARY_STRING) $rowTerm.getString($indexTerm))" case BOOLEAN => s"$rowTerm.getBoolean($indexTerm)" - case BINARY | VARBINARY => + case BINARY | VARBINARY | GEOMETRY | GEOGRAPHY => s"$rowTerm.getBinary($indexTerm)" case DECIMAL => s"$rowTerm.getDecimal($indexTerm, ${getPrecision(t)}, ${getScale(t)})" @@ -594,7 +594,7 @@ object GenerateUtils { s"$writerTerm.writeString($indexTerm, $fieldValTerm)" case BOOLEAN => s"$writerTerm.writeBoolean($indexTerm, $fieldValTerm)" - case BINARY | VARBINARY => + case BINARY | VARBINARY | GEOMETRY | GEOGRAPHY => s"$writerTerm.writeBinary($indexTerm, $fieldValTerm, 0, $fieldValTerm.length)" case DECIMAL => s"$writerTerm.writeDecimal($indexTerm, $fieldValTerm, ${getPrecision(t)})" diff --git a/paimon-codegen/src/main/scala/org/apache/paimon/codegen/ScalarOperatorGens.scala b/paimon-codegen/src/main/scala/org/apache/paimon/codegen/ScalarOperatorGens.scala index fb8a04404968..d2cf7362ba3f 100644 --- a/paimon-codegen/src/main/scala/org/apache/paimon/codegen/ScalarOperatorGens.scala +++ b/paimon-codegen/src/main/scala/org/apache/paimon/codegen/ScalarOperatorGens.scala @@ -61,6 +61,14 @@ object ScalarOperatorGens { generateOperatorIfNotNull(ctx, resultType, left, right)( (leftTerm, rightTerm) => s"$leftTerm.equals($rightTerm)") } + // geospatial values use WKB byte arrays internally + else if ( + (isGeometry(left.resultType) && isGeometry(right.resultType)) || + (isGeography(left.resultType) && isGeography(right.resultType)) + ) { + generateOperatorIfNotNull(ctx, resultType, left, right)( + (leftTerm, rightTerm) => s"java.util.Arrays.equals($leftTerm, $rightTerm)") + } // numeric types else if (isNumeric(left.resultType) && isNumeric(right.resultType)) { generateComparison(ctx, "==", left, right, resultType) diff --git a/paimon-codegen/src/test/java/org/apache/paimon/codegen/EqualiserCodeGeneratorTest.java b/paimon-codegen/src/test/java/org/apache/paimon/codegen/EqualiserCodeGeneratorTest.java index 6bbb4f765135..3d75cdcf41a6 100644 --- a/paimon-codegen/src/test/java/org/apache/paimon/codegen/EqualiserCodeGeneratorTest.java +++ b/paimon-codegen/src/test/java/org/apache/paimon/codegen/EqualiserCodeGeneratorTest.java @@ -82,6 +82,14 @@ public class EqualiserCodeGeneratorTest { TEST_DATA.put( DataTypeRoot.VARBINARY, new GeneratedData(DataTypes.VARBINARY(1), Pair.of("7".getBytes(), "8".getBytes()))); + TEST_DATA.put( + DataTypeRoot.GEOMETRY, + new GeneratedData( + DataTypes.GEOMETRY(), Pair.of("geom-1".getBytes(), "geom-2".getBytes()))); + TEST_DATA.put( + DataTypeRoot.GEOGRAPHY, + new GeneratedData( + DataTypes.GEOGRAPHY(), Pair.of("geog-1".getBytes(), "geog-2".getBytes()))); TEST_DATA.put( DataTypeRoot.DECIMAL, new GeneratedData( diff --git a/paimon-common/src/main/java/org/apache/paimon/data/BinaryArray.java b/paimon-common/src/main/java/org/apache/paimon/data/BinaryArray.java index 5ce7b779f025..7a62523711e1 100644 --- a/paimon-common/src/main/java/org/apache/paimon/data/BinaryArray.java +++ b/paimon-common/src/main/java/org/apache/paimon/data/BinaryArray.java @@ -75,6 +75,8 @@ public static int calculateFixLengthPartSize(DataType type) { case VARCHAR: case BINARY: case VARBINARY: + case GEOMETRY: + case GEOGRAPHY: case DECIMAL: case BIGINT: case DOUBLE: diff --git a/paimon-common/src/main/java/org/apache/paimon/data/BinaryArrayWriter.java b/paimon-common/src/main/java/org/apache/paimon/data/BinaryArrayWriter.java index 58f98dc18933..9b6accdfea23 100644 --- a/paimon-common/src/main/java/org/apache/paimon/data/BinaryArrayWriter.java +++ b/paimon-common/src/main/java/org/apache/paimon/data/BinaryArrayWriter.java @@ -229,6 +229,8 @@ public static NullSetter createNullSetter(DataType elementType) { case VARCHAR: case BINARY: case VARBINARY: + case GEOMETRY: + case GEOGRAPHY: case DECIMAL: case BIGINT: case TIMESTAMP_WITHOUT_TIME_ZONE: diff --git a/paimon-common/src/main/java/org/apache/paimon/data/BinaryWriter.java b/paimon-common/src/main/java/org/apache/paimon/data/BinaryWriter.java index 2e0cd5701b71..47960aa3691b 100644 --- a/paimon-common/src/main/java/org/apache/paimon/data/BinaryWriter.java +++ b/paimon-common/src/main/java/org/apache/paimon/data/BinaryWriter.java @@ -148,6 +148,8 @@ static void write( break; case BINARY: case VARBINARY: + case GEOMETRY: + case GEOGRAPHY: byte[] bytes = (byte[]) o; writer.writeBinary(pos, bytes, 0, bytes.length); break; @@ -181,6 +183,8 @@ static ValueSetter createValueSetter(DataType elementType, Serializer seriali return (writer, pos, value) -> writer.writeBoolean(pos, (boolean) value); case BINARY: case VARBINARY: + case GEOMETRY: + case GEOGRAPHY: return (writer, pos, value) -> { byte[] bytes = (byte[]) value; writer.writeBinary(pos, bytes, 0, bytes.length); diff --git a/paimon-common/src/main/java/org/apache/paimon/data/InternalArray.java b/paimon-common/src/main/java/org/apache/paimon/data/InternalArray.java index 41640f3771af..78382e8bbd23 100644 --- a/paimon-common/src/main/java/org/apache/paimon/data/InternalArray.java +++ b/paimon-common/src/main/java/org/apache/paimon/data/InternalArray.java @@ -86,6 +86,8 @@ static ElementGetter createElementGetter(DataType elementType) { break; case BINARY: case VARBINARY: + case GEOMETRY: + case GEOGRAPHY: elementGetter = InternalArray::getBinary; break; case DECIMAL: diff --git a/paimon-common/src/main/java/org/apache/paimon/data/InternalRow.java b/paimon-common/src/main/java/org/apache/paimon/data/InternalRow.java index e4aa7e335cd6..445d8854a19a 100644 --- a/paimon-common/src/main/java/org/apache/paimon/data/InternalRow.java +++ b/paimon-common/src/main/java/org/apache/paimon/data/InternalRow.java @@ -126,6 +126,8 @@ static Class getDataClass(DataType type) { return Boolean.class; case BINARY: case VARBINARY: + case GEOMETRY: + case GEOGRAPHY: return byte[].class; case DECIMAL: return Decimal.class; @@ -180,6 +182,8 @@ static FieldGetter createFieldGetter(DataType fieldType, int fieldPos) { break; case BINARY: case VARBINARY: + case GEOMETRY: + case GEOGRAPHY: fieldGetter = row -> row.getBinary(fieldPos); break; case DECIMAL: diff --git a/paimon-common/src/main/java/org/apache/paimon/data/columnar/ColumnVectorUtils.java b/paimon-common/src/main/java/org/apache/paimon/data/columnar/ColumnVectorUtils.java index 716d9a582e0e..ee21025b9672 100644 --- a/paimon-common/src/main/java/org/apache/paimon/data/columnar/ColumnVectorUtils.java +++ b/paimon-common/src/main/java/org/apache/paimon/data/columnar/ColumnVectorUtils.java @@ -97,6 +97,8 @@ private static WritableColumnVector createWritableColumnVector( case VARCHAR: case BINARY: case VARBINARY: + case GEOMETRY: + case GEOGRAPHY: case BLOB: return new HeapBytesVector(capacity); case DECIMAL: diff --git a/paimon-common/src/main/java/org/apache/paimon/data/columnar/RowToColumnConverter.java b/paimon-common/src/main/java/org/apache/paimon/data/columnar/RowToColumnConverter.java index e35c7d77b1f6..3e354c4821da 100644 --- a/paimon-common/src/main/java/org/apache/paimon/data/columnar/RowToColumnConverter.java +++ b/paimon-common/src/main/java/org/apache/paimon/data/columnar/RowToColumnConverter.java @@ -55,6 +55,8 @@ import org.apache.paimon.types.DecimalType; import org.apache.paimon.types.DoubleType; import org.apache.paimon.types.FloatType; +import org.apache.paimon.types.GeographyType; +import org.apache.paimon.types.GeometryType; import org.apache.paimon.types.IntType; import org.apache.paimon.types.LocalZonedTimestampType; import org.apache.paimon.types.MapType; @@ -367,6 +369,16 @@ public TypeConverter visit(VarBinaryType varBinaryType) { return binaryConverter(varBinaryType.isNullable()); } + @Override + public TypeConverter visit(GeometryType geometryType) { + return binaryConverter(geometryType.isNullable()); + } + + @Override + public TypeConverter visit(GeographyType geographyType) { + return binaryConverter(geographyType.isNullable()); + } + @Override public TypeConverter visit(DecimalType decimalType) { return createConverter( diff --git a/paimon-common/src/main/java/org/apache/paimon/data/serializer/InternalSerializers.java b/paimon-common/src/main/java/org/apache/paimon/data/serializer/InternalSerializers.java index 6669f347ff27..fb40d944854e 100644 --- a/paimon-common/src/main/java/org/apache/paimon/data/serializer/InternalSerializers.java +++ b/paimon-common/src/main/java/org/apache/paimon/data/serializer/InternalSerializers.java @@ -54,6 +54,8 @@ private static Serializer createInternal(DataType type) { return BooleanSerializer.INSTANCE; case BINARY: case VARBINARY: + case GEOMETRY: + case GEOGRAPHY: return BinarySerializer.INSTANCE; case DECIMAL: return new DecimalSerializer(getPrecision(type), getScale(type)); diff --git a/paimon-common/src/main/java/org/apache/paimon/data/serializer/RowCompactedSerializer.java b/paimon-common/src/main/java/org/apache/paimon/data/serializer/RowCompactedSerializer.java index c7fb58417afa..baa4d0432564 100644 --- a/paimon-common/src/main/java/org/apache/paimon/data/serializer/RowCompactedSerializer.java +++ b/paimon-common/src/main/java/org/apache/paimon/data/serializer/RowCompactedSerializer.java @@ -185,6 +185,8 @@ private static FieldWriter createFieldWriter(DataType fieldType) { break; case BINARY: case VARBINARY: + case GEOMETRY: + case GEOGRAPHY: fieldWriter = (writer, pos, value) -> writer.writeBinary((byte[]) value); break; case DECIMAL: @@ -301,6 +303,8 @@ private static FieldReader createFieldReader(DataType fieldType) { break; case BINARY: case VARBINARY: + case GEOMETRY: + case GEOGRAPHY: fieldReader = (reader, pos) -> reader.readBinary(); break; case DECIMAL: diff --git a/paimon-common/src/main/java/org/apache/paimon/format/SimpleStatsCollector.java b/paimon-common/src/main/java/org/apache/paimon/format/SimpleStatsCollector.java index b446e4fb2228..96c10d78a028 100644 --- a/paimon-common/src/main/java/org/apache/paimon/format/SimpleStatsCollector.java +++ b/paimon-common/src/main/java/org/apache/paimon/format/SimpleStatsCollector.java @@ -21,6 +21,7 @@ import org.apache.paimon.data.InternalRow; import org.apache.paimon.data.serializer.InternalSerializers; import org.apache.paimon.data.serializer.Serializer; +import org.apache.paimon.statistics.CountsSimpleColStatsCollector; import org.apache.paimon.statistics.NoneSimpleColStatsCollector; import org.apache.paimon.statistics.SimpleColStatsCollector; import org.apache.paimon.types.RowType; @@ -52,6 +53,18 @@ public SimpleStatsCollector( numFields, collectorFactory.length); this.statsCollectors = SimpleColStatsCollector.create(collectorFactory); + for (int i = 0; i < numFields; i++) { + switch (rowType.getTypeAt(i).getTypeRoot()) { + case GEOMETRY: + case GEOGRAPHY: + if (!(statsCollectors[i] instanceof NoneSimpleColStatsCollector)) { + statsCollectors[i] = new CountsSimpleColStatsCollector(); + } + break; + default: + // Keep the configured collector for non-geospatial fields. + } + } this.converter = new RowDataToObjectArrayConverter(rowType); this.fieldSerializers = new Serializer[numFields]; for (int i = 0; i < numFields; i++) { diff --git a/paimon-common/src/main/java/org/apache/paimon/types/InternalRowToSizeVisitor.java b/paimon-common/src/main/java/org/apache/paimon/types/InternalRowToSizeVisitor.java index dbac55a07dde..8487100c8f4c 100644 --- a/paimon-common/src/main/java/org/apache/paimon/types/InternalRowToSizeVisitor.java +++ b/paimon-common/src/main/java/org/apache/paimon/types/InternalRowToSizeVisitor.java @@ -88,6 +88,20 @@ public BiFunction visit(VarBinaryType varBinaryTy }; } + @Override + public BiFunction visit(GeometryType geometryType) { + return binarySize(); + } + + @Override + public BiFunction visit(GeographyType geographyType) { + return binarySize(); + } + + private BiFunction binarySize() { + return (row, index) -> row.isNullAt(index) ? NULL_SIZE : row.getBinary(index).length; + } + @Override public BiFunction visit(DecimalType decimalType) { return (row, index) -> { diff --git a/paimon-common/src/main/java/org/apache/paimon/utils/InternalRowUtils.java b/paimon-common/src/main/java/org/apache/paimon/utils/InternalRowUtils.java index 8fb0b8f8050d..f5d0e87934d5 100644 --- a/paimon-common/src/main/java/org/apache/paimon/utils/InternalRowUtils.java +++ b/paimon-common/src/main/java/org/apache/paimon/utils/InternalRowUtils.java @@ -381,6 +381,8 @@ public static Object get(DataGetters dataGetters, int pos, DataType fieldType) { return dataGetters.getRow(pos, ((RowType) fieldType).getFieldCount()); case BINARY: case VARBINARY: + case GEOMETRY: + case GEOGRAPHY: return dataGetters.getBinary(pos); case VARIANT: return dataGetters.getVariant(pos); diff --git a/paimon-common/src/main/java/org/apache/paimon/utils/TypeCheckUtils.java b/paimon-common/src/main/java/org/apache/paimon/utils/TypeCheckUtils.java index c1520be34107..138c2ca9125b 100644 --- a/paimon-common/src/main/java/org/apache/paimon/utils/TypeCheckUtils.java +++ b/paimon-common/src/main/java/org/apache/paimon/utils/TypeCheckUtils.java @@ -26,6 +26,8 @@ import static org.apache.paimon.types.DataTypeRoot.BLOB; import static org.apache.paimon.types.DataTypeRoot.BOOLEAN; import static org.apache.paimon.types.DataTypeRoot.DECIMAL; +import static org.apache.paimon.types.DataTypeRoot.GEOGRAPHY; +import static org.apache.paimon.types.DataTypeRoot.GEOMETRY; import static org.apache.paimon.types.DataTypeRoot.INTEGER; import static org.apache.paimon.types.DataTypeRoot.MAP; import static org.apache.paimon.types.DataTypeRoot.MULTISET; @@ -110,6 +112,14 @@ public static boolean isBlob(DataType type) { return type.getTypeRoot() == BLOB; } + public static boolean isGeometry(DataType type) { + return type.getTypeRoot() == GEOMETRY; + } + + public static boolean isGeography(DataType type) { + return type.getTypeRoot() == GEOGRAPHY; + } + public static boolean isComparable(DataType type) { return !isMap(type) && !isMultiset(type) @@ -117,7 +127,9 @@ public static boolean isComparable(DataType type) { && !isArray(type) && !isVector(type) && !isVariant(type) - && !isBlob(type); + && !isBlob(type) + && !isGeometry(type) + && !isGeography(type); } public static boolean isMutable(DataType type) { diff --git a/paimon-common/src/main/java/org/apache/paimon/utils/VectorMappingUtils.java b/paimon-common/src/main/java/org/apache/paimon/utils/VectorMappingUtils.java index 54007caaf828..b12fda4f7d13 100644 --- a/paimon-common/src/main/java/org/apache/paimon/utils/VectorMappingUtils.java +++ b/paimon-common/src/main/java/org/apache/paimon/utils/VectorMappingUtils.java @@ -55,6 +55,8 @@ import org.apache.paimon.types.DecimalType; import org.apache.paimon.types.DoubleType; import org.apache.paimon.types.FloatType; +import org.apache.paimon.types.GeographyType; +import org.apache.paimon.types.GeometryType; import org.apache.paimon.types.IntType; import org.apache.paimon.types.LocalZonedTimestampType; import org.apache.paimon.types.MapType; @@ -162,6 +164,16 @@ public ColumnVector visit(VarBinaryType varBinaryType) { return bytesColumnVector(); } + @Override + public ColumnVector visit(GeometryType geometryType) { + return bytesColumnVector(); + } + + @Override + public ColumnVector visit(GeographyType geographyType) { + return bytesColumnVector(); + } + @Override public ColumnVector visit(DecimalType decimalType) { return new DecimalColumnVector() { diff --git a/paimon-common/src/test/java/org/apache/paimon/format/GeospatialStatsCollectorTest.java b/paimon-common/src/test/java/org/apache/paimon/format/GeospatialStatsCollectorTest.java new file mode 100644 index 000000000000..53aa2f8e01f2 --- /dev/null +++ b/paimon-common/src/test/java/org/apache/paimon/format/GeospatialStatsCollectorTest.java @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.format; + +import org.apache.paimon.data.GenericRow; +import org.apache.paimon.types.DataTypes; +import org.apache.paimon.types.RowType; + +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; + +class GeospatialStatsCollectorTest { + + @Test + void testWkbDoesNotProduceLexicographicBounds() { + SimpleStatsCollector collector = + new SimpleStatsCollector(RowType.of(DataTypes.GEOMETRY(), DataTypes.GEOGRAPHY())); + + collector.collect(GenericRow.of(new byte[] {2}, new byte[] {9})); + collector.collect(GenericRow.of(new byte[] {1}, null)); + + SimpleColStats[] stats = collector.extract(); + assertThat(stats[0].min()).isNull(); + assertThat(stats[0].max()).isNull(); + assertThat(stats[0].nullCount()).isZero(); + assertThat(stats[1].min()).isNull(); + assertThat(stats[1].max()).isNull(); + assertThat(stats[1].nullCount()).isOne(); + } +} diff --git a/paimon-core/src/main/java/org/apache/paimon/iceberg/IcebergCommitCallback.java b/paimon-core/src/main/java/org/apache/paimon/iceberg/IcebergCommitCallback.java index cfeb6acb1c7d..af6c86a9ab0b 100644 --- a/paimon-core/src/main/java/org/apache/paimon/iceberg/IcebergCommitCallback.java +++ b/paimon-core/src/main/java/org/apache/paimon/iceberg/IcebergCommitCallback.java @@ -53,6 +53,7 @@ import org.apache.paimon.options.Options; import org.apache.paimon.partition.PartitionPredicate; import org.apache.paimon.schema.SchemaManager; +import org.apache.paimon.schema.SchemaValidation; import org.apache.paimon.schema.TableSchema; import org.apache.paimon.table.FileStoreTable; import org.apache.paimon.table.sink.CommitCallback; @@ -1967,6 +1968,8 @@ private IcebergSchema get(long schemaId) { TableSchema schema = schemaManager.schema(id); // backstop: reject variant on each schema as it is emitted checkVariantNotPublishable(schema.logicalRowType()); + SchemaValidation.validateIcebergGeospatialTypes( + schema.logicalRowType(), table.coreOptions()); return IcebergSchema.create(schema); }); } diff --git a/paimon-core/src/main/java/org/apache/paimon/iceberg/manifest/IcebergDataFileMeta.java b/paimon-core/src/main/java/org/apache/paimon/iceberg/manifest/IcebergDataFileMeta.java index 7334f7528af7..950da63b9d11 100644 --- a/paimon-core/src/main/java/org/apache/paimon/iceberg/manifest/IcebergDataFileMeta.java +++ b/paimon-core/src/main/java/org/apache/paimon/iceberg/manifest/IcebergDataFileMeta.java @@ -187,7 +187,9 @@ public static IcebergDataFileMeta create( || typeRoot == DataTypeRoot.MULTISET || typeRoot == DataTypeRoot.VARIANT || typeRoot == DataTypeRoot.VECTOR - || typeRoot == DataTypeRoot.BLOB) { + || typeRoot == DataTypeRoot.BLOB + || typeRoot == DataTypeRoot.GEOMETRY + || typeRoot == DataTypeRoot.GEOGRAPHY) { continue; } diff --git a/paimon-core/src/main/java/org/apache/paimon/iceberg/metadata/IcebergDataField.java b/paimon-core/src/main/java/org/apache/paimon/iceberg/metadata/IcebergDataField.java index 9862ff7f90c4..d69fef3fb3ef 100644 --- a/paimon-core/src/main/java/org/apache/paimon/iceberg/metadata/IcebergDataField.java +++ b/paimon-core/src/main/java/org/apache/paimon/iceberg/metadata/IcebergDataField.java @@ -28,7 +28,10 @@ import org.apache.paimon.types.DateType; import org.apache.paimon.types.DecimalType; import org.apache.paimon.types.DoubleType; +import org.apache.paimon.types.EdgeAlgorithm; import org.apache.paimon.types.FloatType; +import org.apache.paimon.types.GeographyType; +import org.apache.paimon.types.GeometryType; import org.apache.paimon.types.IntType; import org.apache.paimon.types.LocalZonedTimestampType; import org.apache.paimon.types.MapType; @@ -196,6 +199,16 @@ private static Object toTypeObject(DataType dataType, int fieldId, int depth) { return timestampLtzPrecision >= 7 ? "timestamptz_ns" : "timestamptz"; case VARIANT: return "variant"; + case GEOMETRY: + return String.format("geometry(%s)", ((GeometryType) dataType).getCrs()); + case GEOGRAPHY: + GeographyType geographyType = (GeographyType) dataType; + Preconditions.checkArgument( + !geographyType.getCrs().contains(","), + "Geography CRS '%s' cannot contain ',' in Iceberg metadata.", + geographyType.getCrs()); + return String.format( + "geography(%s, %s)", geographyType.getCrs(), geographyType.getAlgorithm()); case ARRAY: ArrayType arrayType = (ArrayType) dataType; return new IcebergListType( @@ -290,6 +303,20 @@ private DataType getDataTypeFromType(Object icebergType, boolean isRequired) { return new LocalZonedTimestampType(!isRequired, 9); case "variant": // iceberg v3 format return new VariantType(!isRequired); + case "geometry": // iceberg v3 format + return new GeometryType(!isRequired, geometryParameter(simpleType)); + case "geography": // iceberg v3 format + String[] parameters = geographyParameters(simpleType); + Preconditions.checkArgument( + parameters.length == 1 || parameters.length == 2, + "Invalid Iceberg geography type: %s", + simpleType); + return new GeographyType( + !isRequired, + parameters[0], + parameters.length == 1 + ? GeographyType.DEFAULT_ALGORITHM + : EdgeAlgorithm.fromName(parameters[1])); default: throw new UnsupportedOperationException( "Unsupported primitive data type: " + icebergType); @@ -321,6 +348,37 @@ private DataType getDataTypeFromType(Object icebergType, boolean isRequired) { } } + private static String geometryParameter(String simpleType) { + if ("geometry".equals(simpleType)) { + return GeometryType.DEFAULT_CRS; + } + int start = simpleType.indexOf('('); + Preconditions.checkArgument( + start >= 0 && simpleType.endsWith(")"), + "Invalid Iceberg geometry type: %s", + simpleType); + return simpleType.substring(start + 1, simpleType.length() - 1).trim(); + } + + private static String[] geographyParameters(String simpleType) { + int start = simpleType.indexOf('('); + if (start < 0) { + if ("geography".equals(simpleType)) { + return new String[] {GeometryType.DEFAULT_CRS, EdgeAlgorithm.SPHERICAL.toString()}; + } + } + Preconditions.checkArgument( + start >= 0 && simpleType.endsWith(")"), + "Invalid Iceberg geospatial type: %s", + simpleType); + String parameters = simpleType.substring(start + 1, simpleType.length() - 1); + String[] result = parameters.split(",", -1); + for (int i = 0; i < result.length; i++) { + result[i] = result[i].trim(); + } + return result; + } + public DataField toDatafield() { return new DataField(id, name, dataType(), doc); } diff --git a/paimon-core/src/main/java/org/apache/paimon/schema/SchemaManager.java b/paimon-core/src/main/java/org/apache/paimon/schema/SchemaManager.java index bcb4a59b4f00..28175245af11 100644 --- a/paimon-core/src/main/java/org/apache/paimon/schema/SchemaManager.java +++ b/paimon-core/src/main/java/org/apache/paimon/schema/SchemaManager.java @@ -25,6 +25,7 @@ import org.apache.paimon.catalog.Identifier; import org.apache.paimon.fs.FileIO; import org.apache.paimon.fs.Path; +import org.apache.paimon.iceberg.IcebergOptions; import org.apache.paimon.schema.ColumnDirectiveUtils.ConvertedColumn; import org.apache.paimon.schema.SchemaChange.AddColumn; import org.apache.paimon.schema.SchemaChange.DropColumn; @@ -1202,11 +1203,25 @@ protected void updateLastColumn(int depth, List newFields, String fie @VisibleForTesting public boolean commit(TableSchema newSchema) throws Exception { SchemaValidation.validateTableSchema(newSchema); + validateHistoricalIcebergGeospatialTypes(newSchema); SchemaValidation.validateFallbackBranch(this, newSchema); Path schemaPath = toSchemaPath(newSchema.id()); return fileIO.tryToWriteAtomic(schemaPath, newSchema.toString()); } + private void validateHistoricalIcebergGeospatialTypes(TableSchema newSchema) { + CoreOptions options = new CoreOptions(newSchema.options()); + IcebergOptions.StorageType storage = + options.toConfiguration().get(IcebergOptions.METADATA_ICEBERG_STORAGE); + if (storage == IcebergOptions.StorageType.DISABLED) { + return; + } + + for (TableSchema schema : listAll()) { + SchemaValidation.validateIcebergGeospatialTypes(schema.logicalRowType(), options); + } + } + /** Read schema for schema id. */ public TableSchema schema(long id) { return fromPath(fileIO, toSchemaPath(id)); diff --git a/paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java b/paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java index 607f5d20fa15..1fe54ea47905 100644 --- a/paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java +++ b/paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java @@ -32,6 +32,7 @@ import org.apache.paimon.globalindex.GlobalIndexer; import org.apache.paimon.globalindex.bitmap.BitmapGlobalIndexerFactory; import org.apache.paimon.globalindex.btree.BTreeGlobalIndexerFactory; +import org.apache.paimon.iceberg.IcebergOptions; import org.apache.paimon.mergetree.compact.aggregate.FieldAggregator; import org.apache.paimon.mergetree.compact.aggregate.factory.FieldAggregatorFactory; import org.apache.paimon.mergetree.compact.aggregate.factory.FieldLastValueAggFactory; @@ -43,6 +44,8 @@ import org.apache.paimon.types.DataField; import org.apache.paimon.types.DataType; import org.apache.paimon.types.DataTypeRoot; +import org.apache.paimon.types.GeographyType; +import org.apache.paimon.types.GeometryType; import org.apache.paimon.types.IntType; import org.apache.paimon.types.LocalZonedTimestampType; import org.apache.paimon.types.MapType; @@ -125,7 +128,9 @@ public class SchemaValidation { RowType.class, MultisetType.class, VectorType.class, - VariantType.class); + VariantType.class, + GeometryType.class, + GeographyType.class); /** * Validate the {@link TableSchema} and {@link CoreOptions}. @@ -226,6 +231,7 @@ public static void validateTableSchema(TableSchema schema, Set dynamicOp FileFormat fileFormat = FileFormat.fromIdentifier(options.formatType(), new Options(schema.options())); RowType tableRowType = new RowType(schema.fields()); + validateGeospatialTypes(schema, options, tableRowType); validateBlobFields(tableRowType, options); Set blobDescriptorFields = validateBlobDescriptorFields(tableRowType, options); Set blobViewFields = @@ -449,6 +455,106 @@ private static void validateOnlyContainPrimitiveType( } } + private static void validateGeospatialTypes( + TableSchema schema, CoreOptions options, RowType rowType) { + boolean hasGeospatial = + containsType( + rowType, + type -> type.isAnyOf(DataTypeRoot.GEOMETRY, DataTypeRoot.GEOGRAPHY)); + if (!hasGeospatial) { + return; + } + + checkArgument( + CoreOptions.FILE_FORMAT_PARQUET.equals(options.formatType()), + "Geometry and geography columns require '%s'='parquet', but was '%s'.", + CoreOptions.FILE_FORMAT.key(), + options.formatType()); + options.fileFormatPerLevel() + .forEach( + (level, format) -> + checkArgument( + CoreOptions.FILE_FORMAT_PARQUET.equals(format), + "Geometry and geography columns require parquet at every level, but '%s' contains '%s:%s'.", + CoreOptions.FILE_FORMAT_PER_LEVEL.key(), + level, + format)); + checkArgument( + options.changelogFileFormat() == null + || CoreOptions.FILE_FORMAT_PARQUET.equals(options.changelogFileFormat()), + "Geometry and geography columns require '%s' to be parquet, but was '%s'.", + CoreOptions.CHANGELOG_FILE_FORMAT.key(), + options.changelogFileFormat()); + validateIcebergGeospatialTypes(rowType, options); + + List geospatialClusteringColumns = + schema.fields().stream() + .filter(field -> options.clusteringColumns().contains(field.name())) + .filter( + field -> + containsType( + field.type(), + type -> + type.isAnyOf( + DataTypeRoot.GEOMETRY, + DataTypeRoot.GEOGRAPHY))) + .map(DataField::name) + .collect(Collectors.toList()); + checkArgument( + geospatialClusteringColumns.isEmpty(), + "Geometry and geography columns cannot be clustering columns: %s.", + geospatialClusteringColumns); + + Set geospatialFields = + schema.fields().stream() + .filter( + field -> + field.type() + .isAnyOf( + DataTypeRoot.GEOMETRY, + DataTypeRoot.GEOGRAPHY)) + .map(DataField::name) + .collect(Collectors.toSet()); + Set geospatialBucketKeys = new HashSet<>(schema.bucketKeys()); + geospatialBucketKeys.retainAll(geospatialFields); + checkArgument( + geospatialBucketKeys.isEmpty(), + "Geometry and geography columns cannot be bucket keys: %s.", + geospatialBucketKeys); + Set geospatialSequenceFields = new HashSet<>(options.sequenceField()); + geospatialSequenceFields.retainAll(geospatialFields); + checkArgument( + geospatialSequenceFields.isEmpty(), + "Geometry and geography columns cannot be sequence fields: %s.", + geospatialSequenceFields); + } + + /** Validate geospatial types in a schema that will be published as Iceberg metadata. */ + public static void validateIcebergGeospatialTypes(DataType dataType, CoreOptions options) { + boolean hasGeospatial = + containsType( + dataType, + type -> type.isAnyOf(DataTypeRoot.GEOMETRY, DataTypeRoot.GEOGRAPHY)); + if (!hasGeospatial) { + return; + } + + IcebergOptions.StorageType icebergStorage = + options.toConfiguration().get(IcebergOptions.METADATA_ICEBERG_STORAGE); + if (icebergStorage != IcebergOptions.StorageType.DISABLED) { + checkArgument( + options.toConfiguration().get(IcebergOptions.FORMAT_VERSION) == 3, + "Geometry and geography columns require '%s'='3' when Iceberg metadata is enabled.", + IcebergOptions.FORMAT_VERSION.key()); + checkArgument( + icebergStorage != IcebergOptions.StorageType.REST_CATALOG, + "Geometry and geography columns do not support '%s'='%s' because the bundled Iceberg REST client cannot parse Iceberg v3 geospatial types.", + IcebergOptions.METADATA_ICEBERG_STORAGE.key(), + IcebergOptions.StorageType.REST_CATALOG); + validateIcebergGeographyCrs(dataType); + } + } + private static void validateStartupMode(CoreOptions options) { if (options.startupMode() == CoreOptions.StartupMode.FROM_TIMESTAMP) { checkExactOneOptionExistInMode( @@ -812,6 +918,30 @@ private static boolean containsType(DataType dataType, Predicate predi return false; } + private static void validateIcebergGeographyCrs(DataType dataType) { + if (dataType.is(DataTypeRoot.GEOGRAPHY)) { + String crs = ((GeographyType) dataType).getCrs(); + checkArgument( + !crs.contains(","), + "Geography CRS '%s' cannot contain ',' when Iceberg metadata is enabled.", + crs); + } else if (dataType instanceof RowType) { + for (DataField field : ((RowType) dataType).getFields()) { + validateIcebergGeographyCrs(field.type()); + } + } else if (dataType instanceof ArrayType) { + validateIcebergGeographyCrs(((ArrayType) dataType).getElementType()); + } else if (dataType instanceof MultisetType) { + validateIcebergGeographyCrs(((MultisetType) dataType).getElementType()); + } else if (dataType instanceof MapType) { + MapType mapType = (MapType) dataType; + validateIcebergGeographyCrs(mapType.getKeyType()); + validateIcebergGeographyCrs(mapType.getValueType()); + } else if (dataType instanceof VectorType) { + validateIcebergGeographyCrs(((VectorType) dataType).getElementType()); + } + } + private static void validateMapSharedShreddingFileFormats(CoreOptions options) { validateMapSharedShreddingFileFormat( CoreOptions.FILE_FORMAT.key(), options.fileFormatString()); diff --git a/paimon-core/src/test/java/org/apache/paimon/iceberg/manifest/IcebergDataFileMetaTest.java b/paimon-core/src/test/java/org/apache/paimon/iceberg/manifest/IcebergDataFileMetaTest.java index c4888312b053..b0b7154b18d2 100644 --- a/paimon-core/src/test/java/org/apache/paimon/iceberg/manifest/IcebergDataFileMetaTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/iceberg/manifest/IcebergDataFileMetaTest.java @@ -228,4 +228,51 @@ void testRequiredNestedFieldSkipped() { assertThat((byte[]) ((GenericMap) meta.upperBounds()).get(1)) .isEqualTo(new byte[] {5, 0, 0, 0}); } + + @Test + @DisplayName("Test geospatial fields publish null counts but not WKB bounds") + void testGeospatialBoundsSkipped() { + IcebergSchema icebergSchema = + new IcebergSchema( + 0, + Arrays.asList( + new IcebergDataField( + new DataField(1, "geom", DataTypes.GEOMETRY())), + new IcebergDataField( + new DataField(2, "geog", DataTypes.GEOGRAPHY())))); + + byte[] wkbPoint = + new byte[] { + 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, (byte) 0xf0, 0x3f, 0, 0, 0, 0, 0, 0, 0x40 + }; + BinaryRow values = new BinaryRow(2); + BinaryRowWriter rowWriter = new BinaryRowWriter(values); + rowWriter.writeBinary(0, wkbPoint, 0, wkbPoint.length); + rowWriter.writeBinary(1, wkbPoint, 0, wkbPoint.length); + rowWriter.complete(); + + BinaryArray nullCounts = new BinaryArray(); + BinaryArrayWriter arrayWriter = new BinaryArrayWriter(nullCounts, 2, 8); + arrayWriter.writeLong(0, 1L); + arrayWriter.writeLong(1, 2L); + arrayWriter.complete(); + + IcebergDataFileMeta meta = + IcebergDataFileMeta.create( + IcebergDataFileMeta.Content.DATA, + "path", + "parquet", + BinaryRow.EMPTY_ROW, + 10, + 100, + icebergSchema, + new SimpleStats(values, values, nullCounts), + null); + + assertThat(meta.nullValueCounts().size()).isEqualTo(2); + assertThat(((GenericMap) meta.nullValueCounts()).get(1)).isEqualTo(1L); + assertThat(((GenericMap) meta.nullValueCounts()).get(2)).isEqualTo(2L); + assertThat(meta.lowerBounds().size()).isZero(); + assertThat(meta.upperBounds().size()).isZero(); + } } diff --git a/paimon-core/src/test/java/org/apache/paimon/iceberg/metadata/IcebergDataFieldTest.java b/paimon-core/src/test/java/org/apache/paimon/iceberg/metadata/IcebergDataFieldTest.java index 0cfd5f0fe5fc..f1daf206f421 100644 --- a/paimon-core/src/test/java/org/apache/paimon/iceberg/metadata/IcebergDataFieldTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/iceberg/metadata/IcebergDataFieldTest.java @@ -28,7 +28,10 @@ import org.apache.paimon.types.DateType; import org.apache.paimon.types.DecimalType; import org.apache.paimon.types.DoubleType; +import org.apache.paimon.types.EdgeAlgorithm; import org.apache.paimon.types.FloatType; +import org.apache.paimon.types.GeographyType; +import org.apache.paimon.types.GeometryType; import org.apache.paimon.types.IntType; import org.apache.paimon.types.LocalZonedTimestampType; import org.apache.paimon.types.MapType; @@ -153,6 +156,53 @@ void testPrimitiveTypeConversions() { assertThat(icebergVarBinary.type()).isEqualTo("binary"); } + @Test + @DisplayName("Test Iceberg v3 geospatial type conversions") + void testGeospatialTypeConversions() { + IcebergDataField geometry = + new IcebergDataField( + new DataField(1, "geom", new GeometryType(false, "EPSG:3857"))); + assertThat(geometry.type()).isEqualTo("geometry(EPSG:3857)"); + + IcebergDataField geography = + new IcebergDataField( + new DataField( + 2, + "geog", + new GeographyType(true, "OGC:CRS84", EdgeAlgorithm.KARNEY))); + assertThat(geography.type()).isEqualTo("geography(OGC:CRS84, karney)"); + + assertThat(new IcebergDataField(3, "geom", true, "geometry(EPSG:3857)", null).dataType()) + .isEqualTo(new GeometryType(false, "EPSG:3857")); + assertThat( + new IcebergDataField( + 4, "geog", false, "geography(OGC:CRS84, vincenty)", null) + .dataType()) + .isEqualTo(new GeographyType(true, "OGC:CRS84", EdgeAlgorithm.VINCENTY)); + + assertThat(new IcebergDataField(5, "geom", false, "geometry", null).dataType()) + .isEqualTo(new GeometryType()); + assertThat(new IcebergDataField(6, "geog", false, "geography", null).dataType()) + .isEqualTo(new GeographyType()); + assertThat(new IcebergDataField(7, "geog", false, "geography(EPSG:4326)", null).dataType()) + .isEqualTo(new GeographyType(true, "EPSG:4326", GeographyType.DEFAULT_ALGORITHM)); + assertThat( + new IcebergDataField(8, "geom", false, "geometry(custom, definition)", null) + .dataType()) + .isEqualTo(new GeometryType("custom, definition")); + + assertThatThrownBy( + () -> + new IcebergDataField( + new DataField( + 9, + "geog", + new GeographyType("custom, definition")))) + .hasMessageContaining("Geography CRS") + .hasMessageContaining("custom, definition") + .hasMessageContaining("Iceberg metadata"); + } + @Test @DisplayName("Test decimal type conversion") void testDecimalTypeConversion() { diff --git a/paimon-core/src/test/java/org/apache/paimon/mergetree/compact/aggregate/FieldAggregatorTest.java b/paimon-core/src/test/java/org/apache/paimon/mergetree/compact/aggregate/FieldAggregatorTest.java index 90ade26f7129..ef3603a7a08c 100644 --- a/paimon-core/src/test/java/org/apache/paimon/mergetree/compact/aggregate/FieldAggregatorTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/mergetree/compact/aggregate/FieldAggregatorTest.java @@ -586,7 +586,9 @@ public void testFieldMaxMinAggComparableTypesAreAllSupported() { DataTypeRoot.ARRAY, DataTypeRoot.VECTOR, DataTypeRoot.VARIANT, - DataTypeRoot.BLOB)); + DataTypeRoot.BLOB, + DataTypeRoot.GEOMETRY, + DataTypeRoot.GEOGRAPHY)); assertThat(sampledRoots) .as("a comparable type root must be covered here and in InternalRowUtils.compare") .isEqualTo(expectedRoots); @@ -604,6 +606,8 @@ public void testFieldMaxMinAggWithIncomparableTypeShouldFail() { DataTypes.ROW(DataTypes.FIELD(0, "f0", DataTypes.INT())), DataTypes.VARIANT(), DataTypes.BLOB(), + DataTypes.GEOMETRY(), + DataTypes.GEOGRAPHY(), DataTypes.VECTOR(3, DataTypes.FLOAT()))) { assertThatThrownBy(() -> new FieldMaxAggFactory().create(incomparable, null, "label")) .isInstanceOf(IllegalArgumentException.class) diff --git a/paimon-core/src/test/java/org/apache/paimon/schema/DataTypeJsonParserTest.java b/paimon-core/src/test/java/org/apache/paimon/schema/DataTypeJsonParserTest.java index fc6b6abd682e..369010a0f2ba 100644 --- a/paimon-core/src/test/java/org/apache/paimon/schema/DataTypeJsonParserTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/schema/DataTypeJsonParserTest.java @@ -31,7 +31,10 @@ import org.apache.paimon.types.DateType; import org.apache.paimon.types.DecimalType; import org.apache.paimon.types.DoubleType; +import org.apache.paimon.types.EdgeAlgorithm; import org.apache.paimon.types.FloatType; +import org.apache.paimon.types.GeographyType; +import org.apache.paimon.types.GeometryType; import org.apache.paimon.types.IntType; import org.apache.paimon.types.LocalZonedTimestampType; import org.apache.paimon.types.MapType; @@ -114,6 +117,14 @@ private static Stream testData() { TestSpec.forString("TIMESTAMP_LTZ(3)").expectType(new LocalZonedTimestampType(3)), TestSpec.forString("VARIANT").expectType(new VariantType()), TestSpec.forString("BLOB").expectType(new BlobType()), + TestSpec.forString("GEOMETRY") + .expectType(new GeometryType(GeometryType.DEFAULT_CRS)), + TestSpec.forString("geometry(ogc:crs84) NOT NULL") + .expectType(new GeometryType(false, "OGC:CRS84")), + TestSpec.forString("GEOGRAPHY") + .expectType(new GeographyType(GeographyType.DEFAULT_CRS)), + TestSpec.forString("GEOGRAPHY(EPSG:4326, karney)") + .expectType(new GeographyType("EPSG:4326", EdgeAlgorithm.KARNEY)), TestSpec.forString("VECTOR") .expectType(DataTypes.VECTOR(3, DataTypes.FLOAT())), TestSpec.forString("VECTOR NOT NULL") @@ -198,7 +209,9 @@ private static Stream testData() { TestSpec.forString("VARCHAR(test)").expectErrorMessage(" expected"), TestSpec.forString("VARCHAR(33333333333)") - .expectErrorMessage("Invalid integer value")); + .expectErrorMessage("Invalid integer value"), + TestSpec.forString("GEOGRAPHY(OGC:CRS84, rhumb)") + .expectErrorMessage("Invalid edge interpolation algorithm")); } @ParameterizedTest(name = "{index}: [From: {0}, To: {1}]") diff --git a/paimon-core/src/test/java/org/apache/paimon/schema/SchemaManagerTest.java b/paimon-core/src/test/java/org/apache/paimon/schema/SchemaManagerTest.java index c080d0eef1e3..044513251c2c 100644 --- a/paimon-core/src/test/java/org/apache/paimon/schema/SchemaManagerTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/schema/SchemaManagerTest.java @@ -26,6 +26,7 @@ import org.apache.paimon.fs.FileIOFinder; import org.apache.paimon.fs.Path; import org.apache.paimon.fs.local.LocalFileIO; +import org.apache.paimon.iceberg.IcebergOptions; import org.apache.paimon.reader.RecordReaderIterator; import org.apache.paimon.table.FileStoreTable; import org.apache.paimon.table.FileStoreTableFactory; @@ -172,6 +173,74 @@ public void testUpdateOptions() throws Exception { assertThat(latest.get().options()).containsEntry("new_k", "new_v"); } + @Test + public void testEnableIcebergMetadataValidatesHistoricalGeospatialSchemas() throws Exception { + Map geospatialOptions = new HashMap<>(); + geospatialOptions.put(CoreOptions.BUCKET.key(), "-1"); + Schema geospatialSchema = + new Schema( + Arrays.asList( + new DataField(0, "id", DataTypes.INT()), + new DataField(1, "geom", DataTypes.GEOMETRY())), + Collections.emptyList(), + Collections.emptyList(), + geospatialOptions, + ""); + + retryArtificialException(() -> manager.createTable(geospatialSchema)); + retryArtificialException(() -> manager.commitChanges(SchemaChange.dropColumn("geom"))); + + assertThatThrownBy( + () -> + retryArtificialException( + () -> + manager.commitChanges( + SchemaChange.setOption( + IcebergOptions + .METADATA_ICEBERG_STORAGE + .key(), + "table-location")))) + .hasStackTraceContaining( + "Geometry and geography columns require 'metadata.iceberg.format-version'='3'"); + + assertThatThrownBy( + () -> + retryArtificialException( + () -> + manager.commitChanges( + Arrays.asList( + SchemaChange.setOption( + IcebergOptions + .METADATA_ICEBERG_STORAGE + .key(), + "rest-catalog"), + SchemaChange.setOption( + IcebergOptions + .FORMAT_VERSION + .key(), + "3"))))) + .hasStackTraceContaining( + "Geometry and geography columns do not support 'metadata.iceberg.storage'='rest-catalog'"); + + assertThatCode( + () -> + retryArtificialException( + () -> + manager.commitChanges( + Arrays.asList( + SchemaChange.setOption( + IcebergOptions + .METADATA_ICEBERG_STORAGE + .key(), + "table-location"), + SchemaChange.setOption( + IcebergOptions + .FORMAT_VERSION + .key(), + "3"))))) + .doesNotThrowAnyException(); + } + @Test public void testChangeMapStorageLayoutForExistingField() throws Exception { retryArtificialException(() -> manager.createTable(mapStorageLayoutSchema("default"))); diff --git a/paimon-core/src/test/java/org/apache/paimon/schema/SchemaValidationTest.java b/paimon-core/src/test/java/org/apache/paimon/schema/SchemaValidationTest.java index 3ba69852f33a..5f5aa2fce0eb 100644 --- a/paimon-core/src/test/java/org/apache/paimon/schema/SchemaValidationTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/schema/SchemaValidationTest.java @@ -19,6 +19,7 @@ package org.apache.paimon.schema; import org.apache.paimon.CoreOptions; +import org.apache.paimon.iceberg.IcebergOptions; import org.apache.paimon.table.BucketMode; import org.apache.paimon.types.DataField; import org.apache.paimon.types.DataType; @@ -1621,6 +1622,204 @@ public void testFileFormatPerLevelAcceptsCompatibleSchema() { new TableSchema(1, fields, 10, emptyList(), singletonList("k"), options, "")); } + @Test + public void testGeospatialTypeValidation() { + List fields = + Arrays.asList( + new DataField(0, "id", DataTypes.INT()), + new DataField(1, "geom", DataTypes.GEOMETRY()), + new DataField(2, "geog", DataTypes.GEOGRAPHY())); + + assertThatNoException() + .isThrownBy( + () -> + validateTableSchema( + geospatialSchema( + fields, + emptyList(), + emptyList(), + new HashMap<>()))); + + Map avroOptions = new HashMap<>(); + avroOptions.put(CoreOptions.FILE_FORMAT.key(), "avro"); + assertThatThrownBy( + () -> + validateTableSchema( + geospatialSchema( + fields, emptyList(), emptyList(), avroOptions))) + .hasMessageContaining("require 'file.format'='parquet'"); + + Map perLevelOptions = new HashMap<>(); + perLevelOptions.put(CoreOptions.FILE_FORMAT_PER_LEVEL.key(), "0:orc"); + assertThatThrownBy( + () -> + validateTableSchema( + geospatialSchema( + fields, emptyList(), emptyList(), perLevelOptions))) + .hasMessageContaining("require parquet at every level"); + + Map changelogOptions = new HashMap<>(); + changelogOptions.put(CoreOptions.CHANGELOG_FILE_FORMAT.key(), "orc"); + assertThatThrownBy( + () -> + validateTableSchema( + geospatialSchema( + fields, + emptyList(), + emptyList(), + changelogOptions))) + .hasMessageContaining("require 'changelog-file.format' to be parquet"); + + Map icebergV2Options = new HashMap<>(); + icebergV2Options.put(IcebergOptions.METADATA_ICEBERG_STORAGE.key(), "table-location"); + assertThatThrownBy( + () -> + validateTableSchema( + geospatialSchema( + fields, + emptyList(), + emptyList(), + icebergV2Options))) + .hasMessageContaining("require 'metadata.iceberg.format-version'='3'"); + + icebergV2Options.put(IcebergOptions.FORMAT_VERSION.key(), "3"); + assertThatNoException() + .isThrownBy( + () -> + validateTableSchema( + geospatialSchema( + fields, + emptyList(), + emptyList(), + icebergV2Options))); + + Map icebergRestOptions = new HashMap<>(); + icebergRestOptions.put(IcebergOptions.METADATA_ICEBERG_STORAGE.key(), "rest-catalog"); + icebergRestOptions.put(IcebergOptions.FORMAT_VERSION.key(), "3"); + assertThatThrownBy( + () -> + validateTableSchema( + geospatialSchema( + fields, + emptyList(), + emptyList(), + icebergRestOptions))) + .hasMessageContaining("do not support 'metadata.iceberg.storage'='rest-catalog'") + .hasMessageContaining("REST client"); + + List customCrsFields = + Arrays.asList( + new DataField(0, "id", DataTypes.INT()), + new DataField( + 1, + "geographies", + DataTypes.ARRAY(DataTypes.GEOGRAPHY("custom, definition")))); + assertThatThrownBy( + () -> + validateTableSchema( + geospatialSchema( + customCrsFields, + emptyList(), + emptyList(), + icebergV2Options))) + .hasMessageContaining("Geography CRS") + .hasMessageContaining("custom, definition") + .hasMessageContaining("Iceberg metadata"); + } + + @Test + public void testGeospatialTypeRejectsKeyAndOrderingSemantics() { + List fields = + Arrays.asList( + new DataField(0, "id", DataTypes.INT()), + new DataField(1, "geom", DataTypes.GEOMETRY()), + new DataField(2, "geog", DataTypes.GEOGRAPHY())); + + assertThatThrownBy( + () -> + validateTableSchema( + geospatialSchema( + fields, + singletonList("geom"), + emptyList(), + new HashMap<>()))) + .hasMessage("The type GeometryType in partition field geom is unsupported"); + + assertThatThrownBy( + () -> + validateTableSchema( + geospatialSchema( + fields, + emptyList(), + singletonList("geog"), + new HashMap<>()))) + .hasMessage("The type GeographyType in primary key field geog is unsupported"); + + Map bucketOptions = new HashMap<>(); + bucketOptions.put(CoreOptions.BUCKET_KEY.key(), "geom"); + bucketOptions.put(BUCKET.key(), "1"); + assertThatThrownBy( + () -> + validateTableSchema( + geospatialSchema( + fields, emptyList(), emptyList(), bucketOptions))) + .hasMessage("Geometry and geography columns cannot be bucket keys: [geom]."); + + Map sequenceOptions = new HashMap<>(); + sequenceOptions.put(CoreOptions.SEQUENCE_FIELD.key(), "geog"); + assertThatThrownBy( + () -> + validateTableSchema( + geospatialSchema( + fields, + emptyList(), + singletonList("id"), + sequenceOptions))) + .hasMessage("Geometry and geography columns cannot be sequence fields: [geog]."); + + Map clusteringOptions = new HashMap<>(); + clusteringOptions.put(CoreOptions.CLUSTERING_COLUMNS.key(), "geom"); + assertThatThrownBy( + () -> + validateTableSchema( + geospatialSchema( + fields, + emptyList(), + emptyList(), + clusteringOptions))) + .hasMessage("Geometry and geography columns cannot be clustering columns: [geom]."); + + List nestedFields = + Arrays.asList( + new DataField(0, "id", DataTypes.INT()), + new DataField( + 1, + "nested", + DataTypes.ROW(DataTypes.FIELD(2, "geog", DataTypes.GEOGRAPHY())))); + Map nestedClusteringOptions = new HashMap<>(); + nestedClusteringOptions.put(CoreOptions.CLUSTERING_COLUMNS.key(), "nested"); + assertThatThrownBy( + () -> + validateTableSchema( + geospatialSchema( + nestedFields, + emptyList(), + emptyList(), + nestedClusteringOptions))) + .hasMessage( + "Geometry and geography columns cannot be clustering columns: [nested]."); + } + + private TableSchema geospatialSchema( + List fields, + List partitionKeys, + List primaryKeys, + Map options) { + options.putIfAbsent(BUCKET.key(), "-1"); + return new TableSchema( + 1, fields, 10, partitionKeys, primaryKeys, options, "geospatial test"); + } + @Test void testManifestSortValidation() { List fields = diff --git a/paimon-core/src/test/java/org/apache/paimon/table/GeospatialTypeTableTest.java b/paimon-core/src/test/java/org/apache/paimon/table/GeospatialTypeTableTest.java new file mode 100644 index 000000000000..309b20b902cd --- /dev/null +++ b/paimon-core/src/test/java/org/apache/paimon/table/GeospatialTypeTableTest.java @@ -0,0 +1,114 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.table; + +import org.apache.paimon.CoreOptions; +import org.apache.paimon.data.GenericArray; +import org.apache.paimon.data.GenericRow; +import org.apache.paimon.data.InternalArray; +import org.apache.paimon.data.InternalRow; +import org.apache.paimon.io.DataFileMeta; +import org.apache.paimon.schema.Schema; +import org.apache.paimon.stats.SimpleStats; +import org.apache.paimon.table.source.DataSplit; +import org.apache.paimon.types.DataTypes; +import org.apache.paimon.types.EdgeAlgorithm; + +import org.junit.jupiter.api.Test; + +import java.util.Comparator; +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +/** Tests table read and write with Iceberg-compatible geospatial types. */ +public class GeospatialTypeTableTest extends TableTestBase { + + private static final byte[] POINT_1_2_WKB = + new byte[] { + 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, (byte) 0xf0, 0x3f, 0, 0, 0, 0, 0, 0, 0, 0x40 + }; + + private static final byte[] POINT_3_4_WKB = + new byte[] {1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x08, 0x40, 0, 0, 0, 0, 0, 0, 0x10, 0x40}; + + @Test + public void testReadWriteAndStats() throws Exception { + createTableDefault(); + FileStoreTable table = getTableDefault(); + + assertThat(table.schema().fields().get(1).type()) + .isEqualTo(DataTypes.GEOMETRY("EPSG:3857")); + assertThat(table.schema().fields().get(2).type()) + .isEqualTo(DataTypes.GEOGRAPHY("OGC:CRS84", EdgeAlgorithm.KARNEY)); + + write( + table, + GenericRow.of( + 1, + POINT_1_2_WKB, + POINT_3_4_WKB, + new GenericArray(new Object[] {POINT_1_2_WKB, null, POINT_3_4_WKB})), + GenericRow.of(2, null, POINT_1_2_WKB, new GenericArray(new Object[0])), + GenericRow.of(3, POINT_3_4_WKB, null, null)); + + List rows = read(table); + rows.sort(Comparator.comparingInt(row -> row.getInt(0))); + + assertThat(rows).hasSize(3); + assertThat(rows.get(0).getBinary(1)).isEqualTo(POINT_1_2_WKB); + assertThat(rows.get(0).getBinary(2)).isEqualTo(POINT_3_4_WKB); + InternalArray geometries = rows.get(0).getArray(3); + assertThat(geometries.size()).isEqualTo(3); + assertThat(geometries.getBinary(0)).isEqualTo(POINT_1_2_WKB); + assertThat(geometries.isNullAt(1)).isTrue(); + assertThat(geometries.getBinary(2)).isEqualTo(POINT_3_4_WKB); + + assertThat(rows.get(1).isNullAt(1)).isTrue(); + assertThat(rows.get(1).getBinary(2)).isEqualTo(POINT_1_2_WKB); + assertThat(rows.get(1).getArray(3).size()).isZero(); + assertThat(rows.get(2).getBinary(1)).isEqualTo(POINT_3_4_WKB); + assertThat(rows.get(2).isNullAt(2)).isTrue(); + assertThat(rows.get(2).isNullAt(3)).isTrue(); + + DataSplit split = (DataSplit) table.newScan().plan().splits().get(0); + assertThat(split.dataFiles()).hasSize(1); + DataFileMeta file = split.dataFiles().get(0); + assertThat(file.fileFormat()).isEqualTo(CoreOptions.FILE_FORMAT_PARQUET); + + SimpleStats stats = file.valueStats(); + assertThat(stats.minValues().isNullAt(1)).isTrue(); + assertThat(stats.maxValues().isNullAt(1)).isTrue(); + assertThat(stats.nullCounts().getLong(1)).isEqualTo(1L); + assertThat(stats.minValues().isNullAt(2)).isTrue(); + assertThat(stats.maxValues().isNullAt(2)).isTrue(); + assertThat(stats.nullCounts().getLong(2)).isEqualTo(1L); + } + + @Override + protected Schema schemaDefault() { + return Schema.newBuilder() + .column("id", DataTypes.INT()) + .column("geom", DataTypes.GEOMETRY("EPSG:3857")) + .column("geog", DataTypes.GEOGRAPHY("OGC:CRS84", EdgeAlgorithm.KARNEY)) + .column("geometries", DataTypes.ARRAY(DataTypes.GEOMETRY())) + .option(CoreOptions.FILE_FORMAT.key(), CoreOptions.FILE_FORMAT_PARQUET) + .build(); + } +} diff --git a/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/DataTypeToLogicalType.java b/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/DataTypeToLogicalType.java index 92ae714ca577..a1fe89efb295 100644 --- a/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/DataTypeToLogicalType.java +++ b/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/DataTypeToLogicalType.java @@ -31,6 +31,8 @@ import org.apache.paimon.types.DecimalType; import org.apache.paimon.types.DoubleType; import org.apache.paimon.types.FloatType; +import org.apache.paimon.types.GeographyType; +import org.apache.paimon.types.GeometryType; import org.apache.paimon.types.IntType; import org.apache.paimon.types.LocalZonedTimestampType; import org.apache.paimon.types.MapType; @@ -155,6 +157,23 @@ public LogicalType visit(BlobType blobType) { org.apache.flink.table.types.logical.VarBinaryType.MAX_LENGTH); } + @Override + public LogicalType visit(GeometryType geometryType) { + throw unsupportedGeospatialType(geometryType); + } + + @Override + public LogicalType visit(GeographyType geographyType) { + throw unsupportedGeospatialType(geographyType); + } + + private UnsupportedOperationException unsupportedGeospatialType(DataType dataType) { + return new UnsupportedOperationException( + "Flink SQL does not support Paimon geospatial type " + + dataType.asSQLString() + + ". Exposing it as VARBINARY would lose its CRS and edge algorithm."); + } + @Override public LogicalType visit(ArrayType arrayType) { return new org.apache.flink.table.types.logical.ArrayType( diff --git a/paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/GeospatialTypeTableITCase.java b/paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/GeospatialTypeTableITCase.java new file mode 100644 index 000000000000..4a319a645986 --- /dev/null +++ b/paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/GeospatialTypeTableITCase.java @@ -0,0 +1,73 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.flink; + +import org.apache.paimon.CoreOptions; +import org.apache.paimon.catalog.Identifier; +import org.apache.paimon.schema.Schema; +import org.apache.paimon.types.DataTypes; +import org.apache.paimon.types.EdgeAlgorithm; + +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** Tests Flink SQL interoperability with Paimon geospatial columns. */ +public class GeospatialTypeTableITCase extends CatalogITCaseBase { + + private static final String TABLE_NAME = "geospatial_table"; + + @Test + public void testRejectGeospatialColumnsInFlinkSql() throws Exception { + createGeospatialTable(); + + assertUnsupported(() -> batchSql("SELECT * FROM %s", TABLE_NAME)); + assertUnsupported( + () -> + batchSql( + "CREATE TABLE geospatial_like LIKE %s (EXCLUDING OPTIONS)", + TABLE_NAME)); + assertUnsupported( + () -> batchSql("CREATE TABLE geospatial_ctas AS SELECT * FROM %s", TABLE_NAME)); + } + + private void createGeospatialTable() throws Exception { + flinkCatalog() + .catalog() + .createTable( + Identifier.create(tEnv.getCurrentDatabase(), TABLE_NAME), + Schema.newBuilder() + .column("id", DataTypes.INT()) + .column("geom", DataTypes.GEOMETRY("EPSG:3857")) + .column( + "geog", + DataTypes.GEOGRAPHY("OGC:CRS84", EdgeAlgorithm.SPHERICAL)) + .option( + CoreOptions.FILE_FORMAT.key(), + CoreOptions.FILE_FORMAT_PARQUET) + .build(), + false); + } + + private void assertUnsupported(org.assertj.core.api.ThrowableAssert.ThrowingCallable callable) { + assertThatThrownBy(callable) + .hasStackTraceContaining("Flink SQL does not support Paimon geospatial type") + .hasStackTraceContaining("Exposing it as VARBINARY would lose its CRS"); + } +} diff --git a/paimon-format/src/main/java/org/apache/paimon/format/ArrowSchemaMetadata.java b/paimon-format/src/main/java/org/apache/paimon/format/ArrowSchemaMetadata.java index e8abaae336a8..acf1e3dd3dcd 100644 --- a/paimon-format/src/main/java/org/apache/paimon/format/ArrowSchemaMetadata.java +++ b/paimon-format/src/main/java/org/apache/paimon/format/ArrowSchemaMetadata.java @@ -33,6 +33,8 @@ import org.apache.paimon.types.DecimalType; import org.apache.paimon.types.DoubleType; import org.apache.paimon.types.FloatType; +import org.apache.paimon.types.GeographyType; +import org.apache.paimon.types.GeometryType; import org.apache.paimon.types.IntType; import org.apache.paimon.types.LocalZonedTimestampType; import org.apache.paimon.types.MapType; @@ -46,6 +48,7 @@ import org.apache.paimon.types.VarCharType; import org.apache.paimon.types.VariantType; import org.apache.paimon.types.VectorType; +import org.apache.paimon.utils.JsonSerdeUtil; import com.google.flatbuffers.FlatBufferBuilder; @@ -67,10 +70,16 @@ * FlatBuffers layout, enum values, and defaults are adapted from Apache Arrow Java / Arrow format * generated classes. This class implements only the subset needed by Paimon field metadata so that * {@code paimon-format} can stay compatible with {@code ARROW:schema} without depending on the - * Arrow runtime. + * Arrow runtime. Geospatial types are the exception to the direct {@code ArrowUtils} mapping: this + * metadata-only encoder writes the standard GeoArrow WKB extension metadata, while the Java Arrow + * API rejects geospatial conversion until it can expose the extension type itself. */ class ArrowSchemaMetadata { + private static final String ARROW_EXTENSION_NAME = "ARROW:extension:name"; + private static final String ARROW_EXTENSION_METADATA = "ARROW:extension:metadata"; + private static final String GEOARROW_WKB_EXTENSION_NAME = "geoarrow.wkb"; + private static final String LIST_DATA_VECTOR_NAME = "$data$"; private static final String MAP_DATA_VECTOR_NAME = "entries"; private static final String MAP_KEY_NAME = "key"; @@ -396,7 +405,10 @@ private static ArrowField withMetadata(ArrowField field, Map met private static ArrowField toArrowField( String fieldName, int fieldId, DataType dataType, int depth, String fieldIdKey) { ArrowTypeInfo type = dataType.accept(ArrowFieldTypeVisitor.INSTANCE); - Map metadata = fieldIdMetadata(fieldId, fieldIdKey); + Map metadata = new LinkedHashMap<>(fieldIdMetadata(fieldId, fieldIdKey)); + if (dataType instanceof GeometryType || dataType instanceof GeographyType) { + metadata.putAll(geospatialMetadata(dataType)); + } List children = Collections.emptyList(); if (dataType instanceof ArrayType || dataType instanceof VectorType) { DataType elementType = @@ -447,6 +459,22 @@ private static ArrowField toArrowField( return new ArrowField(fieldName, dataType.isNullable(), type, children, metadata); } + private static Map geospatialMetadata(DataType dataType) { + Map extensionMetadata = new LinkedHashMap<>(); + if (dataType instanceof GeographyType) { + GeographyType geographyType = (GeographyType) dataType; + extensionMetadata.put("edges", geographyType.getAlgorithm().toString()); + extensionMetadata.put("crs", geographyType.getCrs()); + } else { + extensionMetadata.put("crs", ((GeometryType) dataType).getCrs()); + } + + Map metadata = new LinkedHashMap<>(); + metadata.put(ARROW_EXTENSION_NAME, GEOARROW_WKB_EXTENSION_NAME); + metadata.put(ARROW_EXTENSION_METADATA, JsonSerdeUtil.toFlatJson(extensionMetadata)); + return metadata; + } + private static ArrowField toArrowMapEntryField( int fieldId, MapType mapType, int depth, String fieldIdKey) { ArrowField keyField = @@ -511,8 +539,10 @@ private ArrowField( this.metadata = metadata; } - private ArrowField withMetadata(Map metadata) { - return new ArrowField(name, nullable, type, children, metadata); + private ArrowField withMetadata(Map additionalMetadata) { + Map mergedMetadata = new LinkedHashMap<>(metadata); + mergedMetadata.putAll(additionalMetadata); + return new ArrowField(name, nullable, type, children, mergedMetadata); } } @@ -565,6 +595,16 @@ public ArrowTypeInfo visit(VarBinaryType varBinaryType) { return ArrowTypeInfo.simple(TYPE_BINARY); } + @Override + public ArrowTypeInfo visit(GeometryType geometryType) { + return ArrowTypeInfo.simple(TYPE_BINARY); + } + + @Override + public ArrowTypeInfo visit(GeographyType geographyType) { + return ArrowTypeInfo.simple(TYPE_BINARY); + } + @Override public ArrowTypeInfo visit(DecimalType decimalType) { ArrowTypeInfo type = ArrowTypeInfo.simple(TYPE_DECIMAL); diff --git a/paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetSchemaConverter.java b/paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetSchemaConverter.java index 309ed5c5ffd4..912baec0721c 100644 --- a/paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetSchemaConverter.java +++ b/paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetSchemaConverter.java @@ -25,6 +25,9 @@ import org.apache.paimon.types.DataType; import org.apache.paimon.types.DataTypes; import org.apache.paimon.types.DecimalType; +import org.apache.paimon.types.EdgeAlgorithm; +import org.apache.paimon.types.GeographyType; +import org.apache.paimon.types.GeometryType; import org.apache.paimon.types.IntType; import org.apache.paimon.types.LocalZonedTimestampType; import org.apache.paimon.types.MapType; @@ -34,6 +37,8 @@ import org.apache.paimon.types.VectorType; import org.apache.paimon.utils.Pair; +import org.apache.parquet.column.schema.EdgeInterpolationAlgorithm; +import org.apache.parquet.schema.ColumnOrder; import org.apache.parquet.schema.ConversionPatterns; import org.apache.parquet.schema.GroupType; import org.apache.parquet.schema.LogicalTypeAnnotation; @@ -128,6 +133,23 @@ public static Type convertToParquetType(String name, DataType type, int fieldId, return Types.primitive(PrimitiveType.PrimitiveTypeName.BINARY, repetition) .named(name) .withId(fieldId); + case GEOMETRY: + return Types.primitive(PrimitiveType.PrimitiveTypeName.BINARY, repetition) + .as(LogicalTypeAnnotation.geometryType(((GeometryType) type).getCrs())) + .columnOrder(ColumnOrder.undefined()) + .named(name) + .withId(fieldId); + case GEOGRAPHY: + GeographyType geographyType = (GeographyType) type; + return Types.primitive(PrimitiveType.PrimitiveTypeName.BINARY, repetition) + .as( + LogicalTypeAnnotation.geographyType( + geographyType.getCrs(), + EdgeInterpolationAlgorithm.valueOf( + geographyType.getAlgorithm().name()))) + .columnOrder(ColumnOrder.undefined()) + .named(name) + .withId(fieldId); case DECIMAL: int precision = ((DecimalType) type).getPrecision(); int scale = ((DecimalType) type).getScale(); @@ -336,6 +358,22 @@ public static DataField convertToPaimonField(Type parquetType) { case BINARY: if (logicalType instanceof LogicalTypeAnnotation.StringLogicalTypeAnnotation) { paimonDataType = DataTypes.STRING(); + } else if (logicalType + instanceof LogicalTypeAnnotation.GeometryLogicalTypeAnnotation) { + paimonDataType = + DataTypes.GEOMETRY( + ((LogicalTypeAnnotation.GeometryLogicalTypeAnnotation) + logicalType) + .getCrs()); + } else if (logicalType + instanceof LogicalTypeAnnotation.GeographyLogicalTypeAnnotation) { + LogicalTypeAnnotation.GeographyLogicalTypeAnnotation geography = + (LogicalTypeAnnotation.GeographyLogicalTypeAnnotation) logicalType; + EdgeAlgorithm algorithm = + geography.getAlgorithm() == null + ? null + : EdgeAlgorithm.valueOf(geography.getAlgorithm().name()); + paimonDataType = DataTypes.GEOGRAPHY(geography.getCrs(), algorithm); } else { paimonDataType = DataTypes.BYTES(); } diff --git a/paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetSimpleStatsExtractor.java b/paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetSimpleStatsExtractor.java index fa91dbf28927..39492e8bdf2a 100644 --- a/paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetSimpleStatsExtractor.java +++ b/paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetSimpleStatsExtractor.java @@ -152,6 +152,10 @@ private SimpleColStats toFieldStats( binaryStats.genericGetMax().getBytes(), nullCount); break; + case GEOMETRY: + case GEOGRAPHY: + fieldStats = new SimpleColStats(null, null, nullCount); + break; case BOOLEAN: assertStatsClass(field, stats, BooleanStatistics.class); BooleanStatistics boolStats = (BooleanStatistics) stats; diff --git a/paimon-format/src/main/java/org/apache/paimon/format/parquet/reader/ParquetVectorUpdaterFactory.java b/paimon-format/src/main/java/org/apache/paimon/format/parquet/reader/ParquetVectorUpdaterFactory.java index 6239c9fbb51a..c0445b3f136d 100644 --- a/paimon-format/src/main/java/org/apache/paimon/format/parquet/reader/ParquetVectorUpdaterFactory.java +++ b/paimon-format/src/main/java/org/apache/paimon/format/parquet/reader/ParquetVectorUpdaterFactory.java @@ -45,6 +45,8 @@ import org.apache.paimon.types.DecimalType; import org.apache.paimon.types.DoubleType; import org.apache.paimon.types.FloatType; +import org.apache.paimon.types.GeographyType; +import org.apache.paimon.types.GeometryType; import org.apache.paimon.types.IntType; import org.apache.paimon.types.LocalZonedTimestampType; import org.apache.paimon.types.MapType; @@ -129,6 +131,16 @@ public UpdaterFactory visit(VarBinaryType varBinaryType) { }; } + @Override + public UpdaterFactory visit(GeometryType geometryType) { + return c -> new BinaryUpdater(); + } + + @Override + public UpdaterFactory visit(GeographyType geographyType) { + return c -> new BinaryUpdater(); + } + @Override public UpdaterFactory visit(DecimalType decimalType) { return c -> { diff --git a/paimon-format/src/main/java/org/apache/paimon/format/parquet/writer/ParquetRowDataWriter.java b/paimon-format/src/main/java/org/apache/paimon/format/parquet/writer/ParquetRowDataWriter.java index 89a45c38c610..3594f6fcd7ae 100644 --- a/paimon-format/src/main/java/org/apache/paimon/format/parquet/writer/ParquetRowDataWriter.java +++ b/paimon-format/src/main/java/org/apache/paimon/format/parquet/writer/ParquetRowDataWriter.java @@ -93,6 +93,8 @@ private FieldWriter createWriter(DataType t, Type type) { return new BooleanWriter(); case BINARY: case VARBINARY: + case GEOMETRY: + case GEOGRAPHY: return new BinaryWriter(); case BLOB: return new BlobDescriptorWriter(); diff --git a/paimon-format/src/main/java/org/apache/parquet/hadoop/ColumnCompressionPageWriteStore.java b/paimon-format/src/main/java/org/apache/parquet/hadoop/ColumnCompressionPageWriteStore.java index 12250aa3f8df..a5c267b92c8a 100644 --- a/paimon-format/src/main/java/org/apache/parquet/hadoop/ColumnCompressionPageWriteStore.java +++ b/paimon-format/src/main/java/org/apache/parquet/hadoop/ColumnCompressionPageWriteStore.java @@ -44,6 +44,7 @@ import org.apache.parquet.internal.column.columnindex.ColumnIndexBuilder; import org.apache.parquet.internal.column.columnindex.OffsetIndexBuilder; import org.apache.parquet.io.ParquetEncodingException; +import org.apache.parquet.schema.LogicalTypeAnnotation; import org.apache.parquet.schema.MessageType; import org.apache.parquet.util.AutoCloseables; import org.slf4j.Logger; @@ -76,6 +77,7 @@ public class ColumnCompressionPageWriteStore implements PageWriteStore, BloomFil private static final class ColumnChunkPageWriter implements PageWriter, BloomFilterWriter { private final ColumnDescriptor path; + private final boolean geospatial; private final CompressionCodecFactory.BytesInputCompressor compressor; private final ByteArrayOutputStream tempOutputStream = new ByteArrayOutputStream(); @@ -123,6 +125,11 @@ private ColumnChunkPageWriter( int rowGroupOrdinal, int columnOrdinal) { this.path = path; + LogicalTypeAnnotation logicalType = path.getPrimitiveType().getLogicalTypeAnnotation(); + this.geospatial = + logicalType instanceof LogicalTypeAnnotation.GeometryLogicalTypeAnnotation + || logicalType + instanceof LogicalTypeAnnotation.GeographyLogicalTypeAnnotation; this.compressor = compressor; this.releaser = new ByteBufferReleaser(allocator); this.buf = new ConcatenatingByteBufferCollector(allocator); @@ -426,6 +433,12 @@ private void mergeColumnStatistics( Statistics statistics, SizeStatistics sizeStatistics, GeospatialStatistics geospatialStatistics) { + if (geospatial && statistics != null && statistics.isNumNullsSet()) { + statistics = + Statistics.getBuilderForReading(path.getPrimitiveType()) + .withNumNulls(statistics.getNumNulls()) + .build(); + } totalSizeStatistics.mergeStatistics(sizeStatistics); if (!totalSizeStatistics.isValid()) { sizeStatistics = null; diff --git a/paimon-format/src/test/java/org/apache/paimon/format/FormatMetadataUtilsTest.java b/paimon-format/src/test/java/org/apache/paimon/format/FormatMetadataUtilsTest.java index f01854f9e314..30213231e37d 100644 --- a/paimon-format/src/test/java/org/apache/paimon/format/FormatMetadataUtilsTest.java +++ b/paimon-format/src/test/java/org/apache/paimon/format/FormatMetadataUtilsTest.java @@ -145,4 +145,33 @@ public void testBuildArrowSchemaWithoutFieldIdMetadata() { assertThat(metadata.get("name")) .doesNotContainKey(FormatMetadataUtils.PARQUET_FIELD_ID_KEY); } + + @Test + public void testBuildArrowSchemaWithGeospatialMetadata() { + RowType rowType = + DataTypes.ROW( + DataTypes.FIELD(0, "geom", DataTypes.GEOMETRY()), + DataTypes.FIELD(1, "geog", DataTypes.GEOGRAPHY())); + + byte[] schemaBytes = + FormatMetadataUtils.buildArrowSchemaMetadata( + rowType, + java.util.Collections.emptyMap(), + FormatMetadataUtils.PARQUET_FIELD_ID_KEY); + + Map> metadata = + FormatMetadataUtils.readFieldMetadata(schemaBytes); + assertThat(metadata.get("geom")) + .containsEntry("ARROW:extension:name", "geoarrow.wkb") + .containsEntry("ARROW:extension:metadata", "{\"crs\":\"OGC:CRS84\"}") + .containsEntry(FormatMetadataUtils.PARQUET_FIELD_ID_KEY, "0") + .doesNotContainKey("paimon.type"); + assertThat(metadata.get("geog")) + .containsEntry("ARROW:extension:name", "geoarrow.wkb") + .containsEntry( + "ARROW:extension:metadata", + "{\"edges\":\"spherical\",\"crs\":\"OGC:CRS84\"}") + .containsEntry(FormatMetadataUtils.PARQUET_FIELD_ID_KEY, "1") + .doesNotContainKey("paimon.type"); + } } diff --git a/paimon-format/src/test/java/org/apache/paimon/format/parquet/ParquetFormatReadWriteTest.java b/paimon-format/src/test/java/org/apache/paimon/format/parquet/ParquetFormatReadWriteTest.java index 40e714291c67..2784f157a113 100644 --- a/paimon-format/src/test/java/org/apache/paimon/format/parquet/ParquetFormatReadWriteTest.java +++ b/paimon-format/src/test/java/org/apache/paimon/format/parquet/ParquetFormatReadWriteTest.java @@ -19,7 +19,13 @@ package org.apache.paimon.format.parquet; import org.apache.paimon.data.BinaryString; +import org.apache.paimon.data.GenericArray; +import org.apache.paimon.data.GenericMap; import org.apache.paimon.data.GenericRow; +import org.apache.paimon.data.InternalArray; +import org.apache.paimon.data.InternalMap; +import org.apache.paimon.data.InternalRow; +import org.apache.paimon.data.serializer.InternalRowSerializer; import org.apache.paimon.format.FileFormat; import org.apache.paimon.format.FileFormatFactory; import org.apache.paimon.format.FormatMetadataUtils; @@ -30,6 +36,7 @@ import org.apache.paimon.format.SupportsWriterMetadata; import org.apache.paimon.fs.PositionOutputStream; import org.apache.paimon.options.Options; +import org.apache.paimon.reader.RecordReader; import org.apache.paimon.types.DataTypes; import org.apache.paimon.types.RowType; @@ -46,6 +53,7 @@ import java.nio.charset.StandardCharsets; import java.util.HashMap; +import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import java.util.concurrent.ThreadLocalRandom; @@ -68,6 +76,94 @@ public void testArrayBlobDescriptors() throws Exception { testArrayBlobDescriptorRoundTrip(); } + @Test + public void testGeospatialWkbRoundTrip() throws Exception { + byte[] pointWkb = + new byte[] { + 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, (byte) 0xf0, 0x3f, 0, 0, 0, 0, 0, 0, 0, 0x40 + }; + RowType rowType = + DataTypes.ROW( + DataTypes.FIELD(0, "geom", DataTypes.GEOMETRY()), + DataTypes.FIELD(1, "geog", DataTypes.GEOGRAPHY()), + DataTypes.FIELD(2, "geometries", DataTypes.ARRAY(DataTypes.GEOMETRY())), + DataTypes.FIELD( + 3, + "geospatial_map", + DataTypes.MAP(DataTypes.GEOMETRY(), DataTypes.GEOGRAPHY())), + DataTypes.FIELD( + 4, + "nested", + DataTypes.ROW( + DataTypes.FIELD(5, "nested_geom", DataTypes.GEOMETRY())))); + Map map = new LinkedHashMap<>(); + map.put(pointWkb, pointWkb); + + write( + fileFormat().createWriterFactory(rowType), + file, + GenericRow.of( + pointWkb, + pointWkb, + new GenericArray(new Object[] {pointWkb, null}), + GenericMap.fromBinaryKeyMap(map), + GenericRow.of(pointWkb))); + + try (RecordReader reader = + fileFormat() + .createReaderFactory(rowType, rowType, java.util.Collections.emptyList()) + .createReader( + new FormatReaderContext( + fileIO, file, fileIO.getFileSize(file), null, null))) { + InternalRow row = new InternalRowSerializer(rowType).copy(reader.readBatch().next()); + Assertions.assertThat(row.getBinary(0)).isEqualTo(pointWkb); + Assertions.assertThat(row.getBinary(1)).isEqualTo(pointWkb); + InternalArray geometries = row.getArray(2); + Assertions.assertThat(geometries.getBinary(0)).isEqualTo(pointWkb); + Assertions.assertThat(geometries.isNullAt(1)).isTrue(); + InternalMap geospatialMap = row.getMap(3); + Assertions.assertThat(geospatialMap.keyArray().getBinary(0)).isEqualTo(pointWkb); + Assertions.assertThat(geospatialMap.valueArray().getBinary(0)).isEqualTo(pointWkb); + Assertions.assertThat(row.getRow(4, 1).getBinary(0)).isEqualTo(pointWkb); + } + + try (ParquetFileReader reader = + ParquetUtil.getParquetReader( + fileIO, file, fileIO.getFileSize(file), new Options())) { + Map columns = new HashMap<>(); + for (ColumnChunkMetaData column : reader.getFooter().getBlocks().get(0).getColumns()) { + columns.put(column.getPath().toDotString(), column); + } + Assertions.assertThat(columns) + .containsKeys( + "geom", + "geog", + "geometries.list.element", + "geospatial_map.key_value.key", + "geospatial_map.key_value.value", + "nested.nested_geom"); + for (ColumnChunkMetaData column : columns.values()) { + Assertions.assertThat(column.getStatistics().hasNonNullValue()) + .as(column.getPath().toDotString()) + .isFalse(); + Assertions.assertThat(column.getStatistics().isNumNullsSet()) + .as(column.getPath().toDotString()) + .isTrue(); + } + Assertions.assertThat( + columns.get("geometries.list.element").getStatistics().getNumNulls()) + .isEqualTo(1); + Assertions.assertThat(columns.get("geom").getGeospatialStatistics()).isNotNull(); + Assertions.assertThat(columns.get("geometries.list.element").getGeospatialStatistics()) + .isNotNull(); + Assertions.assertThat( + columns.get("geospatial_map.key_value.key").getGeospatialStatistics()) + .isNotNull(); + Assertions.assertThat(columns.get("nested.nested_geom").getGeospatialStatistics()) + .isNotNull(); + } + } + @Test public void testWriteMetadata() throws Exception { ParquetFileFormat format = diff --git a/paimon-format/src/test/java/org/apache/paimon/format/parquet/ParquetSchemaConverterTest.java b/paimon-format/src/test/java/org/apache/paimon/format/parquet/ParquetSchemaConverterTest.java index 2808cc535abb..f40550831dff 100644 --- a/paimon-format/src/test/java/org/apache/paimon/format/parquet/ParquetSchemaConverterTest.java +++ b/paimon-format/src/test/java/org/apache/paimon/format/parquet/ParquetSchemaConverterTest.java @@ -21,13 +21,16 @@ import org.apache.paimon.types.ArrayType; import org.apache.paimon.types.DataField; import org.apache.paimon.types.DataTypes; +import org.apache.paimon.types.EdgeAlgorithm; import org.apache.paimon.types.MapType; import org.apache.paimon.types.RowType; +import org.apache.parquet.schema.ColumnOrder; import org.apache.parquet.schema.LogicalTypeAnnotation; import org.apache.parquet.schema.MessageType; import org.apache.parquet.schema.Type; import org.apache.parquet.schema.Types; +import org.assertj.core.api.Assertions; import org.junit.jupiter.api.Test; import java.util.Arrays; @@ -35,6 +38,7 @@ import static org.apache.paimon.format.parquet.ParquetSchemaConverter.convertToPaimonRowType; import static org.apache.paimon.format.parquet.ParquetSchemaConverter.convertToParquetMessageType; import static org.apache.paimon.types.DataTypesTest.assertThat; +import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.BINARY; import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT64; /** Test for {@link ParquetSchemaConverter}. */ @@ -141,4 +145,65 @@ public void testPaimonParquetSchemaConvert() { RowType rowType = convertToPaimonRowType(messageType); assertThat(ALL_TYPES).isEqualTo(rowType); } + + @Test + public void testGeospatialLogicalTypesRoundTrip() { + RowType expected = + new RowType( + Arrays.asList( + new DataField(0, "geom", DataTypes.GEOMETRY("EPSG:3857")), + new DataField( + 1, + "geog", + DataTypes.GEOGRAPHY("OGC:CRS84", EdgeAlgorithm.KARNEY) + .notNull()))); + + MessageType messageType = convertToParquetMessageType(expected); + Type geometry = messageType.getType("geom"); + Type geography = messageType.getType("geog"); + + Assertions.assertThat(geometry.asPrimitiveType().getPrimitiveTypeName()).isEqualTo(BINARY); + Assertions.assertThat(geometry.asPrimitiveType().columnOrder().getColumnOrderName()) + .isEqualTo(ColumnOrder.ColumnOrderName.UNDEFINED); + Assertions.assertThat(geography.asPrimitiveType().columnOrder().getColumnOrderName()) + .isEqualTo(ColumnOrder.ColumnOrderName.UNDEFINED); + Assertions.assertThat(geometry.getLogicalTypeAnnotation()) + .isInstanceOf(LogicalTypeAnnotation.GeometryLogicalTypeAnnotation.class); + Assertions.assertThat( + ((LogicalTypeAnnotation.GeometryLogicalTypeAnnotation) + geometry.getLogicalTypeAnnotation()) + .getCrs()) + .isEqualTo("EPSG:3857"); + Assertions.assertThat( + ((LogicalTypeAnnotation.GeographyLogicalTypeAnnotation) + geography.getLogicalTypeAnnotation()) + .getAlgorithm() + .name()) + .isEqualTo("KARNEY"); + assertThat(expected).isEqualTo(convertToPaimonRowType(messageType)); + } + + @Test + public void testGeographyLogicalTypeDefaults() { + MessageType messageType = + new MessageType( + "geography-defaults", + Types.primitive(BINARY, Type.Repetition.OPTIONAL) + .as(LogicalTypeAnnotation.geographyType()) + .named("default_geography") + .withId(0), + Types.primitive(BINARY, Type.Repetition.OPTIONAL) + .as(LogicalTypeAnnotation.geographyType("EPSG:4326", null)) + .named("default_algorithm") + .withId(1)); + + RowType expected = + new RowType( + Arrays.asList( + new DataField(0, "default_geography", DataTypes.GEOGRAPHY()), + new DataField( + 1, "default_algorithm", DataTypes.GEOGRAPHY("EPSG:4326")))); + + assertThat(expected).isEqualTo(convertToPaimonRowType(messageType)); + } } diff --git a/paimon-spark/paimon-spark-3.2/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTest.scala b/paimon-spark/paimon-spark-3.2/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTest.scala new file mode 100644 index 000000000000..dde6fabf6a09 --- /dev/null +++ b/paimon-spark/paimon-spark-3.2/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTest.scala @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.spark.sql + +class GeospatialUnsupportedTest extends GeospatialUnsupportedTestBase {} diff --git a/paimon-spark/paimon-spark-3.3/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTest.scala b/paimon-spark/paimon-spark-3.3/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTest.scala new file mode 100644 index 000000000000..dde6fabf6a09 --- /dev/null +++ b/paimon-spark/paimon-spark-3.3/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTest.scala @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.spark.sql + +class GeospatialUnsupportedTest extends GeospatialUnsupportedTestBase {} diff --git a/paimon-spark/paimon-spark-3.4/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTest.scala b/paimon-spark/paimon-spark-3.4/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTest.scala new file mode 100644 index 000000000000..dde6fabf6a09 --- /dev/null +++ b/paimon-spark/paimon-spark-3.4/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTest.scala @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.spark.sql + +class GeospatialUnsupportedTest extends GeospatialUnsupportedTestBase {} diff --git a/paimon-spark/paimon-spark-3.5/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTest.scala b/paimon-spark/paimon-spark-3.5/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTest.scala new file mode 100644 index 000000000000..dde6fabf6a09 --- /dev/null +++ b/paimon-spark/paimon-spark-3.5/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTest.scala @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.spark.sql + +class GeospatialUnsupportedTest extends GeospatialUnsupportedTestBase {} diff --git a/paimon-spark/paimon-spark-4.0/src/main/scala/org/apache/spark/sql/paimon/shims/Spark4Shim.scala b/paimon-spark/paimon-spark-4.0/src/main/scala/org/apache/spark/sql/paimon/shims/Spark4Shim.scala index f9e7ba1c0b8d..12dddf4db571 100644 --- a/paimon-spark/paimon-spark-4.0/src/main/scala/org/apache/spark/sql/paimon/shims/Spark4Shim.scala +++ b/paimon-spark/paimon-spark-4.0/src/main/scala/org/apache/spark/sql/paimon/shims/Spark4Shim.scala @@ -381,6 +381,46 @@ class Spark4Shim extends SparkShim { override def SparkVariantType(): org.apache.spark.sql.types.DataType = DataTypes.VariantType + override def toPaimonGeometry(o: Object): Array[Byte] = unsupportedGeospatial() + + override def toPaimonGeometry(row: InternalRow, pos: Int): Array[Byte] = unsupportedGeospatial() + + override def toPaimonGeometry(array: ArrayData, pos: Int): Array[Byte] = unsupportedGeospatial() + + override def toPaimonGeography(o: Object): Array[Byte] = unsupportedGeospatial() + + override def toPaimonGeography(row: InternalRow, pos: Int): Array[Byte] = unsupportedGeospatial() + + override def toPaimonGeography(array: ArrayData, pos: Int): Array[Byte] = unsupportedGeospatial() + + override def toSparkGeometry(wkb: Array[Byte], crs: String): Object = unsupportedGeospatial() + + override def toSparkGeography(wkb: Array[Byte], crs: String, algorithm: String): Object = + unsupportedGeospatial() + + override def isSparkGeometryType(dataType: org.apache.spark.sql.types.DataType): Boolean = false + + override def isSparkGeographyType(dataType: org.apache.spark.sql.types.DataType): Boolean = false + + override def SparkGeometryType(crs: String): org.apache.spark.sql.types.DataType = + unsupportedGeospatial() + + override def SparkGeographyType( + crs: String, + algorithm: String): org.apache.spark.sql.types.DataType = unsupportedGeospatial() + + override def sparkGeometryCrs(dataType: org.apache.spark.sql.types.DataType): String = + unsupportedGeospatial() + + override def sparkGeographyCrs(dataType: org.apache.spark.sql.types.DataType): String = + unsupportedGeospatial() + + override def sparkGeographyAlgorithm(dataType: org.apache.spark.sql.types.DataType): String = + unsupportedGeospatial() + + private def unsupportedGeospatial[T](): T = + throw new UnsupportedOperationException("Geometry and geography require Spark 4.1 or later") + // SQL UDFs (CREATE FUNCTION ... RETURN ...). override def rewritePaimonSQLFunctionCommands(spark: SparkSession): Rule[LogicalPlan] = org.apache.spark.sql.catalyst.parser.extensions.RewritePaimonSQLFunctionCommands(spark) diff --git a/paimon-spark/paimon-spark-4.0/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTest.scala b/paimon-spark/paimon-spark-4.0/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTest.scala new file mode 100644 index 000000000000..dde6fabf6a09 --- /dev/null +++ b/paimon-spark/paimon-spark-4.0/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTest.scala @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.spark.sql + +class GeospatialUnsupportedTest extends GeospatialUnsupportedTestBase {} diff --git a/paimon-spark/paimon-spark-4.1/src/test/java/org/apache/paimon/spark/GeospatialTypeTest.java b/paimon-spark/paimon-spark-4.1/src/test/java/org/apache/paimon/spark/GeospatialTypeTest.java new file mode 100644 index 000000000000..5d46832e5503 --- /dev/null +++ b/paimon-spark/paimon-spark-4.1/src/test/java/org/apache/paimon/spark/GeospatialTypeTest.java @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.spark; + +import org.apache.paimon.data.GenericRow; +import org.apache.paimon.spark.data.SparkInternalRow; +import org.apache.paimon.types.DataTypes; +import org.apache.paimon.types.EdgeAlgorithm; +import org.apache.paimon.types.RowType; + +import org.apache.spark.sql.RowFactory; +import org.apache.spark.sql.catalyst.util.STUtils; +import org.apache.spark.sql.types.Geography; +import org.apache.spark.sql.types.GeographyType; +import org.apache.spark.sql.types.Geometry; +import org.apache.spark.sql.types.GeometryType; +import org.apache.spark.sql.types.StructType; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** Tests Spark 4.1 geometry and geography interoperability. */ +class GeospatialTypeTest { + + private static final byte[] POINT_WKB = + new byte[] {1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, (byte) 0xf0, 0x3f, 0, 0, 0, 0, 0, 0, 0x40}; + + @Test + void testTypeRoundTrip() { + RowType paimonType = + DataTypes.ROW( + DataTypes.FIELD(0, "geom", DataTypes.GEOMETRY()), + DataTypes.FIELD( + 1, + "geog", + DataTypes.GEOGRAPHY("OGC:CRS84", EdgeAlgorithm.SPHERICAL))); + + StructType sparkType = SparkTypeUtils.fromPaimonRowType(paimonType); + assertThat(sparkType.apply("geom").dataType()).isInstanceOf(GeometryType.class); + assertThat(((GeometryType) sparkType.apply("geom").dataType()).crs()) + .isEqualTo("OGC:CRS84"); + assertThat(sparkType.apply("geog").dataType()).isInstanceOf(GeographyType.class); + assertThat(((GeographyType) sparkType.apply("geog").dataType()).crs()) + .isEqualTo("OGC:CRS84"); + assertThat(((GeographyType) sparkType.apply("geog").dataType()).algorithm().toString()) + .isEqualTo("SPHERICAL"); + assertThat(SparkTypeUtils.toPaimonType(sparkType)).isEqualTo(paimonType); + + assertThatThrownBy( + () -> + SparkTypeUtils.fromPaimonType( + DataTypes.GEOGRAPHY("OGC:CRS84", EdgeAlgorithm.KARNEY))) + .hasMessageContaining("karney"); + } + + @Test + void testWkbReadWriteRoundTrip() { + RowType paimonType = + DataTypes.ROW( + DataTypes.FIELD(0, "geom", DataTypes.GEOMETRY()), + DataTypes.FIELD(1, "geog", DataTypes.GEOGRAPHY())); + StructType sparkType = SparkTypeUtils.fromPaimonRowType(paimonType); + + SparkInternalRow sparkRow = + SparkInternalRow.create(paimonType).replace(GenericRow.of(POINT_WKB, POINT_WKB)); + assertThat(STUtils.stAsBinary(sparkRow.getGeometry(0))).isEqualTo(POINT_WKB); + assertThat(STUtils.stSrid(sparkRow.getGeometry(0))).isEqualTo(4326); + assertThat(STUtils.stAsBinary(sparkRow.getGeography(1))).isEqualTo(POINT_WKB); + assertThat(STUtils.stSrid(sparkRow.getGeography(1))).isEqualTo(4326); + + SparkInternalRowWrapper internalWrapper = + new SparkInternalRowWrapper(sparkType, 2).replace(sparkRow); + assertThat(internalWrapper.getBinary(0)).isEqualTo(POINT_WKB); + assertThat(internalWrapper.getBinary(1)).isEqualTo(POINT_WKB); + + SparkRow externalWrapper = + new SparkRow( + paimonType, + RowFactory.create( + Geometry.fromWKB(POINT_WKB, 4326), + Geography.fromWKB(POINT_WKB, 4326))); + assertThat(externalWrapper.getBinary(0)).isEqualTo(POINT_WKB); + assertThat(externalWrapper.getBinary(1)).isEqualTo(POINT_WKB); + } +} diff --git a/paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/GeospatialTypeSQLTest.scala b/paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/GeospatialTypeSQLTest.scala new file mode 100644 index 000000000000..204c7d43d483 --- /dev/null +++ b/paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/GeospatialTypeSQLTest.scala @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.spark.sql + +import org.apache.paimon.spark.PaimonSparkTestBase +import org.apache.paimon.types.{DataTypes, EdgeAlgorithm} + +import org.apache.spark.SparkConf +import org.apache.spark.sql.{AnalysisException, Row} + +/** Tests Spark 4.1 SQL interoperability with Paimon geospatial columns. */ +class GeospatialTypeSQLTest extends PaimonSparkTestBase { + + override protected def sparkConf: SparkConf = { + super.sparkConf.set("spark.sql.geospatial.enabled", "true") + } + + test("Spark SQL requires geospatial support to be enabled") { + withSparkSQLConf("spark.sql.geospatial.enabled" -> "false") { + val error = intercept[AnalysisException] { + sql("CREATE TABLE geospatial_disabled (geom GEOMETRY(4326)) USING paimon") + } + assert(error.getMessage.contains("GEOSPATIAL_DISABLED")) + } + } + + test("Spark SQL reads and writes native geospatial values") { + withTable("t") { + sql(""" + |CREATE TABLE t ( + | id INT, + | geom GEOMETRY(4326), + | geog GEOGRAPHY(4326) + |) TBLPROPERTIES ('file.format' = 'parquet') + |""".stripMargin) + + sql(""" + |INSERT INTO t VALUES + | (1, + | ST_SetSrid( + | ST_GeomFromWKB(unhex('0101000000000000000000F03F0000000000000040')), + | 4326), + | ST_GeogFromWKB(unhex('010100000000000000000008400000000000001040'))), + | (2, NULL, + | ST_GeogFromWKB(unhex('0101000000000000000000F03F0000000000000040'))) + |""".stripMargin) + + checkAnswer( + sql(""" + |SELECT id, + | hex(ST_AsBinary(geom)), ST_Srid(geom), + | hex(ST_AsBinary(geog)), ST_Srid(geog) + |FROM t ORDER BY id + |""".stripMargin), + Seq( + Row( + 1, + "0101000000000000000000F03F0000000000000040", + 4326, + "010100000000000000000008400000000000001040", + 4326), + Row(2, null, null, "0101000000000000000000F03F0000000000000040", 4326) + ) + ) + + val fields = loadTable("t").schema().fields() + assert(fields.get(1).`type`() == DataTypes.GEOMETRY("OGC:CRS84")) + assert( + fields.get(2).`type`() == + DataTypes.GEOGRAPHY("OGC:CRS84", EdgeAlgorithm.SPHERICAL)) + } + } +} diff --git a/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/AbstractSparkInternalRow.java b/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/AbstractSparkInternalRow.java index f522994937cf..46c48833d0bc 100644 --- a/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/AbstractSparkInternalRow.java +++ b/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/AbstractSparkInternalRow.java @@ -30,6 +30,7 @@ import org.apache.spark.sql.catalyst.util.ArrayData; import org.apache.spark.sql.catalyst.util.MapData; +import org.apache.spark.sql.paimon.shims.SparkShimLoader; import org.apache.spark.sql.types.BinaryType; import org.apache.spark.sql.types.BooleanType; import org.apache.spark.sql.types.ByteType; @@ -256,6 +257,21 @@ public Object get(int ordinal, org.apache.spark.sql.types.DataType dataType) { if (dataType instanceof UserDefinedType) { return get(ordinal, ((UserDefinedType) dataType).sqlType()); } + if (SparkShimLoader.shim().isSparkGeometryType(dataType)) { + org.apache.paimon.types.GeometryType geometryType = + (org.apache.paimon.types.GeometryType) rowType.getTypeAt(ordinal); + return SparkShimLoader.shim() + .toSparkGeometry(row.getBinary(ordinal), geometryType.getCrs()); + } + if (SparkShimLoader.shim().isSparkGeographyType(dataType)) { + org.apache.paimon.types.GeographyType geographyType = + (org.apache.paimon.types.GeographyType) rowType.getTypeAt(ordinal); + return SparkShimLoader.shim() + .toSparkGeography( + row.getBinary(ordinal), + geographyType.getCrs(), + geographyType.getAlgorithm().toString()); + } throw new UnsupportedOperationException("Unsupported data type " + dataType.simpleString()); } diff --git a/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/DataConverter.java b/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/DataConverter.java index 505df4e91207..d34a9c7da7d5 100644 --- a/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/DataConverter.java +++ b/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/DataConverter.java @@ -30,6 +30,8 @@ import org.apache.paimon.spark.util.shim.TypeUtils; import org.apache.paimon.types.ArrayType; import org.apache.paimon.types.DataType; +import org.apache.paimon.types.GeographyType; +import org.apache.paimon.types.GeometryType; import org.apache.paimon.types.IntType; import org.apache.paimon.types.MapType; import org.apache.paimon.types.MultisetType; @@ -40,6 +42,7 @@ import org.apache.spark.sql.catalyst.util.ArrayData; import org.apache.spark.sql.catalyst.util.DateTimeUtils; import org.apache.spark.sql.catalyst.util.MapData; +import org.apache.spark.sql.paimon.shims.SparkShimLoader; import org.apache.spark.sql.types.Decimal; import org.apache.spark.unsafe.types.UTF8String; @@ -70,6 +73,16 @@ public static Object fromPaimon(Object o, DataType type) { return fromPaimon((InternalRow) o, (RowType) type); case BLOB: return ((Blob) o).toData(); + case GEOMETRY: + return SparkShimLoader.shim() + .toSparkGeometry((byte[]) o, ((GeometryType) type).getCrs()); + case GEOGRAPHY: + GeographyType geographyType = (GeographyType) type; + return SparkShimLoader.shim() + .toSparkGeography( + (byte[]) o, + geographyType.getCrs(), + geographyType.getAlgorithm().toString()); default: return o; } diff --git a/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkInternalRowWrapper.java b/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkInternalRowWrapper.java index 782383981694..8e24dcf712cc 100644 --- a/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkInternalRowWrapper.java +++ b/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkInternalRowWrapper.java @@ -241,6 +241,13 @@ public byte[] getBinary(int pos) { if (actualPos == -1 || internalRow.isNullAt(actualPos)) { return null; } + DataType dataType = tableSchema.fields()[pos].dataType(); + if (SparkShimLoader.shim().isSparkGeometryType(dataType)) { + return SparkShimLoader.shim().toPaimonGeometry(internalRow, actualPos); + } + if (SparkShimLoader.shim().isSparkGeographyType(dataType)) { + return SparkShimLoader.shim().toPaimonGeography(internalRow, actualPos); + } return internalRow.getBinary(actualPos); } @@ -462,6 +469,12 @@ public Timestamp getTimestamp(int pos, int precision) { @Override public byte[] getBinary(int pos) { + if (SparkShimLoader.shim().isSparkGeometryType(elementType)) { + return SparkShimLoader.shim().toPaimonGeometry(arrayData, pos); + } + if (SparkShimLoader.shim().isSparkGeographyType(elementType)) { + return SparkShimLoader.shim().toPaimonGeography(arrayData, pos); + } return arrayData.getBinary(pos); } diff --git a/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkRow.java b/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkRow.java index 643fc016d940..a8e6ae143baa 100644 --- a/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkRow.java +++ b/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkRow.java @@ -33,6 +33,8 @@ import org.apache.paimon.types.ArrayType; import org.apache.paimon.types.DataType; import org.apache.paimon.types.DateType; +import org.apache.paimon.types.GeographyType; +import org.apache.paimon.types.GeometryType; import org.apache.paimon.types.MapType; import org.apache.paimon.types.RowKind; import org.apache.paimon.types.RowType; @@ -160,6 +162,12 @@ public Timestamp getTimestamp(int i, int precision) { @Override public byte[] getBinary(int i) { + if (type.getTypeAt(i) instanceof GeometryType) { + return SparkShimLoader.shim().toPaimonGeometry(row.getAs(i)); + } + if (type.getTypeAt(i) instanceof GeographyType) { + return SparkShimLoader.shim().toPaimonGeography(row.getAs(i)); + } return row.getAs(i); } @@ -345,6 +353,12 @@ public Timestamp getTimestamp(int i, int precision) { @Override public byte[] getBinary(int i) { + if (elementType instanceof GeometryType) { + return SparkShimLoader.shim().toPaimonGeometry(getAs(i)); + } + if (elementType instanceof GeographyType) { + return SparkShimLoader.shim().toPaimonGeography(getAs(i)); + } return getAs(i); } diff --git a/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/SparkTypeUtils.java b/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/SparkTypeUtils.java index cd7de8f53539..7ced9f29ffee 100644 --- a/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/SparkTypeUtils.java +++ b/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/SparkTypeUtils.java @@ -32,6 +32,8 @@ import org.apache.paimon.types.DecimalType; import org.apache.paimon.types.DoubleType; import org.apache.paimon.types.FloatType; +import org.apache.paimon.types.GeographyType; +import org.apache.paimon.types.GeometryType; import org.apache.paimon.types.IntType; import org.apache.paimon.types.LocalZonedTimestampType; import org.apache.paimon.types.MapType; @@ -261,6 +263,18 @@ public DataType visit(VariantType variantType) { return SparkShimLoader.shim().SparkVariantType(); } + @Override + public DataType visit(GeometryType geometryType) { + return SparkShimLoader.shim().SparkGeometryType(geometryType.getCrs()); + } + + @Override + public DataType visit(GeographyType geographyType) { + return SparkShimLoader.shim() + .SparkGeographyType( + geographyType.getCrs(), geographyType.getAlgorithm().toString()); + } + @Override public DataType visit(ArrayType arrayType) { org.apache.paimon.types.DataType elementType = arrayType.getElementType(); @@ -449,6 +463,13 @@ public org.apache.paimon.types.DataType atomic(DataType atomic) { return new TimestampType(); } else if (SparkShimLoader.shim().isSparkVariantType(atomic)) { return new VariantType(); + } else if (SparkShimLoader.shim().isSparkGeometryType(atomic)) { + return new GeometryType(SparkShimLoader.shim().sparkGeometryCrs(atomic)); + } else if (SparkShimLoader.shim().isSparkGeographyType(atomic)) { + return new GeographyType( + SparkShimLoader.shim().sparkGeographyCrs(atomic), + org.apache.paimon.types.EdgeAlgorithm.fromName( + SparkShimLoader.shim().sparkGeographyAlgorithm(atomic))); } throw new UnsupportedOperationException( diff --git a/paimon-spark/paimon-spark-common/src/main/scala/org/apache/spark/sql/paimon/shims/SparkShim.scala b/paimon-spark/paimon-spark-common/src/main/scala/org/apache/spark/sql/paimon/shims/SparkShim.scala index df21168bd343..690c20b85b74 100644 --- a/paimon-spark/paimon-spark-common/src/main/scala/org/apache/spark/sql/paimon/shims/SparkShim.scala +++ b/paimon-spark/paimon-spark-common/src/main/scala/org/apache/spark/sql/paimon/shims/SparkShim.scala @@ -270,6 +270,37 @@ trait SparkShim { def SparkVariantType(): org.apache.spark.sql.types.DataType + // Geometry and geography are available in Spark 4.1 and later. + def toPaimonGeometry(o: Object): Array[Byte] + + def toPaimonGeometry(row: InternalRow, pos: Int): Array[Byte] + + def toPaimonGeometry(array: ArrayData, pos: Int): Array[Byte] + + def toPaimonGeography(o: Object): Array[Byte] + + def toPaimonGeography(row: InternalRow, pos: Int): Array[Byte] + + def toPaimonGeography(array: ArrayData, pos: Int): Array[Byte] + + def toSparkGeometry(wkb: Array[Byte], crs: String): Object + + def toSparkGeography(wkb: Array[Byte], crs: String, algorithm: String): Object + + def isSparkGeometryType(dataType: org.apache.spark.sql.types.DataType): Boolean + + def isSparkGeographyType(dataType: org.apache.spark.sql.types.DataType): Boolean + + def SparkGeometryType(crs: String): org.apache.spark.sql.types.DataType + + def SparkGeographyType(crs: String, algorithm: String): org.apache.spark.sql.types.DataType + + def sparkGeometryCrs(dataType: org.apache.spark.sql.types.DataType): String + + def sparkGeographyCrs(dataType: org.apache.spark.sql.types.DataType): String + + def sparkGeographyAlgorithm(dataType: org.apache.spark.sql.types.DataType): String + // SQL UDFs (`CREATE FUNCTION ... RETURN ...`) are Spark 4.0+; the spark3 shim no-ops these. /** Parser-stage rule rewriting a Paimon-catalog `CreateUserDefinedFunction` into a create command. */ diff --git a/paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTestBase.scala b/paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTestBase.scala new file mode 100644 index 000000000000..cc45c9b1f04c --- /dev/null +++ b/paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/sql/GeospatialUnsupportedTestBase.scala @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.spark.sql + +import org.apache.paimon.catalog.Identifier +import org.apache.paimon.schema.Schema +import org.apache.paimon.spark.PaimonSparkTestBase +import org.apache.paimon.types.DataTypes + +abstract class GeospatialUnsupportedTestBase extends PaimonSparkTestBase { + + test("Spark SQL rejects geospatial columns before Spark 4.1") { + val identifier = Identifier.create(dbName0, "geospatial_table") + paimonCatalog.createTable( + identifier, + Schema.newBuilder + .column("id", DataTypes.INT()) + .column("geom", DataTypes.GEOMETRY()) + .column("geog", DataTypes.GEOGRAPHY()) + .build, + false + ) + + try { + val error = intercept[UnsupportedOperationException] { + sql("SELECT * FROM geospatial_table") + } + assert(error.getMessage.contains("Geometry and geography require Spark 4.1 or later")) + } finally { + paimonCatalog.dropTable(identifier, true) + } + } +} diff --git a/paimon-spark/paimon-spark3-common/src/main/scala/org/apache/spark/sql/paimon/shims/Spark3Shim.scala b/paimon-spark/paimon-spark3-common/src/main/scala/org/apache/spark/sql/paimon/shims/Spark3Shim.scala index 5ffbf6a14530..b568ce16c3c4 100644 --- a/paimon-spark/paimon-spark3-common/src/main/scala/org/apache/spark/sql/paimon/shims/Spark3Shim.scala +++ b/paimon-spark/paimon-spark3-common/src/main/scala/org/apache/spark/sql/paimon/shims/Spark3Shim.scala @@ -355,6 +355,46 @@ class Spark3Shim extends SparkShim { override def SparkVariantType(): org.apache.spark.sql.types.DataType = throw new UnsupportedOperationException() + override def toPaimonGeometry(o: Object): Array[Byte] = unsupportedGeospatial() + + override def toPaimonGeometry(row: InternalRow, pos: Int): Array[Byte] = unsupportedGeospatial() + + override def toPaimonGeometry(array: ArrayData, pos: Int): Array[Byte] = unsupportedGeospatial() + + override def toPaimonGeography(o: Object): Array[Byte] = unsupportedGeospatial() + + override def toPaimonGeography(row: InternalRow, pos: Int): Array[Byte] = unsupportedGeospatial() + + override def toPaimonGeography(array: ArrayData, pos: Int): Array[Byte] = unsupportedGeospatial() + + override def toSparkGeometry(wkb: Array[Byte], crs: String): Object = unsupportedGeospatial() + + override def toSparkGeography(wkb: Array[Byte], crs: String, algorithm: String): Object = + unsupportedGeospatial() + + override def isSparkGeometryType(dataType: org.apache.spark.sql.types.DataType): Boolean = false + + override def isSparkGeographyType(dataType: org.apache.spark.sql.types.DataType): Boolean = false + + override def SparkGeometryType(crs: String): org.apache.spark.sql.types.DataType = + unsupportedGeospatial() + + override def SparkGeographyType( + crs: String, + algorithm: String): org.apache.spark.sql.types.DataType = unsupportedGeospatial() + + override def sparkGeometryCrs(dataType: org.apache.spark.sql.types.DataType): String = + unsupportedGeospatial() + + override def sparkGeographyCrs(dataType: org.apache.spark.sql.types.DataType): String = + unsupportedGeospatial() + + override def sparkGeographyAlgorithm(dataType: org.apache.spark.sql.types.DataType): String = + unsupportedGeospatial() + + private def unsupportedGeospatial[T](): T = + throw new UnsupportedOperationException("Geometry and geography require Spark 4.1 or later") + override def toPaimonVariant(row: InternalRow, pos: Int): Variant = throw new UnsupportedOperationException() diff --git a/paimon-spark/paimon-spark4-common/src/main/scala/org/apache/paimon/spark/data/Spark4ArrayData.scala b/paimon-spark/paimon-spark4-common/src/main/scala/org/apache/paimon/spark/data/Spark4ArrayData.scala index b6904d86cf39..80e0456568e5 100644 --- a/paimon-spark/paimon-spark4-common/src/main/scala/org/apache/paimon/spark/data/Spark4ArrayData.scala +++ b/paimon-spark/paimon-spark4-common/src/main/scala/org/apache/paimon/spark/data/Spark4ArrayData.scala @@ -18,8 +18,9 @@ package org.apache.paimon.spark.data -import org.apache.paimon.types.DataType +import org.apache.paimon.types.{DataType, GeographyType, GeometryType} +import org.apache.spark.sql.paimon.shims.SparkShimLoader import org.apache.spark.unsafe.types.{GeographyVal, GeometryVal, VariantVal} class Spark4ArrayData(override val elementType: DataType) extends AbstractSparkArrayData { @@ -30,8 +31,17 @@ class Spark4ArrayData(override val elementType: DataType) extends AbstractSparkA } override def getGeography(ordinal: Int): GeographyVal = - throw new UnsupportedOperationException("Paimon does not support Geography type") + SparkShimLoader.shim + .toSparkGeography( + paimonArray.getBinary(ordinal), + elementType.asInstanceOf[GeographyType].getCrs, + elementType.asInstanceOf[GeographyType].getAlgorithm.toString) + .asInstanceOf[GeographyVal] override def getGeometry(ordinal: Int): GeometryVal = - throw new UnsupportedOperationException("Paimon does not support Geometry type") + SparkShimLoader.shim + .toSparkGeometry( + paimonArray.getBinary(ordinal), + elementType.asInstanceOf[GeometryType].getCrs) + .asInstanceOf[GeometryVal] } diff --git a/paimon-spark/paimon-spark4-common/src/main/scala/org/apache/paimon/spark/data/Spark4InternalRow.scala b/paimon-spark/paimon-spark4-common/src/main/scala/org/apache/paimon/spark/data/Spark4InternalRow.scala index ea73692c689c..dc54eb4c6094 100644 --- a/paimon-spark/paimon-spark4-common/src/main/scala/org/apache/paimon/spark/data/Spark4InternalRow.scala +++ b/paimon-spark/paimon-spark4-common/src/main/scala/org/apache/paimon/spark/data/Spark4InternalRow.scala @@ -19,8 +19,9 @@ package org.apache.paimon.spark.data import org.apache.paimon.spark.AbstractSparkInternalRow -import org.apache.paimon.types.RowType +import org.apache.paimon.types.{GeographyType, GeometryType, RowType} +import org.apache.spark.sql.paimon.shims.SparkShimLoader import org.apache.spark.unsafe.types.{GeographyVal, GeometryVal, VariantVal} class Spark4InternalRow(rowType: RowType) extends AbstractSparkInternalRow(rowType) { @@ -31,8 +32,22 @@ class Spark4InternalRow(rowType: RowType) extends AbstractSparkInternalRow(rowTy } override def getGeography(ordinal: Int): GeographyVal = - throw new UnsupportedOperationException("Paimon does not support Geography type") + SparkShimLoader.shim + .toSparkGeography( + row.getBinary(ordinal), + rowType.getTypeAt(ordinal).asInstanceOf[GeographyType].getCrs, + rowType + .getTypeAt(ordinal) + .asInstanceOf[GeographyType] + .getAlgorithm + .toString + ) + .asInstanceOf[GeographyVal] override def getGeometry(ordinal: Int): GeometryVal = - throw new UnsupportedOperationException("Paimon does not support Geometry type") + SparkShimLoader.shim + .toSparkGeometry( + row.getBinary(ordinal), + rowType.getTypeAt(ordinal).asInstanceOf[GeometryType].getCrs) + .asInstanceOf[GeometryVal] } diff --git a/paimon-spark/paimon-spark4-common/src/main/scala/org/apache/spark/sql/paimon/shims/Spark4Shim.scala b/paimon-spark/paimon-spark4-common/src/main/scala/org/apache/spark/sql/paimon/shims/Spark4Shim.scala index 00e0b1ae4ff0..eea503d7d5ce 100644 --- a/paimon-spark/paimon-spark4-common/src/main/scala/org/apache/spark/sql/paimon/shims/Spark4Shim.scala +++ b/paimon-spark/paimon-spark4-common/src/main/scala/org/apache/spark/sql/paimon/shims/Spark4Shim.scala @@ -41,7 +41,7 @@ import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Assignment, Colum import org.apache.spark.sql.catalyst.plans.logical.MergeRows.{Copy, Insert, Keep, Update} import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, Distribution} import org.apache.spark.sql.catalyst.rules.Rule -import org.apache.spark.sql.catalyst.util.{ArrayData, GeneratedColumn, IdentityColumn, ResolveDefaultColumns} +import org.apache.spark.sql.catalyst.util.{ArrayData, GeneratedColumn, IdentityColumn, ResolveDefaultColumns, STUtils} import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Column, Identifier, StagingTableCatalog, Table, TableCatalog} import org.apache.spark.sql.connector.expressions.Transform import org.apache.spark.sql.connector.read.Scan @@ -53,7 +53,7 @@ import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2Relation, Data import org.apache.spark.sql.execution.streaming.runtime.MetadataLogFileIndex import org.apache.spark.sql.execution.streaming.sinks.FileStreamSink import org.apache.spark.sql.internal.SQLConf -import org.apache.spark.sql.types.{DataTypes, StructType, VariantType} +import org.apache.spark.sql.types.{DataTypes, Geography, GeographyType, Geometry, GeometryType, StructType, VariantType} import org.apache.spark.unsafe.types.VariantVal import java.util.{Map => JMap} @@ -359,6 +359,74 @@ class Spark4Shim extends SparkShim { override def SparkVariantType(): org.apache.spark.sql.types.DataType = DataTypes.VariantType + override def toPaimonGeometry(o: Object): Array[Byte] = + o.asInstanceOf[Geometry].getBytes + + override def toPaimonGeometry(row: InternalRow, pos: Int): Array[Byte] = + STUtils.stAsBinary(row.getGeometry(pos)) + + override def toPaimonGeometry(array: ArrayData, pos: Int): Array[Byte] = + STUtils.stAsBinary(array.getGeometry(pos)) + + override def toPaimonGeography(o: Object): Array[Byte] = + o.asInstanceOf[Geography].getBytes + + override def toPaimonGeography(row: InternalRow, pos: Int): Array[Byte] = + STUtils.stAsBinary(row.getGeography(pos)) + + override def toPaimonGeography(array: ArrayData, pos: Int): Array[Byte] = + STUtils.stAsBinary(array.getGeography(pos)) + + override def toSparkGeometry(wkb: Array[Byte], crs: String): Object = { + val geometryType = sparkGeometryType(crs) + STUtils.stGeomFromWKB(wkb, geometryType.srid) + } + + override def toSparkGeography(wkb: Array[Byte], crs: String, algorithm: String): Object = { + val geographyType = sparkGeographyType(crs, algorithm) + STUtils.stSetSrid(STUtils.stGeogFromWKB(wkb), geographyType.srid) + } + + override def isSparkGeometryType(dataType: org.apache.spark.sql.types.DataType): Boolean = + dataType.isInstanceOf[GeometryType] + + override def isSparkGeographyType(dataType: org.apache.spark.sql.types.DataType): Boolean = + dataType.isInstanceOf[GeographyType] + + override def SparkGeometryType(crs: String): org.apache.spark.sql.types.DataType = + sparkGeometryType(crs) + + override def SparkGeographyType( + crs: String, + algorithm: String): org.apache.spark.sql.types.DataType = sparkGeographyType(crs, algorithm) + + override def sparkGeometryCrs(dataType: org.apache.spark.sql.types.DataType): String = { + val geometryType = dataType.asInstanceOf[GeometryType] + require(!geometryType.isMixedSrid, "Paimon does not support mixed-SRID geometry values") + geometryType.crs + } + + override def sparkGeographyCrs(dataType: org.apache.spark.sql.types.DataType): String = { + val geographyType = dataType.asInstanceOf[GeographyType] + require(!geographyType.isMixedSrid, "Paimon does not support mixed-SRID geography values") + geographyType.crs + } + + override def sparkGeographyAlgorithm(dataType: org.apache.spark.sql.types.DataType): String = + dataType.asInstanceOf[GeographyType].algorithm.toString + + private def sparkGeometryType(crs: String): GeometryType = { + val geometryType = GeometryType(crs) + require(!geometryType.isMixedSrid, "Paimon does not support mixed-SRID geometry values") + geometryType + } + + private def sparkGeographyType(crs: String, algorithm: String): GeographyType = { + val geographyType = GeographyType(crs, algorithm) + require(!geographyType.isMixedSrid, "Paimon does not support mixed-SRID geography values") + geographyType + } + // SQL UDFs (CREATE FUNCTION ... RETURN ...). override def rewritePaimonSQLFunctionCommands(spark: SparkSession): Rule[LogicalPlan] = org.apache.spark.sql.catalyst.parser.extensions.RewritePaimonSQLFunctionCommands(spark)