diff --git a/CHANGELOG.md b/CHANGELOG.md index 71627928..45f8f30f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,15 @@ All notable changes to this project will be documented in this file. ## [Unreleased] +### Added + +- Document that the Stackable Hive images restore the `get_table` and `get_table_objects_by_name` Thrift methods removed by HIVE-26537, and assert it in the smoke test ([#766]). +- Add support for Hive `4.2.1` ([#766]). + +### Removed + +- Remove support for Hive `4.0.1` ([#766]). + ### Changed - Internal operator refactoring: introduce a build() step in the reconciler that @@ -45,6 +54,7 @@ All notable changes to this project will be documented in this file. [#754]: https://github.com/stackabletech/hive-operator/pull/754 [#759]: https://github.com/stackabletech/hive-operator/pull/759 [#764]: https://github.com/stackabletech/hive-operator/pull/764 +[#766]: https://github.com/stackabletech/hive-operator/pull/766 ## [26.7.0] - 2026-07-21 diff --git a/docs/modules/hive/pages/usage-guide/thrift-api-compatibility.adoc b/docs/modules/hive/pages/usage-guide/thrift-api-compatibility.adoc new file mode 100644 index 00000000..29d33941 --- /dev/null +++ b/docs/modules/hive/pages/usage-guide/thrift-api-compatibility.adoc @@ -0,0 +1,34 @@ += Thrift API compatibility +:description: Stackable Hive images restore two Thrift metastore methods that Hive 4.0.1 removed, so that Spark and Iceberg clients keep working. +:hive-26537: https://issues.apache.org/jira/browse/HIVE-26537 +:iceberg-12878: https://github.com/apache/iceberg/issues/12878 + +Stackable images for Apache Hive serve two metastore Thrift methods that upstream Hive removed in 4.0.1: + +* `get_table` +* `get_table_objects_by_name` + +{hive-26537}[HIVE-26537] deleted them in favour of the request-based `get_table_req` and `get_table_objects_by_name_req`, which take a request struct so that fields can be added without breaking the wire. + +== Why they are restored + +Any metastore client built against Hive 2.3 still calls the removed methods, and Hive 2.3 is what Apache Spark bundles: 2.3.9 in Spark 3.5, and 2.3.10 in Spark 4.0 through 4.2. +Apache Iceberg's `HiveCatalog` reaches the metastore through `IMetaStoreClient`, so it uses whichever client Spark supplies. +Against an unpatched Hive 4.0.1 or newer metastore, an Iceberg job therefore fails with: + +[source] +---- +TApplicationException: Invalid method name: 'get_table' +---- + +This is tracked upstream in {iceberg-12878}[apache/iceberg#12878]. +Restoring the two methods keeps Spark and Iceberg working against a current Hive metastore. + +[IMPORTANT] +==== +This makes the Apache Hive metastore shipped by Stackable more permissive than an upstream Hive 4 metastore. +==== + +== Versions + +All versions of Hive shipped by Stackable contain these methods. diff --git a/docs/modules/hive/partials/nav.adoc b/docs/modules/hive/partials/nav.adoc index 43783a13..2b061e3d 100644 --- a/docs/modules/hive/partials/nav.adoc +++ b/docs/modules/hive/partials/nav.adoc @@ -7,6 +7,7 @@ ** xref:hive:usage-guide/data-storage.adoc[] ** xref:hive:usage-guide/derby-example.adoc[] ** xref:hive:usage-guide/database-driver.adoc[] +** xref:hive:usage-guide/thrift-api-compatibility.adoc[] ** xref:hive:usage-guide/logging.adoc[] ** xref:hive:usage-guide/monitoring.adoc[] ** xref:hive:usage-guide/resources.adoc[] diff --git a/docs/modules/hive/partials/supported-versions.adoc b/docs/modules/hive/partials/supported-versions.adoc index be77923d..acf1d443 100644 --- a/docs/modules/hive/partials/supported-versions.adoc +++ b/docs/modules/hive/partials/supported-versions.adoc @@ -2,23 +2,31 @@ // This is a separate file, since it is used by both the direct Hive-Operator documentation, and the overarching // Stackable Platform documentation. -- 4.2.0 (LTS) -- 4.0.1 (deprecated) +- 4.2.1 (LTS) +- 4.2.0 (deprecated) - 4.0.0 (deprecated) - 3.1.3 (deprecated) -=== Hive 4 issues +=== Hive 4 compatibility -Hive 4 has known compatibility issues, especially when using it with Iceberg or Trino. -The missing compatibility with Iceberg also means that Spark jobs using this storage may fail. +Hive 4.0.1 removed two metastore Thrift methods that clients built against Hive 2.3 still call. +Apache Spark bundles such a client, and Apache Iceberg's `HiveCatalog` uses it, so Spark jobs writing Iceberg tables fail against a metastore that no longer serves them. -Be aware of upgrading Hive (e.g. 4.0.0 to 4.0.1 or 4.0.1 to Hive 4.1.0), as this upgrade is not easily reversible. -Test the new version before upgrading your production workloads and take backups of your database before starting an upgrade. +Every Hive version shipped by Stackable serves both methods. +3.1.3 and 4.0.0 predate the removal. +The images for 4.2.0 and later restore it. +See xref:hive:usage-guide/thrift-api-compatibility.adoc[] for what that means in practice. -**Workaround:** If you encounter issues with Hive 4.x, use Hive 3.1.3 instead until these upstream issues are resolved. +=== Upgrading + +Upgrading Hive, for example from 4.0.0 to 4.2.1, is not easily reversible. +Test the new version before upgrading your production workloads, and back up the metastore database before starting an upgrade. + +Hive 3.1.3 and 4.0.0 are deprecated and both are scheduled for removal in SDP 27.3. For more details, see: * https://github.com/stackabletech/hive-operator/issues/626[Our Stackable tracking issue] -* https://github.com/trinodb/trino/issues/26214[Trino: Repeated ANALYZE fails on Hive metastore 4.0.x] +* https://issues.apache.org/jira/browse/HIVE-26537[HIVE-26537: the change that removed the Thrift methods] * https://github.com/apache/iceberg/issues/12878[Iceberg: org.apache.thrift.TApplicationException: Invalid method name: 'get_table'] +* https://github.com/trinodb/trino/issues/26214[Trino: Repeated ANALYZE fails on Hive metastore 4.0.x] diff --git a/tests/templates/kuttl/smoke/test_metastore.py b/tests/templates/kuttl/smoke/test_metastore.py index 2ffaf6e8..8cc33bbd 100755 --- a/tests/templates/kuttl/smoke/test_metastore.py +++ b/tests/templates/kuttl/smoke/test_metastore.py @@ -64,6 +64,33 @@ def check_table(hive_client, db_name, table_name, location, label): ).table +def check_legacy_get_table(hive_client, db_name, table_name): + """Assert the metastore still answers the pre-Hive-4.0.1 `get_table` Thrift call. + + HIVE-26537 deleted `get_table` and `get_table_objects_by_name` from the Thrift + interface in Hive 4.0.1 and at the time of this writing (2026-09) Spark still + ships with a bundled Hive that requires these methods. + """ + legacy = hive_client.get_table(dbname=db_name, tbl_name=table_name) + if legacy.tableName != table_name: + print( + f"[ERROR]: Legacy get_table returned table {legacy.tableName} - expected {table_name}" + ) + exit(-1) + + legacy_tables = hive_client.get_table_objects_by_name( + dbname=db_name, tbl_names=[table_name] + ) + if [t.tableName for t in legacy_tables] != [table_name]: + print( + f"[ERROR]: Legacy get_table_objects_by_name returned " + f"{[t.tableName for t in legacy_tables]} - expected {[table_name]}" + ) + exit(-1) + + print("[INFO]: Metastore serves the legacy get_table Thrift methods") + + if __name__ == "__main__": all_args = argparse.ArgumentParser(description="Test hive metastore.") all_args.add_argument("-p", "--port", help="Metastore server port", default="9083") @@ -97,6 +124,8 @@ def check_table(hive_client, db_name, table_name, location, label): "local", ) + check_legacy_get_table(hive_client, database_name, local_test_table_name) + # S3 access check_table( hive_client, diff --git a/tests/test-definition.yaml b/tests/test-definition.yaml index 95d1e2d0..9b1b27b0 100644 --- a/tests/test-definition.yaml +++ b/tests/test-definition.yaml @@ -18,14 +18,14 @@ dimensions: values: - 3.1.3 - 4.0.0 - - 4.0.1 - 4.2.0 + - 4.2.1 # Alternatively, if you want to use a custom image, append a comma and the full image name to the product version # as in the example below. # - 4.2.0,oci.stackable.tech/sdp/hive:4.2.0-stackable0.0.0-dev - name: hive-latest values: - - 4.2.0 + - 4.2.1 # Alternatively, if you want to use a custom image, append a comma and the full image name to the product version # as in the example below. # - 4.2.0,oci.stackable.tech/sdp/hive:4.2.0-stackable0.0.0-dev @@ -34,7 +34,7 @@ dimensions: - 3.1.3 - name: hive-new values: - - 4.2.0 + - 4.2.1 - name: hdfs-latest values: - 3.5.0