From eb41bd4598a2262e0aa6161be4819fff64cbd3ed Mon Sep 17 00:00:00 2001 From: Lars Francke Date: Fri, 11 Sep 2026 12:10:47 +0200 Subject: [PATCH] feat: Assert that the metastore serves the legacy get_table Thrift methods Hive 4.0.1 removed get_table and get_table_objects_by_name from the metastore Thrift interface (HIVE-26537). Clients built against a Hive 2.3 metastore client still call them, which is what Spark bundles, so Iceberg's HiveCatalog could not talk to a Hive 4.0.1 or newer metastore. Our images for 4.2.0 restore both methods. The smoke test now asserts that. Without it the patch could be dropped during a Hive bump and nobody would notice until it surfaced in somebody's Spark job; with it the test fails with "Invalid method name: 'get_table'" at the point the image changes. It runs unconditionally: every version we ship serves both methods, either because it predates the removal (3.1.3, 4.0.0) or because it carries the patch (4.2.0 and later). 4.0.1, the one exception, is removed in docker-images. Also documents the divergence. Our metastore is deliberately more permissive than an upstream Hive 4 metastore, which matters to anyone planning to move a workload off SDP, so it gets a usage-guide page rather than a footnote. The supported-versions page previously told users to fall back to Hive 3.1.3 because of the Iceberg and Trino problems; the Iceberg half is fixed by the restored methods and the Trino half by Trino 480, so it now says what is actually true. The supported versions change with it: 4.2.1 is added and becomes the LTS, 4.2.0 is deprecated, 4.0.1 is dropped, and 4.0.0 stays deprecated for one more release so that a platform upgrade and a Hive upgrade need not happen in the same window. Verified on k3d against an image built from the patches: smoke (Postgres), kerberos-hdfs and kerberos-s3 all pass, and the new assertion fails as intended against the unpatched image. --- CHANGELOG.md | 10 ++++++ .../usage-guide/thrift-api-compatibility.adoc | 34 +++++++++++++++++++ docs/modules/hive/partials/nav.adoc | 1 + .../hive/partials/supported-versions.adoc | 26 +++++++++----- tests/templates/kuttl/smoke/test_metastore.py | 29 ++++++++++++++++ tests/test-definition.yaml | 6 ++-- 6 files changed, 94 insertions(+), 12 deletions(-) create mode 100644 docs/modules/hive/pages/usage-guide/thrift-api-compatibility.adoc diff --git a/CHANGELOG.md b/CHANGELOG.md index 71627928..45f8f30f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,15 @@ All notable changes to this project will be documented in this file. ## [Unreleased] +### Added + +- Document that the Stackable Hive images restore the `get_table` and `get_table_objects_by_name` Thrift methods removed by HIVE-26537, and assert it in the smoke test ([#766]). +- Add support for Hive `4.2.1` ([#766]). + +### Removed + +- Remove support for Hive `4.0.1` ([#766]). + ### Changed - Internal operator refactoring: introduce a build() step in the reconciler that @@ -45,6 +54,7 @@ All notable changes to this project will be documented in this file. [#754]: https://github.com/stackabletech/hive-operator/pull/754 [#759]: https://github.com/stackabletech/hive-operator/pull/759 [#764]: https://github.com/stackabletech/hive-operator/pull/764 +[#766]: https://github.com/stackabletech/hive-operator/pull/766 ## [26.7.0] - 2026-07-21 diff --git a/docs/modules/hive/pages/usage-guide/thrift-api-compatibility.adoc b/docs/modules/hive/pages/usage-guide/thrift-api-compatibility.adoc new file mode 100644 index 00000000..29d33941 --- /dev/null +++ b/docs/modules/hive/pages/usage-guide/thrift-api-compatibility.adoc @@ -0,0 +1,34 @@ += Thrift API compatibility +:description: Stackable Hive images restore two Thrift metastore methods that Hive 4.0.1 removed, so that Spark and Iceberg clients keep working. +:hive-26537: https://issues.apache.org/jira/browse/HIVE-26537 +:iceberg-12878: https://github.com/apache/iceberg/issues/12878 + +Stackable images for Apache Hive serve two metastore Thrift methods that upstream Hive removed in 4.0.1: + +* `get_table` +* `get_table_objects_by_name` + +{hive-26537}[HIVE-26537] deleted them in favour of the request-based `get_table_req` and `get_table_objects_by_name_req`, which take a request struct so that fields can be added without breaking the wire. + +== Why they are restored + +Any metastore client built against Hive 2.3 still calls the removed methods, and Hive 2.3 is what Apache Spark bundles: 2.3.9 in Spark 3.5, and 2.3.10 in Spark 4.0 through 4.2. +Apache Iceberg's `HiveCatalog` reaches the metastore through `IMetaStoreClient`, so it uses whichever client Spark supplies. +Against an unpatched Hive 4.0.1 or newer metastore, an Iceberg job therefore fails with: + +[source] +---- +TApplicationException: Invalid method name: 'get_table' +---- + +This is tracked upstream in {iceberg-12878}[apache/iceberg#12878]. +Restoring the two methods keeps Spark and Iceberg working against a current Hive metastore. + +[IMPORTANT] +==== +This makes the Apache Hive metastore shipped by Stackable more permissive than an upstream Hive 4 metastore. +==== + +== Versions + +All versions of Hive shipped by Stackable contain these methods. diff --git a/docs/modules/hive/partials/nav.adoc b/docs/modules/hive/partials/nav.adoc index 43783a13..2b061e3d 100644 --- a/docs/modules/hive/partials/nav.adoc +++ b/docs/modules/hive/partials/nav.adoc @@ -7,6 +7,7 @@ ** xref:hive:usage-guide/data-storage.adoc[] ** xref:hive:usage-guide/derby-example.adoc[] ** xref:hive:usage-guide/database-driver.adoc[] +** xref:hive:usage-guide/thrift-api-compatibility.adoc[] ** xref:hive:usage-guide/logging.adoc[] ** xref:hive:usage-guide/monitoring.adoc[] ** xref:hive:usage-guide/resources.adoc[] diff --git a/docs/modules/hive/partials/supported-versions.adoc b/docs/modules/hive/partials/supported-versions.adoc index be77923d..acf1d443 100644 --- a/docs/modules/hive/partials/supported-versions.adoc +++ b/docs/modules/hive/partials/supported-versions.adoc @@ -2,23 +2,31 @@ // This is a separate file, since it is used by both the direct Hive-Operator documentation, and the overarching // Stackable Platform documentation. -- 4.2.0 (LTS) -- 4.0.1 (deprecated) +- 4.2.1 (LTS) +- 4.2.0 (deprecated) - 4.0.0 (deprecated) - 3.1.3 (deprecated) -=== Hive 4 issues +=== Hive 4 compatibility -Hive 4 has known compatibility issues, especially when using it with Iceberg or Trino. -The missing compatibility with Iceberg also means that Spark jobs using this storage may fail. +Hive 4.0.1 removed two metastore Thrift methods that clients built against Hive 2.3 still call. +Apache Spark bundles such a client, and Apache Iceberg's `HiveCatalog` uses it, so Spark jobs writing Iceberg tables fail against a metastore that no longer serves them. -Be aware of upgrading Hive (e.g. 4.0.0 to 4.0.1 or 4.0.1 to Hive 4.1.0), as this upgrade is not easily reversible. -Test the new version before upgrading your production workloads and take backups of your database before starting an upgrade. +Every Hive version shipped by Stackable serves both methods. +3.1.3 and 4.0.0 predate the removal. +The images for 4.2.0 and later restore it. +See xref:hive:usage-guide/thrift-api-compatibility.adoc[] for what that means in practice. -**Workaround:** If you encounter issues with Hive 4.x, use Hive 3.1.3 instead until these upstream issues are resolved. +=== Upgrading + +Upgrading Hive, for example from 4.0.0 to 4.2.1, is not easily reversible. +Test the new version before upgrading your production workloads, and back up the metastore database before starting an upgrade. + +Hive 3.1.3 and 4.0.0 are deprecated and both are scheduled for removal in SDP 27.3. For more details, see: * https://github.com/stackabletech/hive-operator/issues/626[Our Stackable tracking issue] -* https://github.com/trinodb/trino/issues/26214[Trino: Repeated ANALYZE fails on Hive metastore 4.0.x] +* https://issues.apache.org/jira/browse/HIVE-26537[HIVE-26537: the change that removed the Thrift methods] * https://github.com/apache/iceberg/issues/12878[Iceberg: org.apache.thrift.TApplicationException: Invalid method name: 'get_table'] +* https://github.com/trinodb/trino/issues/26214[Trino: Repeated ANALYZE fails on Hive metastore 4.0.x] diff --git a/tests/templates/kuttl/smoke/test_metastore.py b/tests/templates/kuttl/smoke/test_metastore.py index 2ffaf6e8..8cc33bbd 100755 --- a/tests/templates/kuttl/smoke/test_metastore.py +++ b/tests/templates/kuttl/smoke/test_metastore.py @@ -64,6 +64,33 @@ def check_table(hive_client, db_name, table_name, location, label): ).table +def check_legacy_get_table(hive_client, db_name, table_name): + """Assert the metastore still answers the pre-Hive-4.0.1 `get_table` Thrift call. + + HIVE-26537 deleted `get_table` and `get_table_objects_by_name` from the Thrift + interface in Hive 4.0.1 and at the time of this writing (2026-09) Spark still + ships with a bundled Hive that requires these methods. + """ + legacy = hive_client.get_table(dbname=db_name, tbl_name=table_name) + if legacy.tableName != table_name: + print( + f"[ERROR]: Legacy get_table returned table {legacy.tableName} - expected {table_name}" + ) + exit(-1) + + legacy_tables = hive_client.get_table_objects_by_name( + dbname=db_name, tbl_names=[table_name] + ) + if [t.tableName for t in legacy_tables] != [table_name]: + print( + f"[ERROR]: Legacy get_table_objects_by_name returned " + f"{[t.tableName for t in legacy_tables]} - expected {[table_name]}" + ) + exit(-1) + + print("[INFO]: Metastore serves the legacy get_table Thrift methods") + + if __name__ == "__main__": all_args = argparse.ArgumentParser(description="Test hive metastore.") all_args.add_argument("-p", "--port", help="Metastore server port", default="9083") @@ -97,6 +124,8 @@ def check_table(hive_client, db_name, table_name, location, label): "local", ) + check_legacy_get_table(hive_client, database_name, local_test_table_name) + # S3 access check_table( hive_client, diff --git a/tests/test-definition.yaml b/tests/test-definition.yaml index 95d1e2d0..9b1b27b0 100644 --- a/tests/test-definition.yaml +++ b/tests/test-definition.yaml @@ -18,14 +18,14 @@ dimensions: values: - 3.1.3 - 4.0.0 - - 4.0.1 - 4.2.0 + - 4.2.1 # Alternatively, if you want to use a custom image, append a comma and the full image name to the product version # as in the example below. # - 4.2.0,oci.stackable.tech/sdp/hive:4.2.0-stackable0.0.0-dev - name: hive-latest values: - - 4.2.0 + - 4.2.1 # Alternatively, if you want to use a custom image, append a comma and the full image name to the product version # as in the example below. # - 4.2.0,oci.stackable.tech/sdp/hive:4.2.0-stackable0.0.0-dev @@ -34,7 +34,7 @@ dimensions: - 3.1.3 - name: hive-new values: - - 4.2.0 + - 4.2.1 - name: hdfs-latest values: - 3.5.0