feat: Assert that the metastore serves the legacy get_table Thrift methods

CHANGELOG.md

@@ -4,6 +4,15 @@ All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
 
+### Added
+
+- Document that the Stackable Hive images restore the `get_table` and `get_table_objects_by_name` Thrift methods removed by HIVE-26537, and assert it in the smoke test ([#766]).
+- Add support for Hive `4.2.1` ([#766]).
+
+### Removed
+
+- Remove support for Hive `4.0.1` ([#766]).
+
 ### Changed
 
 - Internal operator refactoring: introduce a build() step in the reconciler that
@@ -45,6 +54,7 @@ All notable changes to this project will be documented in this file.
 [#754]: https://github.com/stackabletech/hive-operator/pull/754
 [#759]: https://github.com/stackabletech/hive-operator/pull/759
 [#764]: https://github.com/stackabletech/hive-operator/pull/764
+[#766]: https://github.com/stackabletech/hive-operator/pull/766
 
 ## [26.7.0] - 2026-07-21
 

docs/modules/hive/pages/usage-guide/thrift-api-compatibility.adoc

@@ -0,0 +1,34 @@
+= Thrift API compatibility
+:description: Stackable Hive images restore two Thrift metastore methods that Hive 4.0.1 removed, so that Spark and Iceberg clients keep working.
+:hive-26537: https://issues.apache.org/jira/browse/HIVE-26537
+:iceberg-12878: https://github.com/apache/iceberg/issues/12878
+
+Stackable images for Apache Hive serve two metastore Thrift methods that upstream Hive removed in 4.0.1:
+
+* `get_table`
+* `get_table_objects_by_name`
+
+{hive-26537}[HIVE-26537] deleted them in favour of the request-based `get_table_req` and `get_table_objects_by_name_req`, which take a request struct so that fields can be added without breaking the wire.
+
+== Why they are restored
+
+Any metastore client built against Hive 2.3 still calls the removed methods, and Hive 2.3 is what Apache Spark bundles: 2.3.9 in Spark 3.5, and 2.3.10 in Spark 4.0 through 4.2.
+Apache Iceberg's `HiveCatalog` reaches the metastore through `IMetaStoreClient`, so it uses whichever client Spark supplies.
+Against an unpatched Hive 4.0.1 or newer metastore, an Iceberg job therefore fails with:
+
+[source]
+----
+TApplicationException: Invalid method name: 'get_table'
+----
+
+This is tracked upstream in {iceberg-12878}[apache/iceberg#12878].
+Restoring the two methods keeps Spark and Iceberg working against a current Hive metastore.
+
+[IMPORTANT]
+====
+This makes the Apache Hive metastore shipped by Stackable more permissive than an upstream Hive 4 metastore.
+====
+
+== Versions
+
+All versions of Hive shipped by Stackable contain these methods.

docs/modules/hive/partials/nav.adoc

@@ -7,6 +7,7 @@
 ** xref:hive:usage-guide/data-storage.adoc[]
 ** xref:hive:usage-guide/derby-example.adoc[]
 ** xref:hive:usage-guide/database-driver.adoc[]
+** xref:hive:usage-guide/thrift-api-compatibility.adoc[]
 ** xref:hive:usage-guide/logging.adoc[]
 ** xref:hive:usage-guide/monitoring.adoc[]
 ** xref:hive:usage-guide/resources.adoc[]

docs/modules/hive/partials/supported-versions.adoc

@@ -2,23 +2,31 @@
 // This is a separate file, since it is used by both the direct Hive-Operator documentation, and the overarching
 // Stackable Platform documentation.
 
-- 4.2.0 (LTS)
-- 4.0.1 (deprecated)
+- 4.2.1 (LTS)
+- 4.2.0 (deprecated)
 - 4.0.0 (deprecated)
 - 3.1.3 (deprecated)
 
-=== Hive 4 issues
+=== Hive 4 compatibility
 
-Hive 4 has known compatibility issues, especially when using it with Iceberg or Trino.
-The missing compatibility with Iceberg also means that Spark jobs using this storage may fail.
+Hive 4.0.1 removed two metastore Thrift methods that clients built against Hive 2.3 still call.
+Apache Spark bundles such a client, and Apache Iceberg's `HiveCatalog` uses it, so Spark jobs writing Iceberg tables fail against a metastore that no longer serves them.
 
-Be aware of upgrading Hive (e.g. 4.0.0 to 4.0.1 or 4.0.1 to Hive 4.1.0), as this upgrade is not easily reversible.
-Test the new version before upgrading your production workloads and take backups of your database before starting an upgrade.
+Every Hive version shipped by Stackable serves both methods.
+3.1.3 and 4.0.0 predate the removal.
+The images for 4.2.0 and later restore it.
+See xref:hive:usage-guide/thrift-api-compatibility.adoc[] for what that means in practice.
 
-**Workaround:** If you encounter issues with Hive 4.x, use Hive 3.1.3 instead until these upstream issues are resolved.
+=== Upgrading
+
+Upgrading Hive, for example from 4.0.0 to 4.2.1, is not easily reversible.
+Test the new version before upgrading your production workloads, and back up the metastore database before starting an upgrade.
+
+Hive 3.1.3 and 4.0.0 are deprecated and both are scheduled for removal in SDP 27.3.
 
 For more details, see:
 
 * https://github.com/stackabletech/hive-operator/issues/626[Our Stackable tracking issue]
-* https://github.com/trinodb/trino/issues/26214[Trino: Repeated ANALYZE fails on Hive metastore 4.0.x]
+* https://issues.apache.org/jira/browse/HIVE-26537[HIVE-26537: the change that removed the Thrift methods]
 * https://github.com/apache/iceberg/issues/12878[Iceberg: org.apache.thrift.TApplicationException: Invalid method name: 'get_table']
+* https://github.com/trinodb/trino/issues/26214[Trino: Repeated ANALYZE fails on Hive metastore 4.0.x]

tests/templates/kuttl/smoke/test_metastore.py

@@ -64,6 +64,33 @@ def check_table(hive_client, db_name, table_name, location, label):
     ).table
 
 
+def check_legacy_get_table(hive_client, db_name, table_name):
+    """Assert the metastore still answers the pre-Hive-4.0.1 `get_table` Thrift call.
+
+    HIVE-26537 deleted `get_table` and `get_table_objects_by_name` from the Thrift
+    interface in Hive 4.0.1 and at the time of this writing (2026-09) Spark still
+    ships with a bundled Hive that requires these methods.
+    """
+    legacy = hive_client.get_table(dbname=db_name, tbl_name=table_name)
+    if legacy.tableName != table_name:
+        print(
+            f"[ERROR]: Legacy get_table returned table {legacy.tableName} - expected {table_name}"
+        )
+        exit(-1)
+
+    legacy_tables = hive_client.get_table_objects_by_name(
+        dbname=db_name, tbl_names=[table_name]
+    )
+    if [t.tableName for t in legacy_tables] != [table_name]:
+        print(
+            f"[ERROR]: Legacy get_table_objects_by_name returned "
+            f"{[t.tableName for t in legacy_tables]} - expected {[table_name]}"
+        )
+        exit(-1)
+
+    print("[INFO]: Metastore serves the legacy get_table Thrift methods")
+
+
 if __name__ == "__main__":
     all_args = argparse.ArgumentParser(description="Test hive metastore.")
     all_args.add_argument("-p", "--port", help="Metastore server port", default="9083")
@@ -97,6 +124,8 @@ def check_table(hive_client, db_name, table_name, location, label):
             "local",
         )
 
+        check_legacy_get_table(hive_client, database_name, local_test_table_name)
+
         # S3 access
         check_table(
             hive_client,

tests/test-definition.yaml

@@ -18,14 +18,14 @@ dimensions:
     values:
       - 3.1.3
       - 4.0.0
-      - 4.0.1
       - 4.2.0
+      - 4.2.1
       # Alternatively, if you want to use a custom image, append a comma and the full image name to the product version
       # as in the example below.
       # - 4.2.0,oci.stackable.tech/sdp/hive:4.2.0-stackable0.0.0-dev
   - name: hive-latest
     values:
-      - 4.2.0
+      - 4.2.1
       # Alternatively, if you want to use a custom image, append a comma and the full image name to the product version
       # as in the example below.
       # - 4.2.0,oci.stackable.tech/sdp/hive:4.2.0-stackable0.0.0-dev
@@ -34,7 +34,7 @@ dimensions:
       - 3.1.3
   - name: hive-new
     values:
-      - 4.2.0
+      - 4.2.1
   - name: hdfs-latest
     values:
       - 3.5.0