Read a Databricks catalog, and pin catalog rows as a shared fixture - #278
Read a Databricks catalog, and pin catalog rows as a shared fixture#278jat255 wants to merge 2 commits into
Conversation
jat255
added
needs-manual-review
3babd70 to
dfed967
Compare
|
Preview deployed to Connect ( Deployed from commit 7aa91a8. |
|
Preview deployed to Connect ( Deployed from commit dfed967. |
|
Preview deployed to Connect ( Deployed from commit 7aa91a8. |
dfed967 to
865cb4e
Compare
|
Manual review completed 2026-09-05 23:35 MDT by @jat255. Findings: 1 MEDIUM (untested |
jat255
added
r
Fourth of the catalog import stack (kata xp65). Ports the catalog-import half of catalog-databricks.R. Unity Catalog is read through system.information_schema, which is a plain query rather than a SHOW, and the information schema itself is excluded so a listing describes the data rather than the catalog. The legacy hive_metastore is not in there at all and needs SHOW TABLES, which is why the listing forks on the catalog name and why selecting hive_metastore requires a schema: there is nothing to enumerate a whole legacy catalog from. DESCRIBE TABLE does not report nullability, so it is read separately from system.information_schema.columns and joined by column name. hive_metastore has no information schema to ask, so its columns keep an unknown nullability rather than a guessed one. DESCRIBE TABLE also appends partition and detail sections after a row whose name starts with `#`; reading past that turns section headings into columns. Metric views are out of scope with the rest of the semantic-model work (kata gcgj). They report as views here, and telling them apart needs the ODBC object type R queries, which only matters once metric views are supported. The comment in the reader says so.
Which rows are relations, what kind each is, which comments count as prose, and where a DESCRIBE reply stops being columns are all user-observable, so they belong in tests/shared/ rather than being asserted in each language. catalog-rows.json carries the rows Snowflake and Databricks actually return and what both implementations must make of them; R's hand-written row assertions are replaced by runners over it. Running the queries stays each language's own business. Only their replies are a contract. One R change beyond the runners: the Snowflake column shaping was inline in snowflake_describe_relation(), which also runs the query, so the contract was not reachable without a connection. It is now snowflake_describe_rows(), called by the same function. Behaviour is unchanged. Databricks already had databricks_columns_from_describe(). Booleans travel as strings, because a bare JSON true reads back as a logical in R and an absent one as NULL, which would make the two runners disagree about a fixture neither had changed. Both runners were confirmed to bite by changing one expected kind and watching each suite fail. Found by roborev job 335.
865cb4e to
7aa91a8
Compare
1 participant
Fourth PR of the catalog import stack (kata
xp65). Stacked on #277. Ports the catalog-import half ofcatalog-databricks.R, and moves both readers' row interpretation into a shared fixture.Metric views are out of scope with the rest of the semantic-model work (kata
gcgj).Row interpretation is now a shared fixture
Which rows are relations, what kind each is, which comments count as prose, and where a
DESCRIBEreply stops being columns are all user-observable, so they belong intests/shared/rather than being asserted in each language.catalog-rows.jsoncarries the rows both warehouses actually return and what both implementations must make of them. Running the queries stays each language's own business; only their replies are a contract.Booleans travel as strings, because a bare JSON
truereads back as a logical in R and an absent one asNULL, which would let the two runners disagree about a fixture neither had changed.R changes
@simonpcouch
R/has no new behavior, but does have one refactor. Snowflake's column shaping was inline insnowflake_describe_relation(), which also runs the query, so the contract it implements could not be reached or tested without a live connection. It is nowsnowflake_describe_rows(), called by that same function with the same rows and returning the same data frame. Databricks already haddatabricks_columns_from_describe()split out, so it needed nothing and this makes Snowflake's implementation more consistent.In the test suite, two hand-written tests are replaced by fixture runners with their expectations now pinned in
catalog-rows.json. The Python suite asserts the same cases.Why it is safe:
test-catalog-snowflake.Randtest-catalog-databricks.Rboth pass.Verification
677 Python tests, ruff and pyrefly clean. The reader was confirmed to bite by removing the
#metadata cutoff and by removing thehive_metastorefork.