Import a warehouse catalog when a data source is built - #282
Draft
jat255 wants to merge 1 commit into
Draft
Conversation
jat255
added
needs-manual-review
A Snowflake or Databricks engine now resolves its selection against the warehouse rather than leaving the readers unused: the session identity is read first, explicitly named relations are access-checked before anything is described, the listing is folded into the authored dictionary, and the session is read again at the end. A selection defaults to the namespace the connection already points at, and exclude drops relations from it. The merge re-keys the dictionary to the warehouse's labels, so a table's definitions are now looked up under the name the author gave it rather than the label it ended up with. Definitions over a column the warehouse spells differently are refused for now, because the compiler cannot yet bind an authored name to the discovered one.
jat255
force-pushed
the
jat255/xp65-source-wiring
branch
from
September 5, 2026 04:58
63dfbf7 to
01c9608
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sixth of the catalog import stack (kata xp65). Stacked on #281. Python only.
Everything the stack has landed so far is a library nothing calls. This wires it in: a Snowflake or Databricks engine now imports its catalog when the source is built. The session identity is read first, since every access answer that follows was decided for it; relations named in
tablesare access-checked before anything is described, so a name the caller got wrong fails at construction rather than mid-conversation; the listing is folded into the authored dictionary; and the session is read again at the end, because a role that moved during discovery invalidates what was just learned.A selection defaults to the namespace the connection already points at. A namespace has to be named with a
Selector, becauseANALYTICS.PUBLICon its own does not say whetherPUBLICis a schema or a table; a plain string is read as a relation, the way it is everywhere else.excludeis accepted only for a warehouse, since nothing else has a listing to drop from.The merge re-keys the dictionary to the warehouse's labels, so a table's definitions are now looked up under the name the author gave it. Definitions over a column the warehouse spells differently are refused for now: the compiler cannot yet bind an authored name to the discovered one, and lowering them as written would emit SQL against columns that do not exist. The next PR in the stack replaces that refusal with real binding.
dictionaryis taken byfrom_engine()andfrom_board()because the merge needs it during construction, but definitions are lowered once, indata_source(). Both docstrings say so.Verification
725 Python tests, ruff and pyrefly clean. A fake backend stands in for the network; the fixture-pinned row interpretation and the real dispatch from
data_source()throughfrom_engine()are both exercised. The definitions lookup was confirmed to bite by reverting it and watching the end-to-end test fail.