Skip to content

fix(cll): restore column attribution for sqlglot 30.22+, lift cap - #1577

Merged
gcko merged 1 commit into
mainfrom
feature/drc-4901-adapt-column-level-lineage-to-sqlglot-3022-and-lift-the
Oct 21, 2026
Merged

gcko merged 1 commit into
mainfrom
feature/drc-4901-adapt-column-level-lineage-to-sqlglot-3022-and-lift-the

Conversation

@gcko

@gcko gcko commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

PR checklist

  • Ensure you have added or ran the appropriate tests for your PR.
  • DCO signed

What type of PR is this?

fix

What this PR does / why we need it:

sqlglot 30.22.0 removed infer_schema from the optimizer (tobymao/sqlglot#8552). Before that, when no schema was given, qualify attributed an unqualified column to the one source in the scope with no known columns. Now it raises OptimizeError: Column 'x' could not be resolved. PR #1573 capped sqlglot<30.22 as a stopgap.

This PR adapts cll() and lifts the cap:

  • recce/util/cll.py: new _qualify() helper. A non-empty schema goes through qualify unchanged. With an empty schema it qualifies with validate_qualify_columns=False, re-applies the removed attribution (_attribute_to_schemaless_source), then calls validate_qualify_columns. This is exact parity with sqlglot <30.22, which also inferred only when the schema was empty.
  • Resolver moved from qualify_columns to its own module in late 2025, so the import falls back for the >=27.29.0 floor.
  • _CACHE_SCHEMA_VERSION 3 → 4, so unknown results cached by users on 30.22–30.23 are dropped. Cache eviction is access-based, so hot entries would otherwise stay stale.
  • pyproject.toml: sqlglot>=27.29.0,<31.0; uv.lock resolves 30.24.0.

Which issue(s) this PR fixes:

Closes DRC-4901

Special notes for your reviewer:

  • Where this showed up in production: through DbtAdapter, cll() only gets an empty schema for a model that is in catalog.json but has no resolvable dbt parents, for example one that hard-codes from raw_events. The new adapter test test_model_reading_hardcoded_relation covers this; it reports unknown without the fix and renamed with it. Without a catalog, get_cll returns before calling cll(), so that path is unaffected.
  • Not in this PR: when a parent is missing from the catalog, the adapter writes schema[parent] = {}, which raises SchemaError on every sqlglot version. That's a separate, older gap, tracked in DRC-4918.
  • Why not a schema stub: sqlglot rejects {} (SchemaError), and {"*": ...} still fails to resolve.
  • Version matrix: test_cll.py, test_dbt_cll.py and test_breaking.py pass on sqlglot 27.29.0, 30.21.0 and 30.24.0. The full suite on 30.24.0 gave 2040 passed, 5 skipped.
  • recce/util/change_classifier.py also calls qualify, but only with a non-empty schema, so it never relied on the removed inference. Its tests pass unchanged on 30.24.
  • The two guard tests (..._ambiguous_without_schema, ..._with_partial_schema) pass on main too. They pin parity so the attribution can't spread to ambiguous or partial-schema cases.

Does this PR introduce a user-facing change?:

Column-level lineage works again with sqlglot 30.22 and later. Models that select from hard-coded relations (no dbt parents) trace their columns again instead of showing them as unknown.

🤖 Generated with Claude Code

sqlglot 30.22.0 removed `infer_schema` from the optimizer
(tobymao/sqlglot#8552). With an empty schema, `qualify` no longer
attributes an unqualified column to the scope's only source without
known columns, so `cll()` raised "Column 'x' could not be resolved".
In recce that hit models that are in the catalog but have no
resolvable dbt parents (hard-coded relations), whose columns fell
back to `unknown`.

When the schema is empty, `cll()` now qualifies with validation off,
re-applies the removed attribution, then validates. A non-empty schema
goes through `qualify` unchanged, matching sqlglot <30.22 exactly.

Also lift the `sqlglot<30.22` cap to `<31.0` (lock: 30.24.0) and bump
the CLL cache version so results cached under 30.22-30.23 are dropped.

Closes DRC-4901

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jared Scott <jared.scott@datarecce.io>
@gcko gcko self-assigned this Oct 21, 2026
@gcko
gcko requested review from iamcxa and kentwelcome October 21, 2026 02:24

@kentwelcome kentwelcome left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@codecov

codecov Bot commented Oct 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.29630% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
recce/util/cll.py 93.54% 2 Missing ⚠️
Files with missing lines Coverage Δ
tests/adapter/dbt_adapter/test_dbt_cll.py 99.36% <100.00%> (+<0.01%) ⬆️
tests/util/test_cll.py 100.00% <100.00%> (ø)
recce/util/cll.py 94.79% <93.54%> (+0.38%) ⬆️

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@gcko
gcko merged commit fc20136 into main Oct 21, 2026
24 checks passed
@gcko
gcko deleted the feature/drc-4901-adapt-column-level-lineage-to-sqlglot-3022-and-lift-the branch October 21, 2026 02:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants