Skip to content

Lower type-checked definitions to DuckDB SQL - #271

Open
jat255 wants to merge 2 commits into
jat255/m2-f6hz-export-typecheckfrom
jat255/m2-f6hz-duckdb-emitter
Open

Lower type-checked definitions to DuckDB SQL#271
jat255 wants to merge 2 commits into
jat255/m2-f6hz-export-typecheckfrom
jat255/m2-f6hz-duckdb-emitter

Conversation

@jat255

@jat255 jat255 commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Third of four PRs porting the data-dict definition compiler (kata f6hz, stage 1). Stacked on #270.

This is where conformance completes. translation.code is now asserted byte for byte and notes exactly, for all 42 corpus definitions, against the shared fixture and against an installed data-dict binary.

Numbers are formatted by widening the significant digits until the text reads back as the same double, which is the R algorithm rather than repr(). repr() switches to exponent notation and would emit 1e+20 where data-dict emits the digits. Perturbing the formatter to repr() fails only the dedicated boundary test and leaves the 42-definition corpus green, so that test earns its place: the corpus never reaches the case.

LIKE lowers to the narrowest form an authored pattern allows, so a wildcard-free pattern becomes an equality and a single anchored wildcard becomes a prefix or suffix test. A pattern that is itself an expression stays a LIKE, because none of that is knowable at compile time.

The notes are why the emitter is hand-written rather than generated through sqlglot: they are collected as the writer walks the IR, and sqlglot has no way to produce them.

_export.py imports the emitter inside _resolve_one rather than at module scope, because the emitter needs Ir from _export. Worth a look if you would rather the IR moved to its own module to break the cycle properly.

Verification: 545 tests pass, ruff and pyrefly clean.

@jat255
jat255 marked this pull request as draft September 4, 2026 05:49
@jat255 jat255 added this to the py-M2: data layer milestone Sep 4, 2026
@jat255 jat255 added needs-manual-review Agent-created work that needs a human review py Affects the Python implementation labels Sep 4, 2026
@jat255
jat255 force-pushed the jat255/m2-f6hz-duckdb-emitter branch from e04128c to 0f0f7b1 Compare September 5, 2026 04:58
@jat255

jat255 commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Reviewed 2026-09-04 23:05 MDT — manual review complete. Findings: 3 MEDIUM (circular-import workaround avoidable via the codebase's TYPE_CHECKING idiom; stale translations docstring/comment; small-magnitude float notation unpinned by tests), 1 LOW (emitter errors bypass translation["error"]), no blockers. All three MEDIUMs fixed in 9daded3; full suite green (545 passed), roborev clean.

@jat255 jat255 removed the needs-manual-review Agent-created work that needs a human review label Sep 6, 2026
Third of four PRs porting the definition compiler (kata f6hz). Completes
conformance: translation.code is now asserted byte for byte and notes
exactly, for all 42 corpus definitions, against the shared fixture and
against an installed data-dict binary.

Numbers are formatted by widening the significant digits until the text
reads back as the same double, which is R's algorithm rather than repr().
repr() switches to exponent notation, so it emits 1e+20 where data-dict
emits the digits. Perturbing the formatter to repr() fails only the
dedicated boundary test and leaves the 42-definition corpus green, which is
the case the spec predicted: pass every corpus case, break on the next
value.

LIKE lowers to the narrowest form an authored pattern allows, so a
wildcard-free pattern is an equality and a single anchored wildcard is a
prefix or suffix test. A pattern that is itself an expression stays a LIKE.

Notes are collected as the writer walks the IR, which is why the emitter is
hand-written rather than generated: sqlglot could produce SQL but not the
NaN, modulus and integer-overflow notes that go with it.
Ir is only used for annotations in _emit_duckdb, which already has
`from __future__ import annotations`, so a TYPE_CHECKING guard removes
the cycle and the deferred import in _resolve_one, matching the idiom
in _measures.py and _backends.py.

Also refresh the stale docstrings that still described lowering as a
later, source-driven step, and pin fixed-notation formatting for
small-magnitude floats, where repr() would give 1e-06.
@jat255
jat255 force-pushed the jat255/m2-f6hz-duckdb-emitter branch from 9daded3 to 5934c22 Compare September 6, 2026 02:07
@jat255
jat255 marked this pull request as ready for review September 6, 2026 02:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

py Affects the Python implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant