[alxmrs]

Stacked on #287. Merge that first; this PR's base is prepared-queries.

Why

DataFusion inlines a CTE or a view at every place a query reads it. A model whose layers each read the layer before more than once grows with every layer. The one-layer GPT in the ML demos plans its loss as 9,882 operators, which are repeats of only 87 distinct subplans. One 233-operator subtree appears 40 times, and its joins output 9.1 million rows per call. DataFusion's CommonSubexprEliminate merges repeated expressions within one node, but never repeated subplans.

How

• src/share.rs (sqlparser 0.62, with its visitor feature): share(query, views, prefix).
  • It hoists the query's CTEs, and the views it reads, into numbered definitions.
  • It counts how often each definition is read, counting only reads from definitions that run.
  • It returns a step for each definition read at least twice, then the query itself, which reads those steps' tables. Every other definition is inlined, as a CTE, into the one place it's read. Each definition then runs exactly once per call.
  • It returns None, and the query is planned as written, for WITH RECURSIVE, a MATERIALIZED CTE, a subquery with CTEs of its own, a view with a column list, more than one statement, or SQL it can't parse.
• Context.prepare (xarray_sql/sql.py):
  • It reads the views of the default catalog and schema from information_schema.views.
  • It plans each step once, and stores the step's rows in a table under a unique _xql_<uuid>_ name.
  • Each call runs the steps in order and writes their rows into those tables in place. When a step's rows can't be written in place, because the row count changes, a column has nulls or a column isn't numeric, it registers the table again and plans the later steps again.

Performance

The GPT demo's training step, on an Intel i7-8700, DataFusion 55.2.0, best of 3 rounds:

	Total per step	Loss
Today's demo loop (ctx.sql)	3,169 ms	2,695 ms
prepare (#287)	2,855 ms	2,538 ms
prepare with shared CTEs	318 ms	16 ms

Most of what's left is ddx's gradient step (xqlsystems/ddx#213).

Tests

tests/test_prepare.py checks that each call returns the rows that planning the query again returns, for:

• a CTE read twice
• nested sharing
• a row count that changes between calls
• nulls and strings
• column aliases
• a view read twice
• a CTE named like a view, which hides the view
• a recursive CTE

Writing stale rows, or skipping the fallback when a write fails, makes the matching tests fail.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NyN7idACeDMh94qrnFMu6h

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: eab8c94e-dd5a-4409-96b9-67f92669229a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}