perf: prune partitions in get_models_runs

[tlangton3]

Important

Draft — depends on elementary-data/dbt-data-reliability#1057, which exposes created_at on the model_run_results view. Until that is merged and this project's package dependency points at it, the guard below has no column to prune on and CI here will fail. Opening now so the change is ready; happy to rebase or close and re-open once the ordering suits you.

It also stacks on #2351, which adds the days_back_filter macro this uses. The diff shown will shrink to a single file once that merges.

What

get_models_runs computes row_number() over the entire model_run_results view and only then filters to days_back:

with model_runs as (
    select *, row_number() over (partition by unique_id order by generated_at desc) ...
    from {{ ref('elementary', 'model_run_results') }}     -- unbounded
)
select ...
from model_runs
where {{ edr_datediff(edr_cast_as_timestamp('generated_at'), now, 'day') }} < days_back

So every report run reads all history. On one warehouse this is the single most expensive query the CLI issues: ~12.4 GB per invocation against a 17M row / 41GB table, and ~1,115 GB over a week.

The fix

Moving the filter into the CTE and bounding created_at, which is the column BigQuery can prune on.

Worth being precise that the filter move earns nothing by itself — it is the created_at bound that prunes. I had also replaced select * in the CTE with an explicit column list, and dropped it again after measuring: BigQuery already prunes to the columns a query references, so both shapes read 31,868 MB, identical. It was ten lines and a second place to maintain the column set for no gain.

On the warehouse above, the 8-day window is 0.85% of the table (146,402 of 17,207,248 rows across 779 partition-days), so this should fall to a fraction of a GB per invocation. I cannot post a measured after-figure until #1057 is in, and would rather leave it blank than estimate.

Why the move is safe

The filter can move above the window function without changing any returned row's rank. For any row R inside the window, every row ranked above R for the same unique_id is newer than R, and therefore also inside the window. So no row that contributed to R's rank is removed, and R's rank is identical either way — which also preserves the invocations_rank_index = 1 → compiled_code selection exactly.

Confirmed rather than argued, over a 7-day window on real data:

rows_in_window          39,185
rank_mismatches              0
compiled_code rows (old) 1,678
compiled_code rows (new) 1,678

Non-BigQuery adapters

Unchanged. default__days_back_filter renders the same edr_datediff(edr_cast_as_timestamp(...)) < days_back predicate this macro already used; only its position in the query moves, and that is rank-preserving as above.

exclude_elementary deliberately stays in the outer query. It could move into the CTE too — filtering whole unique_id groups cannot affect ranks within other groups — but it prunes nothing, so I left the change surface as small as possible.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

👋 @tlangton3
Thank you for raising your pull request.
Please make sure to add tests and document all user-facing changes.
You can do this by editing the docs files in this pull request.