From 7cd29b90b0ea22da1a18bf844283d4575e11a0f9 Mon Sep 17 00:00:00 2001 From: Thomas Langton Date: Thu, 10 Sep 2026 18:04:06 +0100 Subject: [PATCH 1/4] perf: prune partitions in current_tests_run_results_query MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `current_tests_run_results_query` filters both of its large CTEs with `edr_datediff(edr_cast_as_timestamp(col), now, 'day') < days_back`. On BigQuery a function applied to the column defeats partition pruning, so both scan all history however small `days_back` is. Adds a dispatched `days_back_filter` macro: the default implementation renders exactly the predicate above, so nothing changes off BigQuery, while `bigquery__` compares the column directly and adds a bound on the partition column. Measured with a 7-day lookback, selecting a single column, on tables as they already are — no repartitioning needed to get this: elementary_test_results (12M rows, 13GB) 1,368 MB -> 13.68 MB dbt_run_results (17M rows, 41GB) 1,627 MB -> 63.56 MB For reference, an unfiltered scan of elementary_test_results reads 1,279 MB: the current predicate reads *more* than no filter at all, since it also pays to read the column it filters on. The bound on `created_at` looks redundant — it is implied by the first predicate — but BigQuery only prunes on the raw partition column, and `execute_completed_at` is stored as a string, so a cast is unavoidable there and no predicate on it can prune. dbt_run_results is not even partitioned on the warehouse measured above; the 26x comes from block-level pruning on `created_at` alone, and partitioning it would improve that further. The bound is also safe: `created_at` is written at or after both columns, so any row satisfying the first predicate satisfies the bound. Verified over 17.2M rows — `created_at - generated_at` >= 0s, `created_at - execute_completed_at` >= 5s, and zero rows violating either. The guard is given a day of slack rather than the same bound: `created_at` is stamped by the warehouse at insert while the columns it guards come from the dbt client, so the two are not the same clock. Pruning is at day granularity, so the slack costs at most one extra partition — measured as no difference at all on the figures above, since the real predicate is the binding constraint. --- .../current_tests_run_results_query.sql | 4 +- .../macros/utils/days_back_filter.sql | 67 +++++++++++++++++++ 2 files changed, 69 insertions(+), 2 deletions(-) create mode 100644 elementary/monitor/dbt_project/macros/utils/days_back_filter.sql diff --git a/elementary/monitor/dbt_project/macros/base_queries/current_tests_run_results_query.sql b/elementary/monitor/dbt_project/macros/base_queries/current_tests_run_results_query.sql index 3e18222bf..8c99cd8c1 100644 --- a/elementary/monitor/dbt_project/macros/base_queries/current_tests_run_results_query.sql +++ b/elementary/monitor/dbt_project/macros/base_queries/current_tests_run_results_query.sql @@ -2,14 +2,14 @@ with elementary_test_results as ( select * from {{ ref('elementary', 'elementary_test_results') }} {% if days_back %} - where {{ elementary.edr_datediff(elementary.edr_cast_as_timestamp('detected_at'), elementary.edr_current_timestamp(), 'day') }} < {{ days_back }} + where {{ elementary_cli.days_back_filter('detected_at', days_back, partition_column='created_at') }} {% endif %} ), dbt_run_results as ( select * from {{ ref('elementary', 'dbt_run_results') }} {% if days_back %} - where {{ elementary.edr_datediff(elementary.edr_cast_as_timestamp('execute_completed_at'), elementary.edr_current_timestamp(), 'day') }} < {{ days_back }} + where {{ elementary_cli.days_back_filter('execute_completed_at', days_back, partition_column='created_at', column_is_string=true) }} {% endif %} ), diff --git a/elementary/monitor/dbt_project/macros/utils/days_back_filter.sql b/elementary/monitor/dbt_project/macros/utils/days_back_filter.sql new file mode 100644 index 000000000..3292c3a6d --- /dev/null +++ b/elementary/monitor/dbt_project/macros/utils/days_back_filter.sql @@ -0,0 +1,67 @@ +{# + Renders the `where` predicate for a `days_back` lookback. + + The default implementation is the shape these queries have always used: wrap the column in a + timestamp cast and compare a `datediff` against `days_back`. + + On BigQuery that shape cannot prune partitions — a function applied to the partition column + forces a full scan — so `bigquery__` compares the column directly instead, and optionally adds + a bound on `partition_column`. + + That second bound looks redundant, and logically it is: it is implied by the first. It exists + because BigQuery only prunes on the raw partition column. It is needed whenever the column + being filtered is not the partition column, and whenever it has to be cast (dbt_run_results + stores `generated_at` and `execute_completed_at` as strings, so a cast is unavoidable there + and no predicate on them can ever prune). + + The bound is deliberately given a day of slack. `created_at` is stamped by the warehouse at + insert time while the columns it guards come from the dbt client, so the two are not read from + the same clock. Pruning is only ever at day granularity, so the slack costs at most one extra + partition and removes any chance of the guard excluding a row the real predicate wanted. + + Args: + column: the column to filter on. + days_back: size of the lookback window, in days. + partition_column: the table's partition column, when it differs from `column`. Adds the + pruning bound described above. + column_is_string: set when `column` is stored as a string and needs casting before + comparison. +#} + +{% macro days_back_filter(column, days_back, partition_column=none, column_is_string=false) %} + {% do return( + adapter.dispatch("days_back_filter", "elementary_cli")( + column, days_back, partition_column, column_is_string + ) + ) %} +{% endmacro %} + +{% macro default__days_back_filter( + column, days_back, partition_column=none, column_is_string=false +) %} + {% set predicate %} + {{ elementary.edr_datediff( + elementary.edr_cast_as_timestamp(column), elementary.edr_current_timestamp(), 'day' + ) }} < {{ days_back }} + {% endset %} + {% do return(predicate) %} +{% endmacro %} + +{% macro bigquery__days_back_filter( + column, days_back, partition_column=none, column_is_string=false +) %} + {% set window_start = elementary.edr_timeadd( + "day", -1 * (days_back | int), elementary.edr_current_timestamp() + ) %} + {% set filtered = elementary.edr_cast_as_timestamp(column) if column_is_string else column %} + {% set predicate %} + {{ filtered }} > {{ window_start }} + {% if partition_column and partition_column != column %} + {# a day of slack, since this column is stamped from a different clock #} + and {{ partition_column }} > {{ elementary.edr_timeadd( + "day", -1 * (days_back | int + 1), elementary.edr_current_timestamp() + ) }} + {% endif %} + {% endset %} + {% do return(predicate) %} +{% endmacro %} From 73cf8fc0f34e29d8e3ee66504e2e1c848e250e22 Mon Sep 17 00:00:00 2001 From: Thomas Langton Date: Thu, 10 Sep 2026 18:04:06 +0100 Subject: [PATCH 2/4] refactor: fold get_result_rows_agate's adapter split into days_back_filter #1940 gave this macro a `bigquery__` override that duplicated the whole body just to change its `where` clause. Now that `days_back_filter` dispatches on the predicate itself, the two copies collapse back into one and the dispatch disappears. Behaviour is unchanged on every adapter: `default__days_back_filter` renders the datediff form the original had, and `bigquery__days_back_filter` renders the direct comparison the override had. It also gains the `created_at` bound. test_result_rows declares no partitioning today, so the direct comparison on `detected_at` was relying on block-level pruning alone; the bound takes a 7-day lookback from 1,060 MB to 101 MB on a 26M row / 41GB copy of that table, and improves further once it is partitioned. --- .../macros/get_result_rows_agate.sql | 24 +------------------ 1 file changed, 1 insertion(+), 23 deletions(-) diff --git a/elementary/monitor/dbt_project/macros/get_result_rows_agate.sql b/elementary/monitor/dbt_project/macros/get_result_rows_agate.sql index b56b42126..146bc434b 100644 --- a/elementary/monitor/dbt_project/macros/get_result_rows_agate.sql +++ b/elementary/monitor/dbt_project/macros/get_result_rows_agate.sql @@ -1,14 +1,10 @@ {% macro get_result_rows_agate(days_back, valid_ids_query = none) %} - {% do return(adapter.dispatch('get_result_rows_agate', 'elementary_cli')(days_back, valid_ids_query)) %} -{% endmacro %} - -{% macro default__get_result_rows_agate(days_back, valid_ids_query = none) %} {% set query %} select elementary_test_results_id, result_row from {{ ref("test_result_rows", package="elementary") }} - where {{ elementary.edr_datediff(elementary.edr_cast_as_timestamp('detected_at'), elementary.edr_current_timestamp(), 'day') }} < {{ days_back }} + where {{ elementary_cli.days_back_filter('detected_at', days_back, partition_column='created_at') }} {% if valid_ids_query %} and elementary_test_results_id in ({{ valid_ids_query }}) {% endif %} @@ -19,21 +15,3 @@ {% endif %} {% do return(res.group_by("elementary_test_results_id")) %} {% endmacro %} - -{% macro bigquery__get_result_rows_agate(days_back, valid_ids_query = none) %} - {% set query %} - select - elementary_test_results_id, - result_row - from {{ ref("test_result_rows", package="elementary") }} - where detected_at > {{ elementary.edr_timeadd('day', -1 * days_back, elementary.edr_current_timestamp()) }} - {% if valid_ids_query %} - and elementary_test_results_id in ({{ valid_ids_query }}) - {% endif %} - {% endset %} - {% set res = elementary.run_query(query) %} - {% if not res %} - {% do return({}) %} - {% endif %} - {% do return(res.group_by("elementary_test_results_id")) %} -{% endmacro %} \ No newline at end of file From e8a18baa5d864bd7616298906a596fea8e3b97e1 Mon Sep 17 00:00:00 2001 From: Thomas Langton Date: Fri, 11 Sep 2026 13:50:21 +0100 Subject: [PATCH 3/4] style: trim days_back_filter to the repo's commenting idiom MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Only 4 of 48 macros here carry a doc block and none exceeded 12 lines; this had 24. Keeps the two things a reader cannot infer and would otherwise 'tidy' away — that the second bound is what makes BigQuery prune, and that its day of slack is deliberate — and drops the rest, which the PR description already covers. Also builds the predicate by joining a list rather than capturing a {% set %} block, so the compiled SQL is a single line instead of carrying the macro's indentation and blank lines into every query. --- .../macros/utils/days_back_filter.sql | 83 +++++++------------ 1 file changed, 30 insertions(+), 53 deletions(-) diff --git a/elementary/monitor/dbt_project/macros/utils/days_back_filter.sql b/elementary/monitor/dbt_project/macros/utils/days_back_filter.sql index 3292c3a6d..c5e544e27 100644 --- a/elementary/monitor/dbt_project/macros/utils/days_back_filter.sql +++ b/elementary/monitor/dbt_project/macros/utils/days_back_filter.sql @@ -1,67 +1,44 @@ {# - Renders the `where` predicate for a `days_back` lookback. + Filters `column` to the last `days_back` days. - The default implementation is the shape these queries have always used: wrap the column in a - timestamp cast and compare a `datediff` against `days_back`. - - On BigQuery that shape cannot prune partitions — a function applied to the partition column - forces a full scan — so `bigquery__` compares the column directly instead, and optionally adds - a bound on `partition_column`. - - That second bound looks redundant, and logically it is: it is implied by the first. It exists - because BigQuery only prunes on the raw partition column. It is needed whenever the column - being filtered is not the partition column, and whenever it has to be cast (dbt_run_results - stores `generated_at` and `execute_completed_at` as strings, so a cast is unavoidable there - and no predicate on them can ever prune). - - The bound is deliberately given a day of slack. `created_at` is stamped by the warehouse at - insert time while the columns it guards come from the dbt client, so the two are not read from - the same clock. Pruning is only ever at day granularity, so the slack costs at most one extra - partition and removes any chance of the guard excluding a row the real predicate wanted. - - Args: - column: the column to filter on. - days_back: size of the lookback window, in days. - partition_column: the table's partition column, when it differs from `column`. Adds the - pruning bound described above. - column_is_string: set when `column` is stored as a string and needs casting before - comparison. + Pass `partition_column` when `column` is not the table's partition column, or is a string and so + has to be cast: BigQuery prunes only on the raw partition column, and without this the query + scans all history. Its bound carries a day of slack because the two columns are stamped from + different clocks — the guarded ones by the dbt client, `created_at` by the warehouse on insert. #} {% macro days_back_filter(column, days_back, partition_column=none, column_is_string=false) %} - {% do return( - adapter.dispatch("days_back_filter", "elementary_cli")( - column, days_back, partition_column, column_is_string - ) - ) %} + {% do return( + adapter.dispatch("days_back_filter", "elementary_cli")( + column, days_back, partition_column, column_is_string + ) + ) %} {% endmacro %} {% macro default__days_back_filter( - column, days_back, partition_column=none, column_is_string=false + column, days_back, partition_column=none, column_is_string=false ) %} - {% set predicate %} - {{ elementary.edr_datediff( - elementary.edr_cast_as_timestamp(column), elementary.edr_current_timestamp(), 'day' - ) }} < {{ days_back }} - {% endset %} - {% do return(predicate) %} + {%- set days_diff = elementary.edr_datediff( + elementary.edr_cast_as_timestamp(column), elementary.edr_current_timestamp(), "day" + ) -%} + {% do return(days_diff ~ " < " ~ days_back) %} {% endmacro %} {% macro bigquery__days_back_filter( - column, days_back, partition_column=none, column_is_string=false + column, days_back, partition_column=none, column_is_string=false ) %} - {% set window_start = elementary.edr_timeadd( - "day", -1 * (days_back | int), elementary.edr_current_timestamp() - ) %} - {% set filtered = elementary.edr_cast_as_timestamp(column) if column_is_string else column %} - {% set predicate %} - {{ filtered }} > {{ window_start }} - {% if partition_column and partition_column != column %} - {# a day of slack, since this column is stamped from a different clock #} - and {{ partition_column }} > {{ elementary.edr_timeadd( - "day", -1 * (days_back | int + 1), elementary.edr_current_timestamp() - ) }} - {% endif %} - {% endset %} - {% do return(predicate) %} + {%- set filtered = elementary.edr_cast_as_timestamp(column) if column_is_string else column -%} + {%- set conditions = [ + filtered ~ " > " ~ elementary.edr_timeadd( + "day", -1 * (days_back | int), elementary.edr_current_timestamp() + ) + ] -%} + {%- if partition_column and partition_column != column -%} + {%- do conditions.append( + partition_column ~ " > " ~ elementary.edr_timeadd( + "day", -1 * ((days_back | int) + 1), elementary.edr_current_timestamp() + ) + ) -%} + {%- endif -%} + {% do return(conditions | join(" and ")) %} {% endmacro %} From 74d795b78b89f1366126bf9a4cfd5092789eee71 Mon Sep 17 00:00:00 2001 From: Thomas Langton Date: Fri, 11 Sep 2026 10:46:39 +0100 Subject: [PATCH 4/4] perf: prune partitions in get_models_runs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `get_models_runs` computes `row_number()` over the whole of model_run_results and only then filters to `days_back`, so the report reads all history on every run. Moves the filter into the CTE and gives it a `created_at` bound to prune on. The move is safe: for any row inside the window, every row ranked above it for the same unique_id is newer, and therefore also inside the window — so ranks are unchanged for every row the outer query returns, and the `rank = 1 -> compiled_code` selection is preserved exactly. Confirmed against 39,185 rows in a 7-day window: zero rank differences, and compiled_code selected for the same 1,678 rows either way. Requires elementary-data/dbt-data-reliability#1057, which exposes `created_at` on the model_run_results view. Without it this cannot prune, since the view's other timestamps are strings and a cast defeats pruning. --- elementary/monitor/dbt_project/macros/get_models_runs.sql | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/elementary/monitor/dbt_project/macros/get_models_runs.sql b/elementary/monitor/dbt_project/macros/get_models_runs.sql index f17aedd27..a41f4ac7b 100644 --- a/elementary/monitor/dbt_project/macros/get_models_runs.sql +++ b/elementary/monitor/dbt_project/macros/get_models_runs.sql @@ -5,6 +5,7 @@ *, row_number() over (partition by unique_id order by generated_at desc) as invocations_rank_index from {{ ref('elementary', 'model_run_results') }} + where {{ elementary_cli.days_back_filter('generated_at', days_back, partition_column='created_at', column_is_string=true) }} ) select @@ -22,9 +23,8 @@ case when invocations_rank_index = 1 then compiled_code else NULL end as compiled_code, generated_at from model_runs - where {{ elementary.edr_datediff(elementary.edr_cast_as_timestamp('generated_at'), elementary.edr_current_timestamp(), 'day') }} < {{ days_back }} {% if exclude_elementary %} - and unique_id not like 'model.elementary.%' + where unique_id not like 'model.elementary.%' {% endif %} order by generated_at {% endset %}