From 490bf6e95bd897defdc813c769be837613688e8e Mon Sep 17 00:00:00 2001 From: Shani Solomon <84128749+shanisolomon@users.noreply.github.com> Date: Fri, 30 Oct 2026 18:14:38 -0400 Subject: [PATCH] Document smart hot cache for query acceleration (#7608) * Document smart hot cache for query acceleration Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify system-managed smart hot value Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify smart hot cache behavior guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify smart hot profile use cases Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refine smart hot cache configuration guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- data-explorer/kusto-tocs/management/toc.yml | 3 + ...merge-query-acceleration-policy-command.md | 13 +++- ...alter-query-acceleration-policy-command.md | 18 ++++- .../management/query-acceleration-policy.md | 4 +- .../show-query-acceleration-policy-command.md | 4 +- .../kusto/management/smart-hot-cache.md | 71 +++++++++++++++++++ data-explorer/kusto/management/toc.yml | 3 + 7 files changed, 108 insertions(+), 8 deletions(-) create mode 100644 data-explorer/kusto/management/smart-hot-cache.md diff --git a/data-explorer/kusto-tocs/management/toc.yml b/data-explorer/kusto-tocs/management/toc.yml index d2454f3883..b83b6b7a94 100644 --- a/data-explorer/kusto-tocs/management/toc.yml +++ b/data-explorer/kusto-tocs/management/toc.yml @@ -608,6 +608,9 @@ items: - name: Query acceleration policy href: /kusto/management/query-acceleration-policy?view=azure-data-explorer&preserve-view=true displayName: external table, delta table + - name: Smart hot cache + href: /kusto/management/smart-hot-cache?view=azure-data-explorer&preserve-view=true + displayName: smart acceleration, adaptive hot cache, external table, delta table - name: Update policy over external delta tables href: /kusto/management/update-policy?view=azure-data-explorer&preserve-view=true#update-policy-over-external-delta-tables-preview displayName: external table, delta table diff --git a/data-explorer/kusto/management/alter-merge-query-acceleration-policy-command.md b/data-explorer/kusto/management/alter-merge-query-acceleration-policy-command.md index c870056d5a..bdf4967cfb 100644 --- a/data-explorer/kusto/management/alter-merge-query-acceleration-policy-command.md +++ b/data-explorer/kusto/management/alter-merge-query-acceleration-policy-command.md @@ -3,7 +3,7 @@ title: .alter-merge query acceleration policy Command description: Learn how to use the .alter-merge query acceleration policy command to accelerate queries over external delta tables. ms.reviewer: sharmaanshul ms.topic: reference -ms.date: 09/16/2025 +ms.date: 09/22/2026 --- # `.alter-merge query acceleration policy` command @@ -38,10 +38,13 @@ You must have at least [Database Admin](../access-control/role-based-access-cont | Property | Type | Required | Description | | --------------------- | ---------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | IsEnabled | `Boolean` | | Indicates whether the policy is enabled. This property is required if no query acceleration policy is defined on the external table. | -| Hot | `Timespan` | | The hot period defined in the query acceleration policy. Minimum value = 1 d. This property is required if no query acceleration policy is defined on the external table. | +| Hot | `Timespan` | | The rolling hot period for a static policy. Minimum value = 1 day. If no policy exists, specify `Hot` or both `MinHot` and `MaxHot`. Don't specify this property when configuring smart hot cache because the system manages its value. | | HotWindows | `DateTime` | | One or more optional time windows. Delta data files created within these time windows are accelerated. | | MaxAge | `Timespan` | | The external table returns accelerated data if the last index refresh time is greater than @now - MaxAge. Otherwise, external table operates in nonaccelerated mode. Default is 5 minutes. Minimum is 1 minute. | | HotDateTimeColumn | `String` | | Name of a datetime column used to determine hot-cache eligibility. A data file will be cached if its min/max stats for this column are within the configured Hot period and/or HotWindows. If not specified, eligibility is determined using the file modificationTime. Eligibility is computed using [delta log](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#add-file-and-remove-file) metadata only - setting this property does not add query-time overhead| +| MinHot | `Timespan` | | The minimum rolling hot period for [smart hot cache](smart-hot-cache.md). Must be specified together with `MaxHot`. Minimum value = 1 day. | +| MaxHot | `Timespan` | | The maximum rolling hot period for [smart hot cache](smart-hot-cache.md). Must be specified together with `MinHot`. | +| Profile | `String` | | The smart hot cache optimization profile. Supported values are `Performance` (default) and `Cost`. | ### Example @@ -57,6 +60,12 @@ In case the external table doesn't have query acceleration policy defined: { "IsEnabled": true, "Hot": "1.00:00:00" } ``` +To enable smart hot cache: + +```json +{ "IsEnabled": true, "MinHot": "1.00:00:00", "MaxHot": "30.00:00:00", "Profile": "Performance" } +``` + ## Returns diff --git a/data-explorer/kusto/management/alter-query-acceleration-policy-command.md b/data-explorer/kusto/management/alter-query-acceleration-policy-command.md index 638d100e06..aab49974ae 100644 --- a/data-explorer/kusto/management/alter-query-acceleration-policy-command.md +++ b/data-explorer/kusto/management/alter-query-acceleration-policy-command.md @@ -3,7 +3,7 @@ title: .alter query acceleration policy command description: Learn how to use the .alter query acceleration policy command to accelerate queries over external delta tables. ms.reviewer: sharmaanshul ms.topic: reference -ms.date: 06/29/2026 +ms.date: 09/22/2026 --- # `.alter query acceleration policy` command @@ -39,10 +39,13 @@ You must have at least [Database Admin](../access-control/role-based-access-cont | Property | Type | Required | Description | | ------------------- | ---------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | IsEnabled | `Boolean` | :heavy_check_mark: | Indicates whether the policy is enabled. | -| Hot | `Timespan` | :heavy_check_mark: | The hot period defined in the query acceleration policy. Minimum value = 1 d. | +| Hot | `Timespan` | | The rolling hot period for a static policy. Minimum value = 1 day. Required unless both `MinHot` and `MaxHot` are specified. Don't specify this property when configuring smart hot cache because the system manages its value. | | HotWindows | `DateTime` | | One or more optional time windows. Delta data files created within these time windows are accelerated. | | MaxAge | `Timespan` | | The external table returns accelerated data if the last index refresh time is greater than @now - MaxAge. Otherwise, the external table operates in non-accelerated mode. Default is 5 minutes. Minimum is 1 minute. This property can also be overridden at query time by using the [`external_table()` function's](../query/external-table-function.md) `MaxAgeOverride` parameter. | | HotDateTimeColumn | `String` | | The Name of a datetime column used to determine hot-cache eligibility. A data file will be cached if its min/max stats for this column are within the configured Hot period and/or HotWindows. If not specified, eligibility is determined using the file modificationTime. Eligibility is computed using [delta log](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#add-file-and-remove-file) metadata only - setting this property does not add query-time overhead | +| MinHot | `Timespan` | | The minimum rolling hot period for [smart hot cache](smart-hot-cache.md). Must be specified together with `MaxHot`. Minimum value = 1 day. | +| MaxHot | `Timespan` | | The maximum rolling hot period for [smart hot cache](smart-hot-cache.md). Must be specified together with `MinHot`. | +| Profile | `String` | | The smart hot cache optimization profile. Supported values are `Performance` (default) and `Cost`. | ::: moniker-end @@ -51,11 +54,14 @@ You must have at least [Database Admin](../access-control/role-based-access-cont | Property | Type | Required | Description | | ------------------- | ---------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | IsEnabled | `Boolean` | :heavy_check_mark: | Indicates whether the policy is enabled. | -| Hot | `Timespan` | :heavy_check_mark: | The hot period defined in the query acceleration policy. Minimum value = 1 d. | +| Hot | `Timespan` | | The rolling hot period for a static policy. Minimum value = 1 day. Required unless both `MinHot` and `MaxHot` are specified. Don't specify this property when configuring smart hot cache because the system manages its value. | | HotWindows | `DateTime` | | One or more optional time windows. Delta data files created within these time windows are accelerated. | | MaxAge | `Timespan` | | The external table returns accelerated data if the last index refresh time is greater than @now - MaxAge. Otherwise, the external table operates in non-accelerated mode. Default is 5 minutes. Minimum is 1 minute. This property can also be overridden at query time by using the [`external_table()` function's](../query/external-table-function.md) `MaxAgeOverride` parameter. | | ManagedIdentity | `string` | | Optional managed identity for which the query acceleration background operations are executed. This identity must have relevant delta table permissions and must be enabled for AutomatedFlows in the cluster / database managed identity policy. For more information, see [Managed identities overview](/azure/data-explorer/managed-identities-overview). | | HotDateTimeColumn | `String` | | Optional. The name of a datetime column in the Delta table whose values will be used to determine hot-cache eligibility. When set, data files whose rows have values within the configured Hot period (and/or HotWindows) are selected for caching. | +| MinHot | `Timespan` | | The minimum rolling hot period for [smart hot cache](smart-hot-cache.md). Must be specified together with `MaxHot`. Minimum value = 1 day. | +| MaxHot | `Timespan` | | The maximum rolling hot period for [smart hot cache](smart-hot-cache.md). Must be specified together with `MinHot`. | +| Profile | `String` | | The smart hot cache optimization profile. Supported values are `Performance` (default) and `Cost`. | ::: moniker-end @@ -69,6 +75,12 @@ You must have at least [Database Admin](../access-control/role-based-access-cont { "IsEnabled": true, "Hot": "1.00:00:00" } ``` +To enable smart hot cache: + +```json +{ "IsEnabled": true, "MinHot": "1.00:00:00", "MaxHot": "30.00:00:00", "Profile": "Performance" } +``` + ## Returns The command returns a table with one record that includes the modified policy object. diff --git a/data-explorer/kusto/management/query-acceleration-policy.md b/data-explorer/kusto/management/query-acceleration-policy.md index 2799bfb47e..79fb6448c2 100644 --- a/data-explorer/kusto/management/query-acceleration-policy.md +++ b/data-explorer/kusto/management/query-acceleration-policy.md @@ -3,7 +3,7 @@ title: Query acceleration policy description: Learn how to use the query acceleration policy to accelerate queries over external delta tables. ms.reviewer: sharmaanshul ms.topic: reference -ms.date: 06/29/2026 +ms.date: 09/22/2026 --- # Query acceleration policy @@ -21,6 +21,8 @@ Query acceleration is supported in Eventhouse over OneLake, Azure Data Lake Stor To enable query acceleration in the Fabric UI, see [Query acceleration over OneLake shortcuts](https://go.microsoft.com/fwlink/?linkid=2296674). ::: moniker-end +To automatically align the accelerated data period with observed query patterns, configure [smart hot cache](smart-hot-cache.md). + > [!IMPORTANT] > To troubleshoot problems with query acceleration, see [Troubleshoot query acceleration over external delta tables](query-acceleration-troubleshooting-guide.md). diff --git a/data-explorer/kusto/management/show-query-acceleration-policy-command.md b/data-explorer/kusto/management/show-query-acceleration-policy-command.md index 5dc3e3f0ae..286739c698 100644 --- a/data-explorer/kusto/management/show-query-acceleration-policy-command.md +++ b/data-explorer/kusto/management/show-query-acceleration-policy-command.md @@ -3,7 +3,7 @@ title: .show query acceleration policy command description: Learn how to use the .show query acceleration policy command to accelerate queries over external delta tables. ms.reviewer: sharmaanshul ms.topic: reference -ms.date: 09/16/2025 +ms.date: 09/22/2026 --- # `.show query acceleration policy` command @@ -50,7 +50,7 @@ The command returns a table with a record per external table with the following | PolicyName | EntityName | Policy | ChildEntities | EntityType | | ----------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | ------------- | -| QueryAccelerationPolicy | [MyDatabase].[MyExternalTable] | `{
"IsEnabled": true,
"Hot": "1.00:00:00",
"HotWindows": [{"MinValue":"2025-07-06 07:53:55","MaxValue":"2025-07-07 07:53:55"}], "MaxAge": "5m", "HotDateTimeColumn":"dt1"}` | | ExternalTable | +| QueryAccelerationPolicy | [MyDatabase].[MyExternalTable] | `{
"IsEnabled": true,
"Hot": "7.00:00:00",
"MinHot": "1.00:00:00",
"MaxHot": "30.00:00:00",
"Profile": "Performance",
"MaxAge": "5m",
"HotDateTimeColumn": "dt1"
}` | | ExternalTable | ## Related content diff --git a/data-explorer/kusto/management/smart-hot-cache.md b/data-explorer/kusto/management/smart-hot-cache.md new file mode 100644 index 0000000000..d40c66c731 --- /dev/null +++ b/data-explorer/kusto/management/smart-hot-cache.md @@ -0,0 +1,71 @@ +--- +title: Smart hot cache for query acceleration +description: Learn how smart hot cache automatically adjusts the query acceleration hot period for external delta tables. +ms.reviewer: sharmaanshul +ms.topic: concept-article +ms.date: 09/22/2026 +--- + +# Smart hot cache for query acceleration + +> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] + +The [query acceleration policy](query-acceleration-policy.md) uses the `Hot` property to determine the rolling period of external delta table data to accelerate. With a static policy, you set `Hot` manually and update it as query patterns change. + +Smart hot cache automatically adjusts `Hot` based on observed query patterns. You define the allowed range with `MinHot` and `MaxHot`, and the system periodically selects a value within that range. + +> [!NOTE] +> Smart hot cache adjusts only the rolling `Hot` period. It doesn't create or modify `HotWindows`. + +## Optimization profiles + +Use the `Profile` property to control what happens when no query signal is available. + +| Profile | No-signal behavior | Recommended use case | +|---|---|---| +| `Performance` | Keeps the current `Hot` value. This profile is the default. | You want to optimize for query performance. | +| `Cost` | Sets `Hot` to `MinHot`, reducing caching to the configured minimum when there are no queries. | You want to optimize cost by minimizing the hot-cache footprint during periods without query activity. | + +When query signal is available, both profiles use the calculated P95 lookback constrained by `MinHot` and `MaxHot`. + +## How smart hot cache works + +Smart hot cache observes query patterns and adjusts `Hot` once a day based on the 95th percentile (P95) query lookback over the previous seven days. The adjusted value is constrained to the range from `MinHot` through `MaxHot`. + +If you configure `HotDateTimeColumn`, smart hot cache uses query patterns for that column. Otherwise, it uses the first datetime column in the external table schema that has observed query activity. + +## Configure smart hot cache + +Set `MinHot` and `MaxHot` to different values to enable smart hot cache. Both properties are required. + +The following example configures a range of one through 30 days and uses the default `Performance` profile: + +```kusto +.alter external table MyExternalTable policy query_acceleration '{"IsEnabled": true, "MinHot": "1.00:00:00", "MaxHot": "30.00:00:00", "Profile": "Performance"}' +``` + +When you configure `MinHot` and `MaxHot`, don't specify `Hot`. The system initializes `Hot` to `MinHot` and then manages its value within the configured range. + +## Configuration rules + +* Set `MinHot` and `MaxHot` together. +* `MinHot` must be less than or equal to `MaxHot`. +* `MinHot` must be at least one day. +* Don't specify `Hot` when configuring smart hot cache. The system manages this value. +* Enable smart hot cache only when the policy is enabled and `MinHot` and `MaxHot` have different values. +* If `MinHot` and `MaxHot` are equal, the policy uses a static `Hot` period. + +## Return to a static hot period + +Set `MinHot` and `MaxHot` to the same value to use a static hot period. When both values are equal, smart hot cache doesn't adjust the period. + +```kusto +.alter external table MyExternalTable policy query_acceleration '{"IsEnabled": true, "MinHot": "7.00:00:00", "MaxHot": "7.00:00:00"}' +``` + +## Related content + +* [Query acceleration policy](query-acceleration-policy.md) +* [`.alter query acceleration policy` command](alter-query-acceleration-policy-command.md) +* [`.alter-merge query acceleration policy` command](alter-merge-query-acceleration-policy-command.md) +* [`.show query acceleration policy` command](show-query-acceleration-policy-command.md) diff --git a/data-explorer/kusto/management/toc.yml b/data-explorer/kusto/management/toc.yml index e9d947ec36..33b4bb91e8 100644 --- a/data-explorer/kusto/management/toc.yml +++ b/data-explorer/kusto/management/toc.yml @@ -607,6 +607,9 @@ items: - name: Query acceleration policy href: query-acceleration-policy.md displayName: external table, delta table + - name: Smart hot cache + href: smart-hot-cache.md + displayName: smart acceleration, adaptive hot cache, external table, delta table - name: Update policy over external delta tables href: update-policy.md#update-policy-over-external-delta-tables-preview displayName: external table, delta table