diff --git a/documentation/concepts/resource-groups.md b/documentation/concepts/resource-groups.md new file mode 100644 index 000000000..9466ca879 --- /dev/null +++ b/documentation/concepts/resource-groups.md @@ -0,0 +1,239 @@ +--- +title: Resource groups +sidebar_label: Resource groups +description: + Resource groups isolate query workloads inside one QuestDB instance. Learn how + a query is assigned to a group, and what admission, CPU weight, CPU caps and + memory limits actually guarantee. +--- + +import { EnterpriseNote } from "@site/src/components/EnterpriseNote" + + + Resource groups isolate competing query workloads inside a single QuestDB + instance. + + +A resource group is a named policy that limits what a set of principals may +consume while their queries run. One instance typically serves several workloads +at once: dashboards that must answer in milliseconds, an ad-hoc analyst, and a +nightly report that scans a year of data. When these workloads compete without +resource controls, the report can increase dashboard latency. + +Resource groups control four things at the query execution boundary: + +- **Admission** — how many queries a group may run at once, how many may wait, + and how long they may wait. +- **Weighted CPU** — the share of query CPU a group receives while groups + compete. +- **A CPU rate limit** — an absolute ceiling expressed as a percentage of + instance capacity. +- **Memory** — process and group budgets for tracked native query memory. + +The design is cooperative. QuestDB executes query work on shared worker pools, +and resource groups do not create one operating-system thread pool per group. A +query and all of its parallel tasks use the same resource group, while every +worker stays available to every group. + +## How a query is assigned to a group + +Assignment follows the authenticated principal, not the statement: + +1. A **direct mapping** on the user or service account wins. +2. Otherwise, for users only, QuestDB looks at the mappings of the ACL groups + the user belongs to and takes the highest `mapping_priority`. Ties go to the + most recently created mapping. +3. Otherwise the query runs in **DEFAULT**. + +Service accounts inherit nothing from ACL groups; they are either mapped +directly or they run in DEFAULT. + +The group is resolved once, when the query registers, and stays fixed for the +statement's lifetime. Changing a mapping affects statements that start after the +change, never one already running. + +`DEFAULT` always exists. By default it carries no limits of its own, so unmapped +principals run with a CPU weight of 100, no CPU cap, unlimited admission, and +the instance-wide memory limits. You can change its policy, but you cannot drop +or rename it. + +The default behaviour is: + +| Setting | Behaviour | +| ----------------------------------- | ------------------------------------------------------ | +| Feature enabled | Yes, when the SQL worker pools support Fiber execution | +| Group admission | Unlimited active and queued queries | +| Group CPU | Weight 100; no percentage cap | +| Group and process memory budgets | Unlimited unless configured | +| Existing single-query memory limits | Still apply, including principal-specific limits | +| Memory accounting without limits | Remains enabled for tracked native query memory | + +## What is managed + +Resource groups govern the statements that read data: + +- `SELECT` +- the source query of `CREATE TABLE ... AS SELECT` and `INSERT ... SELECT` +- query exports + +Everything else runs outside the feature and consumes no admission slot, CPU +grant or group memory budget: `EXPLAIN`, value `INSERT`, `UPDATE`, ordinary DDL, +`COPY`, transaction and session control, ILP ingestion, WAL apply, materialized +and live view refresh, and QuestDB's own internal SQL. + +`EXPLAIN` is deliberately outside the feature. It walks a plan tree and opens no +base cursor, so it reads no data, and holding an admission slot for it would +block the one statement an operator reaches for while a group is saturated. + +For `CREATE TABLE ... AS SELECT` and `INSERT ... SELECT` the owner covers cursor +open, the source scan, transforms, parallel query work and the row pump. Source +evaluation and writer append are fused in that pump, so inseparable foreground +CPU may be charged conservatively to the group. Durability, the commit and any +work handed to writer or WAL queues are outside the guarantee. + +Resource groups account **tracked native query memory**. They do not represent +JVM heap, resident set size, memory-mapped table pages or long-lived engine +caches. Existing process memory protection remains the outer boundary. + +## What each control guarantees + +The four controls differ in how strong their guarantee is, which matters when +you decide what to configure. + +### Admission is a hard gate + +`max_active_queries` is an exact count. A group at its limit queues the next +query until a slot frees, up to `max_queued_queries`; beyond that the query is +rejected immediately. A queued query that waits longer than `queue_timeout` +fails. + +A slot is held only while the query is actually executing a segment. A protocol +cursor that is suspended between pages releases its slot and passes through the +gate again when the client asks for more rows, so a paging client does not hold +capacity while the application thinks. The consequence is that admission can be +refused on a later page: a client that received its first rows may still see the +queue-full or timeout error when it asks for more, and the connection stays +usable. + +### CPU weight is a share, not a reservation + +Weights only matter when groups compete. A group that is alone on the instance +uses everything it can, regardless of its weight. When two groups both have +work, the scheduler hands out CPU so that measured CPU divided by `cpu_weight` +stays balanced: weights 100 and 50 converge to a 2:1 split of query CPU. + +Weights are relative. 100 and 50 are the same as 2 and 1. A group that becomes +active starts level with the groups already running, so it neither banks the CPU +it did not use while idle nor is punished for having been busy. + +Within a group, pending execution requests are served in arrival order. A +parallel query can submit more than one request, so this does not promise equal +CPU shares between individual queries. + +### The CPU cap is a rate, not an instantaneous ceiling + +`cpu_max_percent` is enforced with a token bucket measured in CPU nanoseconds +against the instance's CPU capacity. It is an average over a short window, not a +per-instant limit: a capped group that has been idle may burst for about 100 ms +of accumulated allowance before it is pushed back to its configured rate. Usage +beyond a grant becomes debt that must be repaid before the group runs again, so +the average holds even when an individual query overruns. + +Capacity comes from `resource.groups.cpu.capacity.cores`, which detects +container CPU quota by default. On a fractional quota such as 500m, detection +preserves the fraction, so a 50% cap really means half of half a core. + +### Memory limits use batched accounting + +Accounting has three levels: query, group and process. Allocation and release +deltas accumulate locally on the executing worker and are published to the +shared counters at an adaptive threshold or an execution boundary. Exceeding a +checked limit fails the query with `query memory limit exceeded`; it does not +queue the allocation until memory becomes available. + +The single-query ceiling starts with the principal's effective query memory +limit, when set, or the instance default `cairo.query.memory.limit.bytes`. Any +group `memory_limit` and process memory budget further cap that ceiling. The +group budget also bounds the total tracked memory held by its queries; the +process budget covers tracked native query memory across groups. + +An unset group `memory_limit` adds no group ceiling. A process budget of `0` +adds no process ceiling. Existing single-query limits still apply, and memory +accounting remains enabled even when all limits are unlimited. + +The counters can temporarily omit worker-local deltas. A group can therefore +briefly overshoot its limit by a bounded amount related to the number of workers +running its queries. These budgets are not byte-exact, instantaneous ceilings. + +## Why CPU control is cooperative + +QuestDB does not preempt a running query. The scheduler grants a query a short +slice of CPU on a worker and expects it to reach a cooperative checkpoint, which +is the same circuit breaker check that makes queries cancellable. At that point +the query either renews its grant or yields the worker to another group. + +Yielding returns the worker to its dispatch loop rather than to the end of the +query. The loop interleaves other queries on that worker and, within a bounded +window, hands it back to network I/O so new connections are accepted. A long +single-threaded query that reaches these checkpoints can therefore share its +worker before finishing. This improves responsiveness while heavy queries run, +even before any custom group policy is written. Resource groups add this CPU +time slicing to the existing cancellation and I/O suspension mechanisms. + +Two consequences follow. + +The guarantee is statistical over a short window. Between checkpoints a query +holds its worker, so instantaneous CPU can deviate from the configured share. +Settlement charges the measured CPU either way, so a query that overran repays +it and the average is preserved. + +A query that cannot reach a checkpoint keeps its worker. When managed CPU +accounting is engaged, its CPU is charged when the grant settles, but no +cooperative limit can shorten that stretch. + +## Behaviour under failure and on replicas + +Resource groups are stored in a replicated system catalog, so a read-only +replica receives group definitions and mappings through normal replication. + +- A **fresh replica** that has not yet received the catalog runs queries + unmanaged, exactly as if the feature were disabled, and reports how many + queries took that path. It does not reject queries or serve them under a + policy it cannot see yet. +- A **replica being promoted** validates the catalog after replication has + switched and before writes are admitted. If the old primary predated resource + groups and never created the catalog table, the promoted node creates it and + continues. With the feature enabled, a catalog that is unreadable or still + lagging refuses the promotion: the switch fails part-way, the node lands in + the `UNKNOWN` role and keeps serving reads as before, and the log names + `RESOURCE_GROUP_CATALOG_UNAVAILABLE` or `RESOURCE_GROUP_CATALOG_LAGGING`. + Retrying the switch repeats the check. With the feature disabled the condition + is logged and the promotion proceeds. +- At **startup** an unreadable catalog stops an instance with the feature + enabled from starting, in either role. A lagging catalog does not: the + instance starts and the refresh job catches up. +- If **CPU scheduling** hits an internal fault, it degrades: queries continue to + run without CPU grants, and the condition is visible in metrics until the + instance restarts. Admission and memory limits do not depend on CPU scheduling + and stay enforced. +- A **fault on one query** fences only that query's owner. Other queries and + other groups are unaffected, and the faulted segment is charged conservatively + rather than being dropped from the accounting. + +## Cost when nothing competes + +While a single uncapped group owns all running queries, dispatch takes a +lock-free path that avoids CPU sampling and weighted scheduling accounting. +Query registration, admission, cooperative checks and memory accounting still +run, so this does not imply the same cost as disabling the feature. Actual +overhead depends on the workload. Managed scheduling engages as soon as a second +group has work or a capped group is active, and disengages again when it does +not. The transition happens at the next dispatch boundary, not at a query +boundary, so a newly arriving group does not wait for a long query to finish +before its policy applies. + +## See also + +- [Configure and use resource groups](/docs/operations/resource-groups/) +- [Resource groups configuration](/docs/configuration/resource-groups/) +- [Role-based access control](/docs/security/rbac/) diff --git a/documentation/configuration/resource-groups.md b/documentation/configuration/resource-groups.md new file mode 100644 index 000000000..d2cbca800 --- /dev/null +++ b/documentation/configuration/resource-groups.md @@ -0,0 +1,101 @@ +--- +title: Resource groups +sidebar_label: Resource groups +description: + Configuration settings for QuestDB Enterprise resource groups, covering the + master switch, CPU capacity, memory ceiling and admission defaults. +--- + +:::note + +Resource groups are [Enterprise](/enterprise/) only. + +::: + +[Resource groups](/docs/concepts/resource-groups/) isolate competing query +workloads inside one instance. These settings are instance-wide. The per-group +policy that decides admission, CPU share and memory budgets is set in SQL, not +here. See [Configure and use resource groups](/docs/operations/resource-groups/) +for those statements. + +None of these settings are reloadable: changing any of them requires a restart. + +Resource groups also require access control to be enabled (`acl.enabled=true`) +before principals can be mapped to a group, and every pool that executes SQL +must run in Fiber mode, which is the default. What happens when a pool is in +legacy mode depends on how the feature was turned on. Left at its default, it +turns itself off and logs an error naming the pool and the setting to change. +Asked for explicitly, it fails startup with the same error, because an explicit +request and a legacy pool cannot both be honoured. + +## General + +### resource.groups.enabled + +- **Default**: `true` +- **Reloadable**: no + +Master switch. When `false`, resource group admission, CPU scheduling and group +memory accounting are disabled. Group definitions and principal mappings remain +in the catalog, so turning the feature back on restores the policies that were +already there. + +`true` also makes the catalog a hard dependency: an instance whose catalog +cannot be read does not start, and a replica whose catalog is not current is not +promoted. With `false`, both conditions are logged and ignored. See +[Behaviour under failure and on replicas](/docs/concepts/resource-groups/#behaviour-under-failure-and-on-replicas). + +Existing principal-specific and instance-default single-query memory limits +continue to apply when resource groups are disabled. + +Left unset, this resolves to `false` on an instance whose SQL pools are in +legacy mode, so upgrading such an instance does not turn the feature on and does +not stop the instance from starting. `SHOW PARAMETERS` then reports `false`, +which is the value that took effect. Set it to `true` explicitly and a legacy +pool becomes a startup error instead. + +### resource.groups.cpu.capacity.cores + +- **Default**: `auto` +- **Reloadable**: no + +The CPU capacity that `cpu_max_percent` is a percentage of, as `auto` or a +positive decimal number of cores. `auto` detects the process affinity mask and +the most restrictive cgroup quota, preserving fractional quotas such as `500m`, +so a 50% cap on half a core really means a quarter of a core. An explicit value +is capped by successful detection; if detection fails, the explicit value stands +and the failure is logged. + +When detection fails and no explicit value is set, capacity falls back to the +processor count and `questdb_resource_groups_cpu_capacity_fallback` reports `1`. + +### resource.groups.process.memory.limit.bytes + +- **Default**: `0` +- **Reloadable**: no + +Ceiling for tracked native query memory across all groups. `0` leaves the +instance without a process ceiling, which is the default. When set, it bounds +every group and every query, so no group policy can grant more than this. + +An unlimited process budget does not disable memory accounting or remove an +existing single-query limit. The group-level SQL parameter `memory_limit` uses +`RESET (memory_limit)` to clear its ceiling; it does not accept `0`. + +This is not a process RSS limit. It covers tracked query memory only, not JVM +heap, memory-mapped table pages or long-lived engine caches. + +### resource.groups.queue.timeout.millis + +- **Default**: `30000` +- **Reloadable**: no + +How long a queued query waits for an admission slot in a group that does not set +its own `queue_timeout`. A query that waits longer fails with +`Resource Group admission queue timeout`. + +## See also + +- [Resource groups concept](/docs/concepts/resource-groups/) +- [Configure and use resource groups](/docs/operations/resource-groups/) +- [Identity and Access Management configuration](/docs/configuration/iam/) diff --git a/documentation/operations/logging-metrics.md b/documentation/operations/logging-metrics.md index 240647f3c..ce3e3bf94 100644 --- a/documentation/operations/logging-metrics.md +++ b/documentation/operations/logging-metrics.md @@ -1,10 +1,12 @@ --- title: Logging and metrics -description: Configure and understand QuestDB logging and metrics, including log levels, configuration options, and Prometheus integration. +description: + Configure and understand QuestDB logging and metrics, including log levels, + configuration options, and Prometheus integration. --- - -This page outlines logging in QuestDB. It covers how to configure logs via `log.conf` and expose metrics via Prometheus. +This page outlines logging in QuestDB. It covers how to configure logs via +`log.conf` and expose metrics via Prometheus. - [Logging](/docs/operations/logging-metrics/#logging) - [Metrics](/docs/operations/logging-metrics/#metrics) @@ -206,19 +208,20 @@ For configuration options, see the :::warning On systems with -[8 Cores and less](/docs/getting-started/capacity-planning/#cpu-cores), contention -for threads might increase the latency of health check service responses. If you -use a load balancer, and it thinks the QuestDB service is dead with nothing -apparent in the QuestDB logs, you may need to configure a dedicated thread pool -for the health check service. To do so, increase `http.min.worker.count` to `1`. +[8 Cores and less](/docs/getting-started/capacity-planning/#cpu-cores), +contention for threads might increase the latency of health check service +responses. If you use a load balancer, and it thinks the QuestDB service is dead +with nothing apparent in the QuestDB logs, you may need to configure a dedicated +thread pool for the health check service. To do so, increase +`http.min.worker.count` to `1`. ::: #### Lifecycle endpoint `GET /lifecycle` on the same port returns the startup and shutdown state of -every server component as JSON, for probes and coordinators that need more -than the `200` of the health check: +every server component as JSON, for probes and coordinators that need more than +the `200` of the health check: ```shell curl http://127.0.0.1:9003/lifecycle @@ -338,23 +341,23 @@ When [cold storage](/docs/concepts/cold-storage/) is enabled, the endpoint exposes fifteen additional metrics under the `questdb_cold_chunk_` prefix, covering the chunk cache and the range reads that serve remote partitions: -| Metric | Type | Description | -| ------ | ---- | ----------- | -| `questdb_cold_chunk_acquire_full_hit_total` | counter | Reads that found every chunk already resident | -| `questdb_cold_chunk_acquire_partial_hit_total` | counter | Reads that found some chunks and fetched the rest | -| `questdb_cold_chunk_acquire_full_miss_total` | counter | Reads where every chunk had to be fetched | -| `questdb_cold_chunk_acquire_hit_chunks_total` | counter | Chunk lookups served from the cache | -| `questdb_cold_chunk_acquire_miss_chunks_total` | counter | Chunk lookups that had to be fetched | -| `questdb_cold_chunk_download_started_total` | counter | Range requests dispatched, one per coalesced group | -| `questdb_cold_chunk_download_finished_total` | counter | Range requests that returned data | -| `questdb_cold_chunk_download_failed_total` | counter | Range requests that failed after retries | -| `questdb_cold_chunk_download_coalesced_total` | counter | Readers that attached to an in-flight download instead of starting a new one | -| `questdb_cold_chunk_release_evictions_total` | counter | Chunks evicted when their last lease was released | -| `questdb_cold_chunk_in_flight_downloads` | gauge | Range requests dispatched but not yet complete | -| `questdb_cold_chunk_pending_batches` | gauge | Batches the read coordinator is tracking | -| `questdb_cold_chunk_busy_leases` | gauge | Currently allocated leases | -| `questdb_cold_chunk_ready_chunks` | gauge | Chunks resident in the ready cache | -| `questdb_cold_chunk_pinned_bytes` | gauge | Compressed bytes resident in the ready cache | +| Metric | Type | Description | +| ---------------------------------------------- | ------- | ---------------------------------------------------------------------------- | +| `questdb_cold_chunk_acquire_full_hit_total` | counter | Reads that found every chunk already resident | +| `questdb_cold_chunk_acquire_partial_hit_total` | counter | Reads that found some chunks and fetched the rest | +| `questdb_cold_chunk_acquire_full_miss_total` | counter | Reads where every chunk had to be fetched | +| `questdb_cold_chunk_acquire_hit_chunks_total` | counter | Chunk lookups served from the cache | +| `questdb_cold_chunk_acquire_miss_chunks_total` | counter | Chunk lookups that had to be fetched | +| `questdb_cold_chunk_download_started_total` | counter | Range requests dispatched, one per coalesced group | +| `questdb_cold_chunk_download_finished_total` | counter | Range requests that returned data | +| `questdb_cold_chunk_download_failed_total` | counter | Range requests that failed after retries | +| `questdb_cold_chunk_download_coalesced_total` | counter | Readers that attached to an in-flight download instead of starting a new one | +| `questdb_cold_chunk_release_evictions_total` | counter | Chunks evicted when their last lease was released | +| `questdb_cold_chunk_in_flight_downloads` | gauge | Range requests dispatched but not yet complete | +| `questdb_cold_chunk_pending_batches` | gauge | Batches the read coordinator is tracking | +| `questdb_cold_chunk_busy_leases` | gauge | Currently allocated leases | +| `questdb_cold_chunk_ready_chunks` | gauge | Chunks resident in the ready cache | +| `questdb_cold_chunk_pinned_bytes` | gauge | Compressed bytes resident in the ready cache | Watch rates and ratios rather than raw totals. Sustained `download_failed_total`, `pending_batches` sitting at its configured cap, or @@ -369,10 +372,52 @@ _Enterprise only._ Two gauges describe the state of an in-place [role switch](/docs/high-availability/failover/): -| Metric | Type | Description | -| ------ | ---- | ----------- | +| Metric | Type | Description | +| ---------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `questdb_replication_pending_upload_txn` | gauge | Committed transactions not yet uploaded to the object store, summed over the replicated tables. Poll it before demoting a primary: a demote that cannot bring it to zero within its timeout is not completed | -| `questdb_backup_active_at_last_demote` | gauge | `1` if a backup was still running when the node was last demoted, `0` otherwise. Cleared by the next promotion | +| `questdb_backup_active_at_last_demote` | gauge | `1` if a backup was still running when the node was last demoted, `0` otherwise. Cleared by the next promotion | + +### Resource group metrics + +_Enterprise only._ + +When [resource groups](/docs/concepts/resource-groups/) are enabled, the +endpoint exposes one series per group, labelled with `resource_group`: + +| Metric | Type | Description | +| --------------------------------------------------- | ------- | ------------------------------------------------------------ | +| `questdb_resource_group_active_queries` | gauge | Queries holding an admission slot | +| `questdb_resource_group_queued_queries` | gauge | Queries waiting for a slot | +| `questdb_resource_group_oldest_queue_wait_millis` | gauge | How long the longest waiting query has waited | +| `questdb_resource_group_memory_bytes` | gauge | Tracked query memory in use | +| `questdb_resource_group_memory_limit_bytes` | gauge | Effective group memory ceiling, `0` when the group sets none | +| `questdb_resource_group_cpu_nanos_total` | counter | CPU charged to the group | +| `questdb_resource_group_cpu_wait_nanos_total` | counter | Time the group spent waiting for CPU | +| `questdb_resource_group_cpu_max_percent` | gauge | Effective CPU cap, `-1` when uncapped | +| `questdb_resource_group_admission_rejections_total` | counter | Queries rejected because the queue was full | +| `questdb_resource_group_admission_timeouts_total` | counter | Queries that timed out while queued | + +The single uncapped group dispatch path does not sample CPU. Consequently, +`cpu_nanos_total` counts CPU measured by managed scheduling, not every query's +CPU consumption. A flat counter does not imply that the group is idle; also +check `questdb_resource_groups_cpu_managed_dispatch` and query activity. Memory +gauges show published accounting and can lag worker-local deltas. + +Instance-wide series describe the feature itself: + +| Metric | Type | Description | +| ------------------------------------------------------- | ------- | --------------------------------------------------------------------- | +| `questdb_resource_groups_enabled` | gauge | `1` when the feature is on | +| `questdb_resource_groups_catalog_current` | gauge | `1` when the group catalog is current; `0` while a replica catches up | +| `questdb_resource_groups_catalog_lag_unmanaged_queries` | counter | Queries that ran unmanaged because the catalog was not current yet | +| `questdb_resource_groups_cpu_capacity_microcores` | gauge | Capacity that `cpu_max_percent` applies to | +| `questdb_resource_groups_cpu_capacity_fallback` | gauge | `1` when capacity detection failed and the processor count was used | +| `questdb_resource_groups_cpu_managed_dispatch` | gauge | `1` while managed CPU scheduling is engaged | +| `questdb_resource_groups_cpu_scheduler_degraded` | gauge | `1` when CPU scheduling has degraded to unmanaged | + +A non-zero `questdb_resource_groups_cpu_scheduler_degraded` means CPU shares are +no longer enforced until the instance restarts. Admission and memory limits stay +enforced. ### Prometheus Alertmanager diff --git a/documentation/operations/resource-groups.md b/documentation/operations/resource-groups.md new file mode 100644 index 000000000..e44999e16 --- /dev/null +++ b/documentation/operations/resource-groups.md @@ -0,0 +1,577 @@ +--- +title: Resource groups +sidebar_label: Resource groups +description: + Create resource groups, map users and ACL groups to them, and tune admission, + CPU and memory limits so one workload cannot starve another. +--- + +import { EnterpriseNote } from "@site/src/components/EnterpriseNote" + + + Resource groups isolate competing query workloads inside a single QuestDB + instance. + + +This page covers day-to-day use: creating groups, mapping principals, choosing +limits, and watching the result. For what the limits actually guarantee, read +[the concept page](/docs/concepts/resource-groups/) first. + +## Quick start + +This example separates reporting from the default workload and limits its +concurrency and memory. Run it as an administrator on an instance that meets the +[requirements](#requirements). Use unused example names and replace the password +placeholders. Later examples on this page can be adapted independently. + +First create the ACL principals and allow SQL connections: + +```questdb-sql +CREATE GROUP analysts; +GRANT HTTP, PGWIRE TO analysts; +CREATE USER reporting_user WITH PASSWORD ''; +ADD USER reporting_user TO analysts; + +CREATE USER nightly_batch WITH PASSWORD ''; +GRANT HTTP, PGWIRE TO nightly_batch; +``` + +Then create the resource group and mappings: + +```questdb-sql +-- 1. Create a group. Unset parameters fall back to the instance defaults. +CREATE RESOURCE GROUP reporting WITH ( + cpu_weight = 50, + max_active_queries = 4, + max_queued_queries = 32, + queue_timeout = '15s', + memory_limit = '2G' +); + +-- 2. reporting_user inherits this mapping unless a higher-precedence one applies. +ALTER GROUP analysts SET RESOURCE GROUP reporting MAPPING PRIORITY 10; + +-- 3. Map one user directly. A direct mapping beats any ACL group mapping. +ALTER USER nightly_batch SET RESOURCE GROUP reporting; +``` + +Verify: + +```questdb-sql +SELECT name, cpu_weight, max_active_queries, active_queries, queued_queries +FROM resource_groups(); + +SELECT * FROM resource_group_mappings(); +``` + +Reconnect as `reporting_user` or `nightly_batch` and run: + +```questdb-sql +SELECT current_resource_group(); +``` + +| current_resource_group | +| ---------------------- | +| reporting | + +These grants allow connections. Grant access to the application's tables +separately, as described in [RBAC](/docs/security/rbac/). + +Everything not mapped keeps running in `DEFAULT`, which has a CPU weight of 100. +Against `reporting`'s weight of 50, that is a 2:1 split of query CPU while both +have work. + +## Requirements + +- QuestDB Enterprise. +- Access control enabled (`acl.enabled=true`). Groups can be created without it, + but mapping statements require it, since mappings attach to ACL principals. +- The pools that execute SQL must run in Fiber mode, which is the default, + because cooperative admission and CPU control cannot be made complete on the + legacy path. On an instance whose pools are in legacy mode, resource groups + left at their default turn themselves off and log an error naming the pool and + the setting to change. Setting `resource.groups.enabled=true` on such an + instance fails startup with that same error. +- Administrator rights for group management, mappings and instance-wide + inspection. Ordinary users can call `current_resource_group()` to check their + own query's group. + +## Configuration + +Resource groups are enabled by default. These are instance-wide settings; the +per-group policy is set in SQL. Each setting is described in full in the +[resource groups configuration reference](/docs/configuration/resource-groups/). + +| Property | Default | Meaning | +| -------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------- | +| `resource.groups.enabled` | `true` | Set to `false` to disable resource group enforcement. Existing single-query memory limits still apply. | +| `resource.groups.cpu.capacity.cores` | `auto` | CPU capacity that `cpu_max_percent` is a percentage of. `auto` detects container quota, including fractional quotas. | +| `resource.groups.process.memory.limit.bytes` | `0` | Ceiling for tracked query memory across all groups, `0` for none. Every group limit is capped by it. | +| `resource.groups.queue.timeout.millis` | `30000` | Default admission queue timeout for groups that do not set `queue_timeout`. | + +Turning the feature off is a restart with `resource.groups.enabled=false`. +Definitions and mappings stay in the catalog, so nothing is lost and the +policies apply again when it is re-enabled. + +## Managing groups + +```questdb-sql +CREATE RESOURCE GROUP analytics; + +CREATE RESOURCE GROUP IF NOT EXISTS analytics WITH (cpu_weight = 300); + +ALTER RESOURCE GROUP analytics SET (cpu_weight = 300, cpu_max_percent = 25.5); + +-- Clear parameters so they fall back to the instance defaults again. +ALTER RESOURCE GROUP analytics RESET (memory_limit, cpu_max_percent); + +ALTER RESOURCE GROUP analytics RENAME TO reporting; + +DROP RESOURCE GROUP reporting; +DROP RESOURCE GROUP IF EXISTS reporting; +``` + +A group policy change applies online to the shared group budget. It does not +cancel existing queries at the moment `ALTER` runs: + +| Change | Effect on existing work | +| ---------------------- | ------------------------------------------------------------------------------------------------- | +| CPU weight or cap | Subsequent scheduling uses the new policy; issued CPU grants are settled normally | +| Active-query limit | Existing slots are retained; subsequent admission, including a resumed cursor, uses the new limit | +| Queue limit or timeout | New admission requests use the new settings; an already queued request keeps its deadline | +| Group memory limit | Subsequent allocations check the new budget; existing memory is released normally | + +Lowering a memory budget below current usage can make subsequent allocations +fail. The principal-specific or instance-default single-query limit is captured +when the query starts; updating the group budget does not replace that limit. +Changing a principal mapping affects new queries only. + +`DROP` is refused while any live principal is still mapped to the group; unmap +them first. Once unmapped, a group can be dropped while queries still use it. It +disappears from `resource_groups()` immediately. Running and queued queries, +including suspended cursors, continue using the deleted group's existing +settings. Their memory still counts towards the process budget. + +Recreating a group with the same name starts fresh usage counters. Queries that +still use the deleted group do not move to the new group or use its settings. +Map principals to the new group to assign their subsequent queries to it. + +`DEFAULT` cannot be dropped or renamed, but it can be altered: + +```questdb-sql +ALTER RESOURCE GROUP DEFAULT SET (max_active_queries = 16); +``` + +## Mapping principals + +```questdb-sql +ALTER USER alice SET RESOURCE GROUP analytics; +ALTER SERVICE ACCOUNT ingest_bot SET RESOURCE GROUP analytics; +ALTER GROUP analysts SET RESOURCE GROUP analytics MAPPING PRIORITY 10; + +ALTER USER alice UNSET RESOURCE GROUP; +ALTER GROUP analysts UNSET RESOURCE GROUP; +``` + +`MAPPING PRIORITY` is a non-negative integer and applies only to ACL group +mappings, because a user can belong to several ACL groups. The highest priority +wins; ties go to the most recent mapping. It defaults to 0 and is rejected on +user and service account mappings, which are one-to-one. + +Resolution order for a query is: direct mapping on the principal, then the +highest-priority mapping among the user's ACL groups, then `DEFAULT`. Service +accounts do not inherit ACL group mappings. + +## Policy parameters + +All parameters are optional. An unset parameter is not "unlimited" in every +case: it falls back to the instance default shown here. + +| Parameter | Accepted values | Unset behaviour | +| -------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------- | +| `cpu_weight` | integer, 1 to 10000 | 100 | +| `cpu_max_percent` | 0.01 to 100, at most two decimals | no cap | +| `max_active_queries` | integer, 1 or more | unlimited | +| `max_queued_queries` | integer, 0 or more | unlimited | +| `queue_timeout` | a positive whole number of milliseconds, or a duration such as `'15s'`, `'2m'` | `resource.groups.queue.timeout.millis` | +| `memory_limit` | a positive byte size, plain or suffixed such as `'8G'` | no group ceiling; other memory limits still apply | + +`memory_limit` is the budget for everything the group runs at once. Where the +instance sets `resource.groups.process.memory.limit.bytes`, the group budget is +capped by it, so a group cannot be granted more than the instance allows. A +group ceiling only lowers what its queries may use; it never raises a limit set +elsewhere. + +A group that does not set `memory_limit` carries no ceiling of its own, and +`resource_groups().memory_limit_bytes` reports `0` for it. Its queries are then +bounded by any existing single-query limit and the process limit. A principal's +effective query memory limit takes precedence over the instance default +`cairo.query.memory.limit.bytes`; group and process budgets can only lower the +resulting ceiling. Resource groups do not have a separate `query_memory_limit` +policy parameter. + +To remove a group memory ceiling, use +`ALTER RESOURCE GROUP reporting RESET (memory_limit)`. Setting the SQL parameter +to `0` is invalid; `0` means unlimited for the instance process-memory property. +Accounting continues when limits are unlimited. + +Two examples of what the values mean in practice: + +```questdb-sql +-- A share: reporting gets a third of query CPU when DEFAULT also has work, +-- and all of it when DEFAULT is idle. +CREATE RESOURCE GROUP reporting WITH (cpu_weight = 50); + +-- A ceiling: exports never average more than a quarter of instance CPU, +-- even when the instance is otherwise idle. +CREATE RESOURCE GROUP exports WITH (cpu_max_percent = 25); +``` + +Use `cpu_weight` to decide who wins under contention, and `cpu_max_percent` to +leave headroom for work that resource groups do not manage, such as ingestion +and WAL apply. Setting a cap on a group also switches the whole instance to +managed scheduling while that group has queries. + +## Common scenarios + +### The instance stops answering while CPU looks idle + +Every HTTP or PGWire worker is occupied by a long query, new requests are not +picked up, and instance CPU is low because those queries run on one core each. +Clients time out and retry, which produces more of the same queries. A plan such +as a `LATEST ON` over a non-indexed filter is a typical cause: it scans frames +on one thread, so it is slow without ever being CPU-hungry. + +Four steps. The first is a prerequisite to confirm, the second is what enabling +the feature already gives you, and the last two are policy you choose. + +**1. Confirm the SQL pools are Fiber pools.** This is the prerequisite for +everything below. A protocol runs either on its own pool, when its worker count +is above zero, or on the shared network pool. The setting that matters is the +one for the pool it actually uses: + +| Where the protocol runs | Setting to check | +| ---------------------------------------------------- | ------------------------------------- | +| Its own HTTP pool (`http.worker.count` above zero) | `http.worker.fiber.enabled` | +| Its own PGWire pool (`pg.worker.count` above zero) | `pg.worker.fiber.enabled` | +| The shared network pool (worker count zero, default) | `shared.network.worker.fiber.enabled` | + +Parallel query work is separate and follows `shared.query.worker.fiber.enabled` +whenever the shared query pool has workers. A shared query pool set to zero +workers turns parallel SQL off by default and needs no check of its own. + +The first two settings default to `true`, so a dedicated pool is a Fiber pool +unless someone turned it off. `shared.network.worker.fiber.enabled` defaults to +`true` exactly when HTTP or PGWire actually runs there, which is the case out of +the box because both worker counts default to zero. You normally have nothing to +change here; check these only when the instance was tuned by hand. + +After the restart, confirm the feature came up. `SHOW PARAMETERS` must report +`resource.groups.enabled` as `true`, and `questdb_resource_groups_enabled` must +be `1`. If a pool that executes SQL is in legacy mode and resource groups were +left at their default, the feature disables itself and logs the reason. An +explicit `resource.groups.enabled=true` fails startup in that configuration. + +**2. Enabling the feature already frees the workers.** A query yields its worker +at the checkpoints that already make it cancellable, so a long single-threaded +scan releases the worker while it is still running and the instance keeps +accepting connections. This needs no group and no policy, and it holds even when +every query resolves to `DEFAULT`. + +There is no separate switch to verify. Cooperative yielding is on exactly when +resource groups are on, which step 1 already confirmed. Fiber pools on their own +do not produce it: the checkpoints are compiled into every build, but they only +yield while resource groups are enabled. A query that never reaches a checkpoint +still holds its worker, so this does not remove every cause of an unresponsive +instance. + +**3. Separate the workloads so shares apply.** While a single uncapped group +owns every running query, dispatch stays on the unmanaged path and weights have +nothing to arbitrate. Two groups with queries in flight at the same time, or any +group with a `cpu_max_percent`, is what engages weighted scheduling: + +```questdb-sql +CREATE RESOURCE GROUP dashboards WITH (cpu_weight = 400); +CREATE RESOURCE GROUP adhoc WITH (cpu_weight = 100); + +ALTER USER app SET RESOURCE GROUP dashboards; +ALTER USER analyst SET RESOURCE GROUP adhoc; +``` + +**4. Bound concurrent requests with admission.** + +```questdb-sql +ALTER RESOURCE GROUP adhoc SET ( + max_active_queries = 4, + max_queued_queries = 8, + queue_timeout = '5s' +); +``` + +The fifth concurrent query waits instead of running, and it does not hold a +worker while it waits. The thirteenth fails immediately with +`Resource Group admission queue is full`, so a client that keeps resending gets +a clear answer in seconds instead of adding to the pile. + +Admission directly bounds the number of concurrent queries. A CPU cap bounds +their combined CPU rate and can also slow a single-threaded query: on an 8-core +instance, a 10% cap permits 0.8 cores of CPU. Choose a cap when that rate limit +is useful; it does not replace the admission limits in this scenario. Clients +should use bounded retries with backoff after admission failures. + +Afterwards the symptom is also diagnosable rather than mysterious. Low instance +CPU together with a high `queued_queries` and a rising +`oldest_queue_wait_millis` on one group says the work is being held at the +admission gate, not that the machine is busy. `query_activity()` shows which +group each running query was admitted to. + +What resource groups do not do here: they do not make the slow plan faster, and +they do not bound how long one query may run. Wall-clock limits still come from +the instance-wide +[`query.timeout`](/docs/configuration/cairo-engine/#querytimeout). + +### Dashboards must stay responsive while analysts run heavy queries + +Use weights. Shares are per group, not per query, so a group running fifty +queries does not outvote a group running one: + +```questdb-sql +CREATE RESOURCE GROUP dashboards WITH (cpu_weight = 400); +CREATE RESOURCE GROUP analysts WITH (cpu_weight = 100); +``` + +When these are the only competing groups and both can use their shares, weights +target a 4:1 split of managed query CPU. Actual use also depends on runnable +work, available parallelism and any CPU caps. When analysts are idle, dashboards +can use the available query CPU. Weights are integers from 1 to 10000 and every +group starts at 100, so a group left alone keeps an equal share against any +group you do not change. + +### A background job must never take the whole instance + +Use a cap, which applies whether or not anything else is running: + +```questdb-sql +CREATE RESOURCE GROUP exports WITH (cpu_max_percent = 20); +``` + +This also leaves headroom for work resource groups do not manage, such as +ingestion and WAL apply. The cap accepts two decimals, down to `0.01`, and is a +percentage of the +[detected CPU capacity](/docs/configuration/resource-groups/#resourcegroupscpucapacitycores), +not of the host's core count, so it stays correct under a container quota. + +### One workload must not exhaust query memory + +Bound the group rather than each query, so the limit holds however many queries +the workload starts: + +```questdb-sql +CREATE RESOURCE GROUP reporting WITH (memory_limit = '8G'); +``` + +A query that would push the group over its budget fails with +`query memory limit exceeded` and releases what it held. + +### An ingestion or automation account runs queries too + +Service accounts resolve differently from users: they honour a direct mapping, +but they never inherit a mapping from an ACL group. A service account with no +direct mapping runs in `DEFAULT` however its ACL groups are mapped, so map it +explicitly: + +```questdb-sql +CREATE RESOURCE GROUP automation WITH (cpu_weight = 50, max_active_queries = 2); + +ALTER SERVICE ACCOUNT ingest_bot SET RESOURCE GROUP automation; +``` + +This governs the queries the account runs. It does not throttle ingestion +itself, which resource groups do not manage. + +### Many teams share one instance + +Map ACL groups rather than individual users, and use `MAPPING PRIORITY` to +decide what happens to someone who belongs to more than one: + +```questdb-sql +ALTER GROUP analysts SET RESOURCE GROUP adhoc MAPPING PRIORITY 10; +ALTER GROUP oncall SET RESOURCE GROUP dashboards MAPPING PRIORITY 20; +``` + +Someone in both groups resolves to `dashboards`, because the higher priority +wins. If two mappings tie on priority, the more recently created one wins. A +direct mapping on the user beats every group mapping regardless of priority, +which is the way to make one person an exception without touching the groups: + +```questdb-sql +ALTER USER lead_analyst SET RESOURCE GROUP dashboards; +``` + +`MAPPING PRIORITY` is rejected on user and service account mappings, because +those are one-to-one and have nothing to break a tie between. Confirm any of +this from the client's own session with `SELECT current_resource_group();`. + +## Inspecting + +`resource_groups()` returns one row per group, combining the configured policy +with live counters: + +| Column | Meaning | +| ------------------------------------------------------------------ | --------------------------------------------------- | +| `name` | Group name | +| `memory_limit_bytes` | Effective group memory budget | +| `max_active_queries`, `max_queued_queries`, `queue_timeout_millis` | Effective admission policy | +| `cpu_weight`, `cpu_max_percent` | Effective CPU policy | +| `active_queries`, `queued_queries` | Live admission state | +| `oldest_queue_wait_millis` | How long the longest waiting query has waited | +| `memory_used_bytes` | Tracked query memory in use | +| `cpu_nanos_total`, `cpu_wait_nanos_total` | Cumulative CPU consumed and spent waiting for CPU | +| `admission_rejections`, `admission_timeouts` | Cumulative queue-full rejections and queue timeouts | + +`resource_group_mappings()` returns one row per mapping with `principal_type`, +`principal_name`, `principal_generation`, `resource_group_id`, `resource_group`, +`mapping_priority` and `mapping_revision`. + +`current_resource_group()` returns the calling query's group, which is the +quickest way to confirm a mapping from the client's own connection: + +```questdb-sql +SELECT current_resource_group(); +``` + +It returns `NULL` when that execution is unmanaged, including when the feature +is disabled or a replica's group catalog is not ready. See the +[function reference](/docs/query/functions/meta/#current_resource_group) for +permissions and return values, and the references for +[`resource_groups()`](/docs/query/functions/meta/#resource_groups) and +[`resource_group_mappings()`](/docs/query/functions/meta/#resource_group_mappings) +for complete schemas. + +`query_activity()` carries a `resource_group` column, so you can see which group +each running query was admitted to. It is `NULL` for executions that resource +groups do not manage: + +```questdb-sql +SELECT resource_group, username, query_start, query +FROM query_activity() +WHERE resource_group IS NOT NULL +ORDER BY query_start; +``` + +## Monitoring + +The Prometheus endpoint exposes one series per group, labelled with +`resource_group`. The full list lives in the +[metrics reference](/docs/operations/logging-metrics/#resource-group-metrics): + +``` +questdb_resource_group_active_queries{resource_group="reporting"} +questdb_resource_group_queued_queries{resource_group="reporting"} +questdb_resource_group_oldest_queue_wait_millis{resource_group="reporting"} +questdb_resource_group_memory_bytes{resource_group="reporting"} +questdb_resource_group_memory_limit_bytes{resource_group="reporting"} +questdb_resource_group_cpu_nanos_total{resource_group="reporting"} +questdb_resource_group_cpu_wait_nanos_total{resource_group="reporting"} +questdb_resource_group_cpu_max_percent{resource_group="reporting"} +questdb_resource_group_admission_rejections_total{resource_group="reporting"} +questdb_resource_group_admission_timeouts_total{resource_group="reporting"} +``` + +Instance-wide series: + +| Metric | Meaning | +| ------------------------------------------------------- | --------------------------------------------------------------------- | +| `questdb_resource_groups_enabled` | 1 when the feature is on | +| `questdb_resource_groups_catalog_current` | 1 when the catalog is current; 0 while a replica is still catching up | +| `questdb_resource_groups_catalog_lag_unmanaged_queries` | Queries that ran unmanaged because the catalog was not current yet | +| `questdb_resource_groups_cpu_capacity_microcores` | Capacity that `cpu_max_percent` applies to | +| `questdb_resource_groups_cpu_capacity_fallback` | 1 when capacity detection failed and the processor count was used | +| `questdb_resource_groups_cpu_managed_dispatch` | 1 while managed CPU scheduling is engaged | +| `questdb_resource_groups_cpu_scheduler_degraded` | 1 when CPU scheduling has degraded to unmanaged | + +Two signals are worth alerting on: a non-zero +`questdb_resource_groups_cpu_scheduler_degraded`, which means CPU shares are no +longer enforced until the next restart, and a steadily growing +`questdb_resource_group_admission_timeouts_total`, which means a group's queue +settings are rejecting work the application expects to succeed. + +## Errors clients see + +| Message | Cause | Usual fix | +| ----------------------------------------------------------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | +| `Resource Group admission queue is full` | The group is at `max_active_queries` and its queue is at `max_queued_queries` | Raise the limits, or let the client retry | +| `Resource Group admission queue timeout` | The query waited longer than `queue_timeout` | Raise `queue_timeout` or `max_active_queries`, or reduce concurrency | +| Either admission error while fetching a later page | A suspended cursor re-enters admission when the client asks for more rows | Adjust admission limits or retry the query with backoff; the failed cursor cannot continue | +| `query memory limit exceeded` | A single-query, group or process memory limit rejected an allocation | Inspect `query_activity().memory_limit` and the group/process budgets; reduce memory use or adjust the relevant limit | +| `Resource Group is referenced by an active principal link` | `DROP RESOURCE GROUP` while principals are still mapped | `UNSET RESOURCE GROUP` on those principals first | +| `built-in Resource Group cannot be dropped` / `cannot be renamed` | `DROP` or `RENAME` on `DEFAULT` | Alter it instead | + +## Troubleshooting + +**A group's CPU share is not what I configured.** Weights only apply while +groups compete. Check `active_queries` on both groups at the same moment: if one +is idle, the other is expected to use everything. Also confirm the work you are +watching is managed at all, since ingestion, WAL apply and view refresh are +outside the feature. `query_activity()` shows the group each running query +belongs to, which is the quickest way to tell whether the load you are watching +is attributed where you expect. + +**A capped group is slower than the cap suggests.** Very small caps release CPU +in pulses. The cap is a rate over roughly a 100 ms window, so a group whose +share works out to less than one 2 ms slice per window waits between slices. For +example, 0.1% of an 8-core instance allows about 8 ms of CPU per second. Small +caps still allow progress, but the waits between slices can substantially +increase latency; there is no special 0.25% cutoff. + +**Queries on a fresh replica are not limited.** Until the catalog has +replicated, a replica runs queries unmanaged and counts them in +`questdb_resource_groups_catalog_lag_unmanaged_queries`. The counter stops +growing once `questdb_resource_groups_catalog_current` reaches 1. + +**Promotion fails naming the resource group catalog.** With the feature enabled, +`SWITCH ROLE TO PRIMARY` does not admit writes over a catalog that is unreadable +or lagging. The node lands in the `UNKNOWN` role and still serves reads; the +server log names `RESOURCE_GROUP_CATALOG_LAGGING` or +`RESOURCE_GROUP_CATALOG_UNAVAILABLE`. Lagging means the replica has not finished +applying the access control transactions the catalog depends on: wait for WAL +apply to catch up and run `SWITCH ROLE TO PRIMARY` again. Unavailable means the +catalog table is missing or its contents cannot be read, and retrying does not +help: promote another replica, or restart this node as primary with +`resource.groups.enabled=false`, which turns the check into a logged error. See +[Refusals and the torn state](/docs/high-availability/failover/#refusals-and-the-torn-state). + +**Startup fails naming the resource group catalog.** The catalog table cannot be +read while the feature is enabled; the log says +`Resource Group catalog startup validation failed`. Starting with +`resource.groups.enabled=false` logs the condition instead of failing. + +**Startup fails naming a worker pool.** A pool that executes SQL is in legacy +mode while `resource.groups.enabled=true` was set explicitly. Either restore the +default Fiber mode for that pool or stop setting the property, which lets the +instance start with resource groups off. + +**The feature is off although the default is on.** Check the log at startup for +an error naming a worker pool, and check `SHOW PARAMETERS` for the value that +took effect. A legacy SQL pool turns the feature off when the property is left +unset. + +## Limitations + +- Only query statements are managed. See + [what is managed](/docs/concepts/resource-groups/#what-is-managed). +- Memory accounting covers tracked native query memory, not JVM heap, resident + set size or memory-mapped table pages. +- CPU control is cooperative, so shares hold over a short window rather than + instantaneously, and a query that cannot reach a cooperative checkpoint holds + its worker until it does. +- Principal mapping changes affect new queries. Group budgets change online; + dropping a group retains its runtime state for existing queries until they + finish. + +## See also + +- [Resource groups concept](/docs/concepts/resource-groups/) +- [Resource groups configuration](/docs/configuration/resource-groups/) +- [Role-based access control](/docs/security/rbac/) +- [Logging and metrics](/docs/operations/logging-metrics/) diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index d2156a88f..22c7a4003 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -61,9 +61,9 @@ SELECT current_data_id(); ## current database, schema, or user -`current_database()`, `current_schema()`, `current_user()`, and -`session_user()` are standard SQL functions that return information about the -current database, schema, and user. +`current_database()`, `current_schema()`, `current_user()`, and `session_user()` +are standard SQL functions that return information about the current database, +schema, and user. ```questdb-sql -- Get the current database @@ -86,6 +86,28 @@ statement without any arguments. and are interchangeable in QuestDB. Both report the user that authenticated on the current connection, whichever protocol it arrived on. +## current_resource_group + +_QuestDB Enterprise only._ + +Returns the resource group assigned to the calling query. Ordinary users can use +this function to check their own assignment; administrator rights are not +required. See [resource groups](/docs/concepts/resource-groups/) for mapping +precedence and the scope of managed execution. + +**Arguments:** none. + +**Return value:** `STRING`. Returns `NULL` when the execution is unmanaged, +including when resource groups are disabled or a replica's catalog is not ready. +A managed query without a principal mapping returns `DEFAULT`. + +```questdb-sql +SELECT current_resource_group(); +``` + +The result follows the query's acquired group, including across suspended cursor +pages. A subsequent mapping change affects the next query. + ## flush_query_cache() `flush_query_cache' invalidates cached query execution plans. @@ -314,7 +336,6 @@ materialized_views(); | trades_OHLC_15m | immediate | trades | 2025-05-30T16:40:37.562421Z | 2025-05-30T16:40:37.568800Z | SELECT timestamp, symbol, first(price) AS open, max(price) as high, min(price) as low, last(price) AS close, sum(amount) AS volume FROM trades SAMPLE BY 15m | trades_OHLC_15m~27 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null | | trades_latest_1d | immediate | trades | 2025-05-30T16:40:37.554274Z | 2025-05-30T16:40:37.562049Z | SELECT timestamp, symbol, side, last(price) AS price, last(amount) AS amount, last(timestamp) as latest FROM trades SAMPLE BY 1d | trades_latest_1d~28 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null | - ## memory_metrics **Arguments:** @@ -382,10 +403,10 @@ SELECT node_role(); :::warning `node_role()` cannot be used in a materialized view or a live view. Avoid it in -`UPDATE` on a WAL table as well: the statement is re-executed on every node of -a replicated cluster and each node evaluates its own role, so the primary and -its replicas would write different values. Tagging rows on `INSERT` is safe, -because inserted rows replicate as data. +`UPDATE` on a WAL table as well: the statement is re-executed on every node of a +replicated cluster and each node evaluates its own role, so the primary and its +replicas would write different values. Tagging rows on `INSERT` is safe, because +inserted rows replicate as data. ::: @@ -410,18 +431,33 @@ Returns metadata on running SQL queries, including columns such as: - state_change - timestamp of latest query state change, such as a cancellation - state - state of running query, can be `active` or `cancelled` - query - text of sql query +- is_wal - whether the query runs as part of WAL apply +- memory_used, memory_limit - tracked native memory the query holds and its + ceiling, `NULL` when no tracker is bound; `memory_limit` is also `NULL` when + the query has no ceiling +- resource_group - the [resource group](/docs/concepts/resource-groups/) the + query was admitted to in QuestDB Enterprise, `NULL` when resource groups do + not manage the execution **Examples:** ```questdb-sql -SELECT * FROM query_activity(); +SELECT query_id, worker_id, worker_pool, username, query_start, state_change, state, query +FROM query_activity(); ``` | query_id | worker_id | worker_pool | username | query_start | state_change | state | query | | -------- | --------- | ----------- | -------- | --------------------------- | --------------------------- | ------ | --------------------------------------------------------- | -| 62179 | 5 | shared | bob | 2024-01-09T10:03:05.557397Z | 2024-01-09T10:03:05.557397 | active | select \* from query_activity() | +| 62179 | 5 | shared | bob | 2024-01-09T10:03:05.557397Z | 2024-01-09T10:03:05.557397Z | active | SELECT count() FROM trades | | 57777 | 6 | shared | bob | 2024-01-09T08:58:55.988017Z | 2024-01-09T08:58:55.988017Z | active | SELECT symbol,approx_percentile(price, 50, 2) from trades | +To inspect query memory and resource group assignment in QuestDB Enterprise: + +```questdb-sql +SELECT query_id, username, resource_group, memory_used, memory_limit +FROM query_activity(); +``` + ## reader_pool **Arguments:** @@ -470,15 +506,94 @@ Edit `server.conf` and run `reload_config`: SELECT reload_config(); ``` +## resource_group_mappings + +_QuestDB Enterprise only. Requires administrator rights._ + +Returns the principal mappings in the resource group catalog. Definitions remain +available when resource group enforcement is disabled. + +**Arguments:** none. + +**Return value:** a table with these columns: + +| Column | Type | Description | +| ---------------------- | --------- | -------------------------------------------------------------------- | +| `principal_type` | `VARCHAR` | `USER`, `GROUP` or `SERVICE_ACCOUNT` | +| `principal_name` | `VARCHAR` | ACL principal name | +| `principal_generation` | `LONG` | Distinguishes a principal from a later recreation of the same name | +| `resource_group_id` | `LONG` | System-assigned identifier of the mapped resource group | +| `resource_group` | `VARCHAR` | Group name | +| `mapping_priority` | `INT` | Priority for ACL group mappings; defaults to `0` | +| `mapping_revision` | `LONG` | Revision used to break equal-priority ties; the higher revision wins | + +```questdb-sql +SELECT principal_type, principal_name, resource_group, mapping_priority +FROM resource_group_mappings() +ORDER BY principal_type, principal_name; +``` + +This lists mappings rather than expanding inherited assignments into one row per +user. Use `current_resource_group()` from a user's own session to confirm the +resolved assignment. + +## resource_groups + +_QuestDB Enterprise only. Requires administrator rights._ + +Returns one row per current catalog group, including `DEFAULT`, with resolved +policies and live counters. Group definitions remain visible when enforcement is +disabled; their runtime counters are zero. + +**Arguments:** none. + +**Return value:** a table with these columns: + +| Column | Type | Description | +| -------------------------- | --------- | ------------------------------------------------------------------------------------------------------- | +| `name` | `VARCHAR` | Group name | +| `memory_limit_bytes` | `LONG` | Effective group ceiling in bytes, capped by the process budget when enabled; `0` means no group ceiling | +| `max_active_queries` | `INT` | Concurrent admission limit; `2147483647` represents unlimited | +| `max_queued_queries` | `INT` | Queue capacity; `2147483647` represents unlimited, and `0` disables queueing | +| `queue_timeout_millis` | `LONG` | Effective admission timeout in milliseconds | +| `cpu_weight` | `INT` | Relative scheduling weight | +| `cpu_max_percent` | `DOUBLE` | CPU percentage cap; `NULL` when uncapped | +| `active_queries` | `LONG` | Queries currently holding admission slots | +| `queued_queries` | `LONG` | Queries waiting for admission | +| `oldest_queue_wait_millis` | `LONG` | Age of the oldest admission waiter in milliseconds; `0` when none | +| `memory_used_bytes` | `LONG` | Published tracked native query memory in bytes | +| `cpu_nanos_total` | `LONG` | CPU nanoseconds measured by managed scheduling | +| `cpu_wait_nanos_total` | `LONG` | Cumulative query waiting time for CPU, in nanoseconds | +| `admission_rejections` | `LONG` | Cumulative queue-full rejections | +| `admission_timeouts` | `LONG` | Cumulative admission timeouts | + +```questdb-sql +SELECT name, memory_limit_bytes, memory_used_bytes, active_queries, queued_queries +FROM resource_groups() +ORDER BY name; +``` + +Counters describe the current runtime and reset on restart or group recreation. +Worker-local memory deltas can be temporarily unpublished. The single uncapped +group dispatch path does not sample CPU, so `cpu_nanos_total` does not cover all +query CPU use. Dropped groups disappear from this table while their existing +queries finish using retained state. + +For the corresponding +[Prometheus metrics](/docs/operations/logging-metrics/#resource-group-metrics), +an uncapped CPU limit is represented by `-1`, whereas SQL returns `NULL`. An +unlimited group memory ceiling is `0` in both interfaces; it does not remove +principal-specific, instance-default single-query or process memory limits. + ## sleep() -Pauses the query for the given number of seconds, then returns the timestamp -at which it resumed. Intended for testing and demonstration, for example to -hold a query open while inspecting -[`query_activity()`](#query_activity) from another session. +Pauses the query for the given number of seconds, then returns the timestamp at +which it resumed. Intended for testing and demonstration, for example to hold a +query open while inspecting [`query_activity()`](#query_activity) from another +session. -`sleep()` does not hold a worker thread while it waits, so many concurrent -calls can be parked at once without exhausting the shared worker pool. +`sleep()` does not hold a worker thread while it waits, so many concurrent calls +can be parked at once without exhausting the shared worker pool. **Arguments:** @@ -502,8 +617,8 @@ SELECT * FROM sleep(1); :::note -Storage policies — and the `storage_policies` view — are available in -**QuestDB Enterprise** only. +Storage policies — and the `storage_policies` view — are available in **QuestDB +Enterprise** only. ::: @@ -532,9 +647,9 @@ SELECT * FROM storage_policies; - TTL values are rendered in two units: `h` for hours and `m` for **months**. Hour-, day-, and week-based durations are stored as hours (e.g. `3 DAYS` → `72h`, `1 WEEK` → `168h`). Month- and year-based durations are stored as - months (e.g. `1 MONTH` → `1m`, `1 YEAR` → `12m`). Despite the visual - collision with "minute", `m` in this view is **months**; QuestDB's duration - shorthand has no unit for minutes. + months (e.g. `1 MONTH` → `1m`, `1 YEAR` → `12m`). Despite the visual collision + with "minute", `m` in this view is **months**; QuestDB's duration shorthand + has no unit for minutes. - An unset stage renders as `0h`, not blank. **Example:** @@ -557,11 +672,15 @@ stage set and has been temporarily disabled. Every unset stage renders as `0h`. :::note -[Cold storage](/docs/concepts/cold-storage/) and the `table_cold_partitions()` function are available in **QuestDB Enterprise** only. +[Cold storage](/docs/concepts/cold-storage/) and the `table_cold_partitions()` +function are available in **QuestDB Enterprise** only. ::: -`table_cold_partitions('tableName')` returns one row per partition in the table's remote manifest, with the state of its object in the store. Use it to follow a partition through upload and sealing, and to find partitions that are not progressing. +`table_cold_partitions('tableName')` returns one row per partition in the +table's remote manifest, with the state of its object in the store. Use it to +follow a partition through upload and sealing, and to find partitions that are +not progressing. **Arguments:** @@ -609,9 +728,16 @@ WHERE state = 'pending'; **Notes:** -- The cold storage manager answers from its own in-memory view. A refresher answers from its mirrored copy, which it updates when the catalog generation changes, so the two can differ briefly. -- While an instance is transitioning between the manager and refresher roles, the function returns zero rows rather than blocking. Check the live role with [`SWITCH COLD STORAGE STATUS`](/docs/query/sql/switch-cold-storage-role/). -- The function reflects the remote manifest, not local partition state. Use [`SHOW PARTITIONS`](/docs/query/sql/show/#show-partitions) or [`table_partitions()`](#table_partitions) to see whether a partition is actually being served remotely. +- The cold storage manager answers from its own in-memory view. A refresher + answers from its mirrored copy, which it updates when the catalog generation + changes, so the two can differ briefly. +- While an instance is transitioning between the manager and refresher roles, + the function returns zero rows rather than blocking. Check the live role with + [`SWITCH COLD STORAGE STATUS`](/docs/query/sql/switch-cold-storage-role/). +- The function reflects the remote manifest, not local partition state. Use + [`SHOW PARTITIONS`](/docs/query/sql/show/#show-partitions) or + [`table_partitions()`](#table_partitions) to see whether a partition is + actually being served remotely. ## table_columns @@ -634,14 +760,14 @@ Returns a `table` with the following columns: - `symbolCached` - whether this `symbol` column is cached - `symbolCapacity` - how many distinct values this column of `symbol` type is expected to have -- `symbolTableSize` - current number of distinct values stored in this - `symbol` column's table +- `symbolTableSize` - current number of distinct values stored in this `symbol` + column's table - `designated` - if this is set as the designated timestamp column for this table - `upsertKey` - if this column is a part of UPSERT KEYS list for table [deduplication](/docs/concepts/deduplication) -- `indexType` - the [index type](/docs/concepts/deep-dive/indexes/) - (`POSTING`, `POSTING DELTA`, `POSTING EF`, `BITMAP`, or empty) +- `indexType` - the [index type](/docs/concepts/deep-dive/indexes/) (`POSTING`, + `POSTING DELTA`, `POSTING EF`, `BITMAP`, or empty) - `indexInclude` - comma-separated names of columns included in a [posting index's](/docs/concepts/deep-dive/posting-index/) covering sidecar @@ -719,16 +845,16 @@ Returns a table with the following columns: partition will contain the `.detached` extension) - `attachable` - _BOOLEAN_, true if the partition is detached and can be attached (`name` of the partition will contain the `.attachable` extension) -- `hasParquetGenerated` - _BOOLEAN_, true if a Parquet copy of the partition - has been generated. Set by either +- `hasParquetGenerated` - _BOOLEAN_, true if a Parquet copy of the partition has + been generated. Set by either [manual Parquet conversion](/docs/concepts/parquet/#in-place-conversion) (`ALTER TABLE ... CONVERT PARTITION TO PARQUET`) or by a [storage policy](/docs/concepts/storage-policy/)'s `TO PARQUET` stage (Enterprise) - `isParquet` - _BOOLEAN_, true if the partition is stored in Parquet format: - the native files have been removed and reads are served from the Parquet - file. Set the same way as `hasParquetGenerated`: either manually or by a - storage policy's `TO PARQUET` stage + the native files have been removed and reads are served from the Parquet file. + Set the same way as `hasParquetGenerated`: either manually or by a storage + policy's `TO PARQUET` stage - `parquetFileSize` - _LONG_, size in bytes of the partition's `data.parquet` file when `hasParquetGenerated` or `isParquet` is true; `-1` otherwise - `seqTxn` - _LONG_, WAL transaction version the partition was last written at @@ -889,7 +1015,7 @@ Returns a `table` with the following columns: ::: -### Table metrics (table_* prefix) +### Table metrics (table\_\* prefix) | Column | Type | Description | | ----------------------------- | --------- | ----------------------------------------------------------------------------------------- | @@ -912,16 +1038,18 @@ Returns a `table` with the following columns: | `table_merge_rate_p99` | LONG | Throughput that 99% of jobs **exceeded** (slowest 1%) | | `table_merge_rate_max` | LONG | Maximum throughput in rows/second | -Write amplification measures O3 (out-of-order) merge overhead as `physicalRowsWritten / logicalRows`. -A ratio of `1.0` means no amplification. Higher values indicate O3 merge overhead. +Write amplification measures O3 (out-of-order) merge overhead as +`physicalRowsWritten / logicalRows`. A ratio of `1.0` means no amplification. +Higher values indicate O3 merge overhead. :::note -Merge rate P99 shows the *lowest* throughput (worst performance), not the highest. +Merge rate P99 shows the _lowest_ throughput (worst performance), not the +highest. ::: -### WAL metrics (wal_* prefix) +### WAL metrics (wal\_\* prefix) | Column | Type | Description | | --------------------------------- | --------- | ------------------------------------------------------------- | @@ -935,9 +1063,10 @@ Merge rate P99 shows the *lowest* throughput (worst performance), not the highes | `wal_tx_size_p99` | LONG | 99th percentile transaction size in rows | | `wal_tx_size_max` | LONG | Maximum transaction size in rows | -### Replica metrics (replica_* prefix) +### Replica metrics (replica\_\* prefix) -These columns are populated on **replicas only** via replication download tracking: +These columns are populated on **replicas only** via replication download +tracking: | Column | Type | Description | | ------------------------ | ------- | ------------------------------------------------------------------------ | @@ -954,17 +1083,31 @@ On primary instances, these columns will be `0` or `false`. These values are approximations, not precise real-time metrics: -- **Null when not tracked**: Values are `null` for tables not written to since server start, or evicted from the tracker -- **Writer stats updated on pool return**: `table_row_count`, `table_last_write_timestamp`, `table_txn` are captured when TableWriter returns to the pool, not on every commit. A writer held for a long time won't update these columns until released. +- **Null when not tracked**: Values are `null` for tables not written to since + server start, or evicted from the tracker +- **Writer stats updated on pool return**: `table_row_count`, + `table_last_write_timestamp`, `table_txn` are captured when TableWriter + returns to the pool, not on every commit. A writer held for a long time won't + update these columns until released. - **WAL stats updated in real-time**: - - On WAL commit: `wal_pending_row_count` (incremented), `wal_txn`, `wal_max_timestamp`, `wal_tx_size_*` histogram - - On WAL apply: `wal_pending_row_count` (decremented), `wal_dedup_row_count_since_start`, `table_min_timestamp`, `table_max_timestamp`, `table_write_amp_*`, `table_merge_rate_*` -- **LRU eviction**: Tracker maintains bounded memory (default 1000 tables). Least recently written tables are evicted when capacity is exceeded -- **Startup hydration**: Values are hydrated from table metadata (`TxReader`) on startup, but diverge as writes occur - -**Non-WAL tables**: `wal_txn`, `wal_max_timestamp`, `wal_pending_row_count`, `wal_dedup_row_count_since_start`, `table_min_timestamp`, `table_max_timestamp`, `table_memory_pressure_level`, and histogram columns are `null` or `0`. - -**WAL tables**: All columns populated when tracked. `wal_max_timestamp` reflects the max data timestamp from the WAL transaction, not wall-clock time. `table_min_timestamp` and `table_max_timestamp` reflect the actual data range in the table after WAL merge. + - On WAL commit: `wal_pending_row_count` (incremented), `wal_txn`, + `wal_max_timestamp`, `wal_tx_size_*` histogram + - On WAL apply: `wal_pending_row_count` (decremented), + `wal_dedup_row_count_since_start`, `table_min_timestamp`, + `table_max_timestamp`, `table_write_amp_*`, `table_merge_rate_*` +- **LRU eviction**: Tracker maintains bounded memory (default 1000 tables). + Least recently written tables are evicted when capacity is exceeded +- **Startup hydration**: Values are hydrated from table metadata (`TxReader`) on + startup, but diverge as writes occur + +**Non-WAL tables**: `wal_txn`, `wal_max_timestamp`, `wal_pending_row_count`, +`wal_dedup_row_count_since_start`, `table_min_timestamp`, `table_max_timestamp`, +`table_memory_pressure_level`, and histogram columns are `null` or `0`. + +**WAL tables**: All columns populated when tracked. `wal_max_timestamp` reflects +the max data timestamp from the WAL transaction, not wall-clock time. +`table_min_timestamp` and `table_max_timestamp` reflect the actual data range in +the table after WAL merge. ### Configuration @@ -1206,9 +1349,8 @@ concurrent waiters is not bounded by the shared worker pool. **Arguments:** -- `tableName` (`string`): name of the table to wait for. Must be a constant, - not a column reference. On a non-WAL table the call returns `true` - immediately. +- `tableName` (`string`): name of the table to wait for. Must be a constant, not + a column reference. On a non-WAL table the call returns `true` immediately. - `seqTxn` (optional, `long`): the sequencer transaction to wait for. When omitted, the call captures the table's current `seqTxn` when it starts and waits for that, which is what you want after your own write. @@ -1218,8 +1360,8 @@ concurrent waiters is not bounded by the shared worker pool. Returns `boolean`. `true` once the writer has caught up. Throws if the table is dropped while the call is waiting, and if the table -becomes [suspended](/docs/query/sql/alter-table-resume-wal/), since a -suspended table would otherwise never catch up. +becomes [suspended](/docs/query/sql/alter-table-resume-wal/), since a suspended +table would otherwise never catch up. **Examples:** @@ -1243,11 +1385,10 @@ SELECT wait_wal_table('trades', 42); :::note -For monitoring and observability, use [`tables()`](#tables) instead. -`tables()` provides all the same information plus additional metrics -(pending rows, memory pressure, deduplication stats, throughput histograms), -and is fully in-memory. `wal_tables()` reads from disk and is less suitable -for frequent polling. +For monitoring and observability, use [`tables()`](#tables) instead. `tables()` +provides all the same information plus additional metrics (pending rows, memory +pressure, deduplication stats, throughput histograms), and is fully in-memory. +`wal_tables()` reads from disk and is less suitable for frequent polling. ::: @@ -1265,11 +1406,13 @@ Returns a `table` including the following information: - `name` - table or materialized view name - `suspended` - suspended status flag -- `writerTxn` - the last committed transaction in TableWriter (equivalent to `table_txn` in `tables()`) +- `writerTxn` - the last committed transaction in TableWriter (equivalent to + `table_txn` in `tables()`) - `writerLagTxnCount` - the number of transactions that are kept invisible when writing to the table; these transactions will be eventually moved to the table data and become visible for readers (equivalent to `wal_txn - table_txn`) -- `sequencerTxn` - the last committed transaction in the sequencer (equivalent to `wal_txn` in `tables()`) +- `sequencerTxn` - the last committed transaction in the sequencer (equivalent + to `wal_txn` in `tables()`) **Examples:** diff --git a/documentation/sidebars.js b/documentation/sidebars.js index 2c4c83c2a..6ac7b5e11 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -635,6 +635,11 @@ module.exports = { type: "doc", label: "Cold Storage", }, + { + id: "concepts/resource-groups", + type: "doc", + label: "Resource Groups", + }, "concepts/write-ahead-log", ], }, @@ -698,6 +703,11 @@ module.exports = { "configuration/postgres-wire-protocol", "configuration/qwp", "configuration/database-replication", + { + id: "configuration/resource-groups", + type: "doc", + label: "Resource groups", + }, "configuration/shared-workers", "configuration/storage-policy", "configuration/telemetry", @@ -787,6 +797,11 @@ module.exports = { type: "doc", label: "Cold storage", }, + { + id: "operations/resource-groups", + type: "doc", + label: "Resource groups", + }, "operations/logging-metrics", "operations/monitoring-alerting", "operations/data-retention",