diff --git a/docs/how-to/gitops.md b/docs/how-to/gitops.md index feca7dc9..544bbed2 100644 --- a/docs/how-to/gitops.md +++ b/docs/how-to/gitops.md @@ -231,9 +231,9 @@ Add this entry (`config/gitops/flux-healthcheck-coderaiprovider.yaml`) to `spec. ## Health rules for CoderWorkspacePool and CoderWorkspaceClaim -Argo CD and Flux also need health rules to wait for a [workspace pool](workspace-pools.md) and its claims. Without them, Argo CD has no health check for the kinds, and Flux reports a new pool or claim as current before the controller writes its first status. The controller writes the conditions `Ready`, `Reconciling` and `Stalled` of a pool or a claim together, with one reason and message, and with the object's `metadata.generation` as the condition's `observedGeneration`. The rules ignore a condition whose `observedGeneration` is older. +Argo CD and Flux also need health rules to wait for a [workspace pool](workspace-pools.md) and its claims. Without them, Argo CD has no health check for the kinds, and Flux reports a new pool or claim as current before the controller writes its first status. The controller writes the conditions `Ready`, `Reconciling` and `Stalled` of a pool or a claim together, with one reason and message, and with the object's `metadata.generation` as the condition's `observedGeneration`. The Argo CD rules ignore each condition whose `observedGeneration` is older. The Flux rules wait for a `Ready` condition of the current generation, and then read the conditions without that filter. The controller gives all three conditions one generation, so both rules agree on what it writes. -A pool also has `status.observedGeneration`, and both pool rules wait until it equals `metadata.generation`. Every `/scale` write raises the generation, so after `kubectl scale`, or a resize by an HPA or KEDA, the pool shows `Progressing` until the controller has counted its members for the new `spec.replicas`. +A pool also has `status.observedGeneration`, and both pool rules wait until it equals `metadata.generation`. A change of `spec.replicas` through `/scale` raises the generation, so after `kubectl scale`, or a resize by an HPA or KEDA, the pool shows `Progressing` until the controller has counted its members for the new `spec.replicas`. | Pool conditions | Argo CD | Flux | | --- | --- | --- | diff --git a/docs/how-to/troubleshooting.md b/docs/how-to/troubleshooting.md index 8751690d..a9c4fa6d 100644 --- a/docs/how-to/troubleshooting.md +++ b/docs/how-to/troubleshooting.md @@ -316,7 +316,7 @@ kubectl get coderworkspacepools,coderworkspacepoolmembers,coderworkspaceclaims - kubectl get coderworkspacepoolmember -n -o jsonpath='{.status.reason}: {.status.message}{"\n"}' ``` -A wait ends by itself when its cause goes away. A failure is final for a member: its phase becomes `Failed`. The pool deletes a failed unclaimed member and creates a new one. A claimed member that fails stays, and its claim stays `Bound` and not ready until you delete the claim. If its workspace is gone, the claim becomes `Lost` instead. A failure of the pool sets `Stalled=True`. The pool reads Coder again every 60 s, also while it is stalled, so a fix in Coder ends a stall without a spec change. A claim often does not show why its pool cannot create workspaces. If a claim waits for a long time, read the reason of its pool. +A wait ends by itself when its cause goes away. A failure is final for a member: its phase becomes `Failed`. The pool deletes a failed unclaimed member and creates a new one. A claimed member that fails stays, and its claim stays `Bound` and not ready until you delete the claim. If its workspace is gone, the claim becomes `Lost` instead. A failure of the pool sets `Stalled=True`. A stalled pool reads Coder again every 60 s, so a fix in Coder ends the stall without a spec change. A claim often does not show why its pool cannot create workspaces. If a claim waits for a long time, read the reason of its pool. ### Reasons of a pool and of a member before its create request @@ -366,7 +366,7 @@ The pool checks its inputs in Coder, and each member checks them again right bef | `WorkspaceReady` | ready | Every agent is ready. A claim can take the workspace. | None. | | `Used` | wait | The workspace's `last_used_at` moved after it became ready, so someone used it without a claim, or Coder counted something else as a use. Opening the workspace's page in the Coder dashboard counts ([#444](https://github.com/coder/coder-k8s/issues/444)). The member gets `Used=True`, and the pool releases it. | See [Cleanup](workspace-pools.md#cleanup). | | `AgentNotReady` | wait | An agent of a ready workspace disconnected or is no longer ready. No claim takes the workspace meanwhile. A failed Coder read keeps this reason, and the 5 min keep counting. | None, unless it becomes `AgentLost`. | -| `AgentLost` | failure | An agent stayed not ready for 5 min. | Read the agent logs in Coder. | +| `AgentLost` | failure | An agent stayed not ready for 5 min. While Coder is unreachable, the member waits. A failed read keeps `AgentNotReady` and its 5 min. `AgentLost` comes with the first successful read after the 5 min, if the agent is still not ready. | Read the agent logs in Coder. | | `ChangedExternally` | failure | Someone else started a build of the workspace, for example a stop, an update, or Coder's autostop. | None. The pool replaces an unclaimed member. | | `Dormant` | failure | Coder marked the workspace dormant, or scheduled its deletion. | Check the template's dormancy settings. | | `MaxDeadline` | wait | The workspace's max deadline allows no extension of its autostop deadline. Coder stops the workspace at the max deadline, and the pool replaces it. | None. | @@ -412,7 +412,7 @@ After `ShareRevoked` or `ShareRemoved`, Coder no longer lists the user on the wo ### The pool owner is dormant -The pool waits with `OwnerNotEligible`, and the message names the status `dormant`. Coder creates users through its API as `dormant`, and every 15 minutes it marks each active user dormant whose last login or API use is older than 90 days. An activation does not count as a use. Activate the owner in Coder, then log in as it once, for example with `POST /api/v2/users/login`. After that, use the owner's credentials at least once every 90 days. The pool deletes its unclaimed workspaces while the owner is not eligible, and builds new ones once the owner is active. +The pool waits with `OwnerNotEligible`, and the message names the status `dormant`. Coder creates users through its API as `dormant`, unless the create request sets another status. Every 15 minutes, Coder marks each active user dormant whose last login or API use is older than 90 days. An activation does not count as a use. Activate the owner in Coder, then send one request with the owner's credentials, for example a password login with `POST /api/v2/users/login`. A service account cannot log in: send a request such as `GET /api/v2/users/me` with its API token instead. After that, use the owner's credentials at least once every 90 days. The pool deletes its unclaimed workspaces while the owner is not eligible, and builds new ones once the owner is active. ### A claim stays `Pending` and names a member diff --git a/docs/how-to/workspace-pools.md b/docs/how-to/workspace-pools.md index 68f109d0..e6c0fd15 100644 --- a/docs/how-to/workspace-pools.md +++ b/docs/how-to/workspace-pools.md @@ -14,7 +14,7 @@ You need: Every start build hands the owner's token to the template's Terraform. So the owner must pass the same rules as the tester of a template test (see [Test a template version](test-templates.md#1-create-the-tester-user)): status `active`, a service account or a password user, no site role other than `member`, and only `organization-member` or `organization-workspace-access` in each organization. Coder's prebuilds system user is refused. Without a Premium license, Coder has no service accounts, so use a password user that no person logs in as. -Coder creates users through its API as `dormant`. Activate the owner, then log in as it once, for example with `POST /api/v2/users/login`. Coder checks every 15 minutes and marks each active user dormant whose last login or API use is older than 90 days. An activation does not count as a use, so a new owner that never logged in turns dormant at the next check. A dormant owner stops the pool with `OwnerNotEligible`, and the pool deletes its unclaimed workspaces. Use the owner's credentials at least once every 90 days. A claim consumer that connects with the owner's token does that. +Coder creates users through its API as `dormant`, unless the create request sets another status. Activate the owner, then send one request with the owner's credentials: a password login with `POST /api/v2/users/login`, or, for a service account, which cannot log in, a request such as `GET /api/v2/users/me` with its API token. Coder checks every 15 minutes and marks each active user dormant whose last login or API use is older than 90 days. An activation does not count as a use, so a new owner that never sent a request turns dormant at the next check. A dormant owner stops the pool with `OwnerNotEligible`, and the pool deletes its unclaimed workspaces. Use the owner's credentials at least once every 90 days. A claim consumer that connects with the owner's token does that. ## 2. Allow the owner on the control plane @@ -60,17 +60,11 @@ spec: kubectl -n coder get coderworkspacepools,coderworkspacepoolmembers ``` -Each member owns one workspace named `kwp-` plus 28 hex characters of the member's UID. Do not rename these workspaces, and do not use an unclaimed one directly. The pool releases a workspace that someone used without a claim, and never deletes it on its own. Opening an unclaimed workspace's page in the Coder dashboard counts as use too: the page asks the workspace agent for its containers, and in Coder v2.37.2 that request moves `last_used_at` ([#444](https://github.com/coder/coder-k8s/issues/444)). The workspaces list does not. +Each member owns one workspace named `kwp-` plus 28 hex characters of the member's UID. Do not rename these workspaces, and do not use an unclaimed one directly. The pool releases a workspace that someone used without a claim, and never deletes it on its own. Opening an unclaimed workspace's page in the Coder dashboard counts as use too: the page asks the workspace agent for its containers, and in Coder v2.37.2 that request moves `last_used_at` ([#444](https://github.com/coder/coder-k8s/issues/444)). The workspaces list does not. The pool sees a use only through `last_used_at`, which is best-effort, so it can miss a use (see the warning in [step 4](#4-claim-a-workspace)). An unclaimed workspace stays until a claim takes it, a scale-down removes it, or the pool's inputs change. With `version.active: true`, a new active version of the template is such a change. To also replace workspaces that have been ready for a long time, set `maxUnclaimedAgeSeconds` (at least 3600). The pool replaces outdated workspaces new before old: it builds the replacement first, and the old workspace stays claimable until the replacement is ready. So the replacement needs room under `maxWorkspaces`. -`replicas` has a scale subresource, so `kubectl scale`, HPA, and KEDA can change it: - -```bash -kubectl -n coder scale coderworkspacepool ci-runners --replicas=5 -``` - -The pool's `status.selector` matches no pods, so an HPA must use External or Object metrics with target type `AverageValue`. KEDA's HPAs do that by default, and KEDA also moves a pool between 0 and 1. In the Kind end-to-end test, a KEDA v2.21.0 `cron` ScaledObject with `minReplicaCount: 0` moves a pool from 0 to 2 ready workspaces and back to 0. Its HPA reports `ScalingActive=True`. At `replicas: 0`, a claim still gets a workspace: the pool builds one for it. +`replicas` has a scale subresource, so `kubectl scale`, an HPA, and KEDA can change it. See [Autoscale a pool](#autoscale-a-pool). ## 4. Claim a workspace @@ -89,7 +83,7 @@ A claim with a ready workspace in its pool is `Bound` and `Ready` at once. In th To delete claimed workspaces that nobody uses, set `claims.idleTimeoutSeconds` on the pool (at least 600). Without it, the operator never deletes a claimed workspace on its own. !!! warning "Idle reaping deletes the workspace and its data" - The pool deletes a claimed workspace once both the bind and the workspace's `last_used_at` in Coder are older than `claims.idleTimeoutSeconds`. A stopped workspace counts as idle. The claim then becomes `Lost` with reason `IdleTimeout`, and it never binds again. Right before the delete build, the operator reads the workspace once more. If that read shows a use after the pool's check, or no `last_used_at`, the operator keeps the workspace in Coder and only releases its member. The claim message covers both outcomes: the pool deletes the workspace unless its last check sees a use, and a kept workspace stays in Coder under its `kwp-` name, owned by the pool owner. The message names both, so `coder ssh /` still reaches a kept workspace. To delete kept workspaces later, see [Cleanup](#cleanup). Coder writes `last_used_at` about once a minute, so a session that starts less than a minute before the delete build can still be lost. + The pool deletes a claimed workspace once both the bind and the workspace's `last_used_at` in Coder are older than `claims.idleTimeoutSeconds`. A stopped workspace counts as idle. The claim then becomes `Lost` with reason `IdleTimeout`, and it never binds again. Right before the delete build, the operator reads the workspace once more. If that read shows a use after the pool's check, or no `last_used_at`, the operator keeps the workspace in Coder and only releases its member. The claim message covers both outcomes: the pool deletes the workspace unless its last check sees a use, and a kept workspace stays in Coder under its `kwp-` name, owned by the pool owner. The message names both, so `coder ssh /` still reaches a kept workspace. To delete kept workspaces later, see [Cleanup](#cleanup). Coder writes `last_used_at` about once a minute, so a session that starts less than a minute before the delete build can still be lost. `last_used_at` is also best-effort: Coder drops a minute's updates when their database write fails. So idle reaping can delete a workspace whose last use Coder failed to record. ## 5. Grant access @@ -215,13 +209,101 @@ kubectl -n coder wait --for=condition=Ready coderworkspaceclaim/alice-dev The operator stores the user's ID before it shares, so a renamed user keeps the share and a new user with the old name never gets it. It checks the share every minute, and at once when the control plane changes. The claim is `Ready` only while the workspace's sharing list has the user and the policy above still allows it. If a person changes the user's role in Coder, for example to `admin`, the operator keeps that role. When the admin withdraws what allowed the share, for example removes the user from every listed group, empties `shareGroupIDs`, removes the owner from `owners`, or suspends the user, the operator removes the share, and the claim is `Stalled` with `ShareRevoked`. Coder itself drops shares when an admin tightens the organization's sharing setting, and a person can remove one in the Coder UI. The claim is then `Stalled` with `ShareRemoved`. In both cases the user keeps access until the workspace restarts or stops: Coder applies the removal of a share only then ([Shared workspaces](https://coder.com/docs/user-guides/shared-workspaces)), and the operator does not restart or stop the workspace. To end the access at once, restart or stop the workspace in Coder, or delete the claim, which deletes the workspace. The operator never shares a second time, so create a new claim instead. Deleting the claim deletes the workspace, which ends the share. !!! warning "Share mode trusts every claimant with the pool owner" - Every claimant can act as the pool owner, and so reach the workspaces of other claimants, if the template exposes `data.coder_workspace_owner..session_token`, if the owner has external-auth links, or through the owner's Git SSH key. Use a template that never exposes the owner's token and an owner without external-auth links, and treat the owner's Git SSH key as shared by every claimant. Anyone who can create pools on a control plane can act as every owner that the control plane lists. `maxWorkspaces` counts per control plane, so two `CoderControlPlane` objects on one Coder deployment can each allow the full cap for one owner. + Every claimant can act as the pool owner, and so reach the workspaces of other claimants, if the template exposes `data.coder_workspace_owner..session_token`, if the owner has external-auth links, or through the owner's Git SSH key. Use a template that never exposes the owner's token and an owner without external-auth links, and treat the owner's Git SSH key as shared by every claimant. + +!!! warning "A Share workspace that idle reaping keeps stays shared" + When [idle reaping](#4-claim-a-workspace) keeps a claimed workspace because of a late use, the pool releases its member, and the workspace stays in Coder. The workspace of a Share claim then keeps the shared user's `use` role, and the operator no longer checks the share policy for it ([#451](https://github.com/coder/coder-k8s/issues/451)). If the admin later withdraws the share, for example removes the user from every listed group, nothing removes it. Remove the user from the workspace's sharing settings in Coder: that keeps the work. Deleting the workspace in Coder also ends the share, but it deletes the kept work too. + +## Autoscale a pool + +`spec.replicas` has a scale subresource. Every `/scale` write runs the CRD schema, which refuses a value above 100: `kubectl scale` then returns a validation error, and an update by an HPA or KEDA fails, so the pool keeps its old size. Set `maxReplicas` of an HPA and `maxReplicaCount` of a ScaledObject to 100 or less. To resize a pool by hand: + +```bash +kubectl -n coder scale coderworkspacepool ci-runners --replicas=5 +``` + +At `replicas: 0`, the pool keeps no unclaimed workspace, and a claim still gets one: the pool builds it for the claim, so the claim waits for a full build. The Kind end-to-end test checks this. + +To let only the users who can resize a pool autoscale it, add the admission policy in [Grant access](#5-grant-access). + +### HPA + +The pool's `status.selector`, `coder.com/workspace-pool-uid=`, matches no pods. So an HPA must use an `External` or `Object` metric with the target type `AverageValue`: HPA then computes the replicas from the metric and `status.replicas`, and lists no pods. With a `Value` target, HPA counts pods by the selector and fails with `no pods returned by selector` once the metric moves outside its tolerance. This example keeps one ready workspace for each queued CI job, from a metric that your external metrics adapter serves: + +```yaml +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: ci-runners + namespace: coder +spec: + scaleTargetRef: + apiVersion: coder.com/v1alpha1 + kind: CoderWorkspacePool + name: ci-runners + minReplicas: 1 + maxReplicas: 20 + metrics: + - type: External + external: + metric: + name: ci_jobs_queued # your adapter's metric + target: + type: AverageValue + averageValue: "1" +``` + +Without the alpha feature gate `HPAScaleToZero`, HPA cannot scale to 0: `minReplicas` must be at least 1, and HPA stops scaling a pool at `spec.replicas: 0`. When a pool with at least 1 replica is below `minReplicas`, HPA raises it to `minReplicas` before it reads any metric. The Kind test runs HPA only through KEDA. + +### KEDA + +KEDA moves a pool between 0 and 1 itself, and makes an HPA with an `AverageValue` metric for 1 to `maxReplicaCount`. Before it goes back to 0, KEDA waits `cooldownPeriod`, 300 s by default. In the Kind end-to-end test, KEDA v2.21.0 with a `cron` ScaledObject like this one moved a pool from 0 to 2 ready workspaces and back to 0, and its HPA reported `ScalingActive=True`. The test used a 3-minute window and `cooldownPeriod: 30`. + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: ci-runners-office-hours + namespace: coder +spec: + scaleTargetRef: + apiVersion: coder.com/v1alpha1 + kind: CoderWorkspacePool + name: ci-runners + minReplicaCount: 0 + maxReplicaCount: 2 + triggers: + - type: cron + metadata: + timezone: Etc/UTC + start: 0 8 * * 1-5 # 08:00 on weekdays + end: 0 18 * * 1-5 + desiredReplicas: "2" +``` + +### Autoscalers and GitOps + +An autoscaler writes `spec.replicas`, so a GitOps tool that applies `spec.replicas` from Git sets it back at each sync. Leave `spec.replicas` out of the manifest in Git: the pool then starts at the default of 1, and only the autoscaler changes it. In Argo CD, you can instead ignore the field. Argo CD then still applies it at a sync, unless you also set `RespectIgnoreDifferences=true`: + +```yaml +spec: + ignoreDifferences: + - group: coder.com + kind: CoderWorkspacePool + jsonPointers: + - /spec/replicas + syncPolicy: + syncOptions: + - RespectIgnoreDifferences=true +``` + +To make Argo CD or Flux wait for a pool or a claim, install the [health rules](gitops.md#health-rules-for-coderworkspacepool-and-coderworkspaceclaim). Each change of `spec.replicas` makes the pool `Progressing` until the controller has counted its members for the new value. ## Cost Each unclaimed workspace runs all the time, so a pool of `replicas: N` costs N running workspaces plus the claimed ones. A claimed workspace keeps a normal autostop deadline after the bind, so Coder stops it when nobody uses it. A template with a default TTL of 0 has no deadline: its claimed workspaces run until someone stops or deletes them. For such a template, `claims.idleTimeoutSeconds` is the only automatic limit. -Further limits: `maxInFlight` (default 5) caps the members of a pool that build at the same time, and a pool stops creating with `Stalled` and `MemberFailureLimit` after 5 failed members in a row. To cap the number of pools and claims in a namespace, use a `ResourceQuota` on `count/coderworkspacepools.coder.com` and `count/coderworkspaceclaims.coder.com`. +`maxWorkspaces` caps each owner per control plane, so two `CoderControlPlane` objects on one Coder deployment can each allow the full cap for one owner. Further limits: `maxInFlight` (default 5) caps the members of a pool that build at the same time, and a pool stops creating with `Stalled` and `MemberFailureLimit` after 5 failed members in a row. To cap the number of pools and claims in a namespace, use a `ResourceQuota` on `count/coderworkspacepools.coder.com` and `count/coderworkspaceclaims.coder.com`. ### Coder API requests @@ -229,7 +311,7 @@ The operator sends every pool request with its operator token. Measured in Kind | What | Requests per minute | Requests | | --- | --- | --- | -| A ready member | 2 | `GET /api/v2/workspaces/` and `GET /api/v2/users/me` once a minute | +| A ready member | 2 | `GET /api/v2/workspaces/` and `GET /api/v2/users/me` once a minute ([#443](https://github.com/coder/coder-k8s/issues/443) tracks the second read) | | A shared member of a Share claim | 6 | The two requests above, plus the workspace's sharing list, the owner, the shared user, and the organization's sharing setting once a minute (and each listed group other than `Everyone`) | | A member that builds | 24 | The same two requests every 5 s, until the member is ready | | A pool that follows the active version | 6 | The owner, its organization membership, the organization (twice), the template, and the version, once a minute | @@ -250,9 +332,32 @@ Checks that fall due together wait in line, so with a 1 s delay some checks wait ## Cleanup -- Delete a claim to delete its workspace. -- Delete a pool to delete its unclaimed workspaces. Claimed workspaces stay with their claims, and the claims stay `Bound`. -- A member whose workspace someone used without a claim (`Used=True`) leaves the pool: the pool removes itself as the member's owner, so neither a scale-down nor deleting the pool deletes it. A cluster admin who deletes such a member deletes its workspace in Coder, as deleting a claim does. To keep the work, copy it out of the workspace first. Deleting the namespace keeps a used member's workspace in Coder. Find it as the next item says. A used member that still lists the pool as its controller is not released yet, and deleting it keeps the workspace in Coder. -- When the `CoderControlPlane` is deleted, members release their finalizers without a Coder call. Find leftover workspaces in Coder with the filter `owner:` and delete those whose names start with `kwp-`. From the start of the delete, members no longer check or remove shares, so shared users keep access while Coder runs. To end every share first, delete the Share claims before the control plane. +A member deletes its workspace in Coder when the member is deleted, except in the cases that the table names. Before each delete build of an unclaimed workspace that was ready, the member checks `last_used_at` once more. If someone used the workspace, the member keeps it in Coder and goes away alone. + +| What you do | What happens to the workspaces | +| --- | --- | +| Delete a claim | Its workspace is deleted. A share ends with it. | +| Delete a pool | Its unclaimed workspaces are deleted. Claimed workspaces stay with their claims, and the claims stay `Bound`. | +| Lower `spec.replicas` | The pool keeps the new value plus one unclaimed workspace for each waiting claim that can bind, and deletes the unclaimed workspaces above that: first those without a create request, then those that are not ready (newest first), then ready ones (oldest first). Claimed and used workspaces stay. | +| Change the pool's version, preset, or parameters, or wait for a new active version or for `maxUnclaimedAgeSeconds` | Outdated unclaimed workspaces are deleted, new before old (see [step 3](#3-create-a-pool)). | +| Remove the owner or the template from the control plane, or make the owner ineligible, for example dormant | Every unclaimed workspace is deleted. Claimed workspaces stay. | +| Let idle reaping delete a claimed workspace | It is deleted, unless a late use keeps it (see [step 4](#4-claim-a-workspace)). A kept workspace stays in Coder without a member. | +| Use an unclaimed workspace without a claim | Its member gets `Used=True`, and the pool releases it: neither a scale-down nor deleting the pool deletes it. A cluster admin who deletes such a member deletes its workspace in Coder, as deleting a claim does. To keep the work, copy it out first. A used member that still lists the pool as its controller is not released yet, and deleting it keeps the workspace in Coder. | +| Rename a `kwp-` workspace in Coder | Nothing proves that the workspace is the member's, so the operator never deletes it, and the member keeps its finalizer. See [A pool member does not finish deleting](troubleshooting.md#a-pool-member-does-not-finish-deleting). | +| Delete the namespace | The control plane in the namespace is deleted too. Members that have not deleted their workspaces when its deletion starts release without a Coder call, as in the next row, and released used members always keep their workspaces. To delete the workspaces, delete the claims and pools first, and wait until their members are gone. | +| Delete the `CoderControlPlane` | From the start of its deletion, members no longer call Coder, so they neither check nor remove shares ([#448](https://github.com/coder/coder-k8s/issues/448)). A member that is deleted after that releases its finalizer without a Coder call, and its workspace stays in Coder. To end every share first, delete the Share claims, and wait until their members are gone, before you delete the control plane. | + +To find workspaces that stayed in Coder, filter the Coder workspaces list with `owner:`, and look for names that start with `kwp-` that no member lists in `status.workspaceName`. Delete those that you do not need. + +## Security notes + +- Every start build hands the pool owner's token to the template's Terraform. The control plane admin decides which owners and templates pools can use (step 2), and the owner rules (step 1) refuse privileged users. Anyone who can create pools on a control plane can act as every owner that the control plane lists. +- `spec.parameters` are stored in plain text in the pool and in each member. Never put secrets there. +- The install adds to the built-in `view`, `edit`, and `admin` roles only read access to members. Users need a Role as in [step 5](#5-grant-access) for pools and claims, and no user role can write members. Anyone who can create an HPA can still resize a pool, unless you add the admission policy from step 5. +- `status.message` holds the controller's text and Coder's short error message, never Coder's error details or build logs. It can name Coder users and facts about them, for example in Share mode ([#449](https://github.com/coder/coder-k8s/issues/449)). +- In Share mode, every claimant can act as the pool owner through the template or the owner's credentials. See [step 6](#6-share-a-workspace-with-a-person). +- With TLS on the control plane and without `spec.tls.operatorAccess`, pools send the operator token to Coder over plain HTTP inside the cluster. See [Where the operator token goes](deploy-controller.md#where-the-operator-token-goes). + +## Troubleshooting -With TLS on the control plane and without `spec.tls.operatorAccess`, pools send the operator token to Coder over plain HTTP inside the cluster. See [Where the operator token goes](deploy-controller.md#where-the-operator-token-goes). +For every reason of a pool, a member, and a claim, and how to fix it, see [Troubleshoot workspace pools](troubleshooting.md#troubleshoot-workspace-pools).