Skip to content

🤖 feat: GitOps health rules and troubleshooting for workspace pools (#153 PR 16a) - #453

Merged
ThomasK33 merged 7 commits into
mainfrom
pool-gitops-health-troubleshooting
Nov 1, 2026
Merged

ThomasK33 merged 7 commits into
mainfrom
pool-gitops-health-troubleshooting

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

Summary

Argo CD and Flux now have health rules for CoderWorkspacePool and CoderWorkspaceClaim, and the troubleshooting page now explains every reason that the pool, member, and claim controllers can write, with its cause and fix. This is plan PR 16a of #153. The workspace-pools.md restructure follows in 16b, after 15b lands.

Background

Without health rules, Argo CD has no health check for pools and claims, and Flux reports a new pool or claim as Current before the controller writes any status. So a sync wave that waits for a pool can go on too early. The pool and claim controllers also write more than 70 different reasons, but the docs explained only a few. Plan section 7.4 item 2 defines the rules, and plan row 16 asks for a troubleshooting section that covers every reason.

Implementation

  1. config/gitops/argocd-health-coderworkspace{pool,claim}.lua and flux-healthcheck-coderworkspace{pool,claim}.yaml follow the existing CoderAIProvider files.

    • Pool: Healthy when Ready=True and status.observedGeneration equals metadata.generation, Degraded when Stalled=True, otherwise Progressing. A /scale write raises the generation, so a resized pool is Progressing until the controller has counted its members again.
    • Claim: Healthy when Ready=True, Degraded when Stalled=True or the phase is Lost, otherwise Progressing. A claim has no status.observedGeneration, so the rules read only conditions whose observedGeneration matches the generation.
    • Both rules ignore conditions from an older generation, and they show Deleting the workspace pool or Deleting the workspace claim while Argo CD deletes the object.
  2. config/gitops/health_test.go adds 30 fixtures for the two kinds (12 pool, 18 claim). They cover a missing status, stale generations, waits, stalls, Lost, the Share stalls (ClaimModeMismatch, ShareRefused, ShareRevoked), and deletion. The existing doc-copy test keeps the copies in docs/how-to/gitops.md equal to the files.

  3. docs/how-to/gitops.md gets a section with state tables, install steps, and copies of the rules.

  4. docs/how-to/troubleshooting.md gets "Troubleshoot workspace pools". It has one table per object (a pool and a member before its create request, a pool, a member, and a claim), plus five common cases:

    • a Share claim that waits with ShareUserNotEligible (groups other than Everyone need Premium, so Coder answers 403 without it),
    • a user who keeps access after a share ends, until the workspace restarts or stops (🤖 CoderWorkspacePool Share mode: optionally stop the workspace when a share is revoked #450),
    • a dormant pool owner (log in once, then use the owner at least every 90 days),
    • a claim stuck on a reservation (delete the member that status.memberName names),
    • a member that does not finish deleting because someone renamed its kwp- workspace (OwnershipUnknown). The fix is to delete the workspace in Coder, then remove only the operator's finalizer.

    The tables link the open follow-ups CoderWorkspacePool: a dashboard page view marks an unclaimed member Used #444, 🤖 CoderWorkspacePool Share mode: shares stay while a control plane is being deleted #448, 🤖 CoderWorkspaceClaim Share mode: claim reasons reveal facts about named Coder users #449, and 🤖 CoderWorkspacePool Share mode: optionally stop the workspace when a share is revoked #450 where they apply.

Validation

  • Reason coverage. This script lists every CamelCase string literal in the pool, member, claim, and share code and in the shared Coder helpers they call (all of them are reasons, except the kind name CoderWorkspacePool). It fails if troubleshooting.md does not name one of them. It passes with 73 reasons:

    files=$(ls internal/controller/coderworkspacepool_*.go internal/controller/coderworkspacepoolmember_*.go \
      internal/controller/coder_access.go internal/controller/coder_workspace_steps.go internal/controller/controlplane_access.go | grep -v _test.go)
    for r in $(grep -ohE '"[A-Z][A-Za-z]+"' $files | tr -d '"' | grep -vx CoderWorkspacePool | sort -u); do
      grep -q "\`$r\`" docs/how-to/troubleshooting.md || echo "missing: $r"
    done
  • Mutation check of the rules. I broke each branch of the four rule files by hand: the Lost check, the Stalled branch, the generation filters, the stale observedGeneration check, the deletion message, and the Flux Lost clauses. Each break makes at least one fixture fail. One edit fails no fixture: removing the Reconciling clause of the Flux inProgress expression. That clause cannot change a result, because Flux falls back to InProgress anyway. The existing CoderAIProvider rule behaves the same way.

  • An independent reviewer compared every doc claim with the code before I opened this PR. It found 9 wrong or overstated claims, and I fixed all of them. One example: a claimed member that fails is not replaced. Its claim stays Bound and not ready until someone deletes the claim.

  • Local gates: make verify-vendor, make build, make lint, make test, nix develop -c make docs-check, go test ./config/gitops/..., markdownlint-cli2 "docs/**/*.md", and cspell with the CI versions.

  • Dogfood: I ran make docs-serve and took agent-browser screenshots of the new sections at 390 px and 1440 px. They are in a comment below.

Risks

  • Docs and data files only. No controller code changes.
  • The rules assume what the controllers write today: Ready, Reconciling, and Stalled together, with the object's generation in each condition's observedGeneration. A controller change that breaks that assumption must update these rules and fixtures too.
  • CI does not run Argo CD or Flux. The test runs the Lua with gopher-lua and the CEL with cel-go, as for the existing rules.

Refs #153


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

…derWorkspaceClaim (#153)

Change-Id: Id132668b4604dcd75c7dbcff651c3f25512c0e0f
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…hooting (#153)

Change-Id: I39a4bcca90489c23e5fcee27cc10bd611d5bdea1
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…h rules (#153)

Change-Id: I770d1585d1de1c0ae371a3ab910ec1b4449e8ce7
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I3d0054da6ecf6f6a8ed3336751173f0ac5fdc539
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: Icfe185a6bdc6a77437b376f9512b9c47ed9c3652
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-01T02:12:12.329361Z c998543 Manual request
🔒 Security Review ✅ Completed 2026-11-01T02:04:09.761505Z c57c000 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@ThomasK33

Copy link
Copy Markdown
Member Author

Dogfood for plan row 16: nix develop -c make docs-serve on head c57c000, then agent-browser screenshots of the new sections.

Desktop (1440 px):

Troubleshoot workspace pools, desktop
Reasons of a claim, desktop
Pool and claim health rules in gitops.md, desktop

Mobile (390 px). The reason tables scroll sideways, like the other tables on the site:

Troubleshoot workspace pools, mobile
Reasons of a claim, mobile
Pool and claim health rules in gitops.md, mobile

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: c57c000f75

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…ber failure (#153)

Change-Id: Ic1ebf2845d1fca904fb5933fdc834b12cda6a172
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I5bee8847ee7906bdc82f8157c35249be7cc4b5dd
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

The new commits change only docs/how-to/gitops.md: they name ShareRevoking among the Share stalls, and they note that a bound claim whose member failed stays Progressing (tracked in #454).

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: c998543f7a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33
ThomasK33 added this pull request to the merge queue Nov 1, 2026
Merged via the queue into main with commit 8b4944a Nov 1, 2026
12 of 13 checks passed
@ThomasK33
ThomasK33 deleted the pool-gitops-health-troubleshooting branch November 1, 2026 02:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant