feat: police helper-process drift instead of chasing OOM spikes

TESTING.md

@@ -23,6 +23,15 @@ Both modes run the identical sequence of stages; only what each stage is permitt
 
 Because all three stages run for real in dry-run — just scoped away from production — a passing PR is a meaningful signal that a live release would also succeed, not a guess based on static checks alone.
 
+## The memory watchdog is the one part with runtime behaviour
+
+Everything else here is declarative and is covered by the stages above. `script-memory-watchdog.sh` decides at runtime whether to kill a process, so it gets two things neither lint nor a template push can provide:
+
+- **Fixtures** — `./templates/kubernetes/homelab-workspace/script-memory-watchdog-test.sh`, also run by the `watchdog` job in `.github/workflows/test.yaml`. The suite is built around negative assertions paired with the mutation that must flip them: "it did not kill the agent session" proves nothing unless removing one rule makes it kill the agent session. Two of its own safety properties matter as much as its assertions: `kill` is shadowed by a function throughout, because the fixture pids name real processes in whatever container runs the suite; and every load of the watchdog redirects its stdout emission to a temporary file, with the suite exiting outright if that seam did not take — the real default is the container's own stdout, and a run without the seam has already put fixture kill lines into a live workspace's log stream.
+- **A live drill on the `test` workspace**, which has disposable storage and can be wrecked freely. Fixtures cannot answer whether a real process tree classifies correctly, whether a supervisor really does respawn what was killed, or whether the circuit breaker stops a loop rather than joining it. The drill that has been run: a stand-in session root with an over-budget helper that a supervisor respawns, a second helper inside its budget, and a detached tool call larger than both. In `enforce` mode with the dwell shortened, the watchdog killed the drifted helper three times, disarmed that role on the third with the loop message, and left the fourth incarnation, the tool call, and the session roots untouched.
+
+Neither replaces the other, and the live one is where every defect that mattered in this component has been found.
+
 ## After merge
 
 Merging to `main` is what flips the pipeline into live mode — there's no separate promotion step afterward. The dry-run pass on the PR is the actual release gate.

templates/kubernetes/homelab-workspace/coder-agent.tf

@@ -60,5 +60,16 @@ resource "coder_agent" "main" {
     interval = 60
     timeout  = 1
   }
+  metadata {
+    display_name = "Largest Helper"
+    key          = "7_largest_helper"
+    # The biggest restartable helper as a share of its budget, also from the
+    # watchdog. This tile is the point of the drift half of the design: the
+    # operator used to obtain this number by running ps and then killing things
+    # by hand, and a number nobody can see is a number nobody acts on.
+    script   = "cat $${HOME}/.local/state/vscode-memory-watchdog/top 2>/dev/null || echo '-'"
+    interval = 60
+    timeout  = 1
+  }
 
 }

templates/kubernetes/homelab-workspace/env.tf

@@ -4,17 +4,16 @@ resource "coder_env" "welcome_message" {
   value    = local.homebrew_directory
 }
 
-# The switch that arms the memory watchdog. "observe" measures, publishes
-# headroom and logs what it would have done; "enforce" additionally sets
-# RLIMIT_DATA ceilings and sheds load.
+# The switch that arms the memory watchdog. "observe" measures and records what
+# it would have done; "enforce" kills helpers that have been over budget for ten
+# minutes; "enforce-all" adds the two editor roles whose restart the operator can
+# see.
 #
-# Set from a mutable workspace parameter rather than hardcoded here, because the
-# right value is a per-workspace judgement: the tier thresholds are absolute
-# bytes sized for an 8 GiB pod, so the same setting that is right there sits
-# permanently near L1 on a 4 GiB one. It defaults to "observe" and should stay
-# there until the ceilings and thresholds have been set from the calibration data
-# the watchdog collects - too low kills a healthy extension host mid-edit, too
-# high makes the mechanism inert.
+# Set from a mutable workspace parameter rather than hardcoded here, because it
+# is the one control that decides whether the watchdog may signal anything, and
+# because turning it off has to be a parameter change rather than a code change
+# on a bad day. The budgets themselves are derived from the pod's own memory.max,
+# so this value does not have to be reconsidered per pod size.
 resource "coder_env" "memory_watchdog_mode" {
   agent_id = coder_agent.main.id
   name     = "WATCHDOG_MODE"

templates/kubernetes/homelab-workspace/parameters.tf

@@ -64,21 +64,26 @@ data "coder_parameter" "system_packages" {
 data "coder_parameter" "memory_watchdog_mode" {
   name = "memory_watchdog_mode"
 
-  default      = "observe"
+  default      = "enforce"
   display_name = "Memory Watchdog"
-  description  = "What the memory watchdog is allowed to do when the pod runs low on unreclaimable-memory headroom"
+  description  = "What the memory watchdog may do about a helper process that has been over its budget for ten minutes"
   icon         = "/icon/memory.svg"
   mutable      = true
 
   option {
     name        = "Observe only"
     value       = "observe"
-    description = "Measure, publish headroom and log what it would have done. Sets no limits and sends no signals"
+    description = "Measure, record every sweep, and log the kill it would have made. Sends no signals"
   }
   option {
-    name        = "Enforce"
+    name        = "Enforce (helpers)"
     value       = "enforce"
-    description = "Also cap helper processes with RLIMIT_DATA and shed load as headroom falls. Do not enable before the thresholds have been set from calibration data"
+    description = "Also kill drifted helpers: language servers, the file watcher, native extension helpers, and MCP servers an agent session spawned. Each restarts invisibly"
+  }
+  option {
+    name        = "Enforce (helpers and editor)"
+    value       = "enforce-all"
+    description = "Also kill the VS Code extension host and server. These restart visibly, so they are armed separately"
   }
 }
 
@@ -91,7 +96,7 @@ locals {
   # below, and anything unrecognised falls back to the inert mode rather than to
   # whatever was supplied.
   validated_watchdog_mode = contains(
-    ["observe", "enforce"], data.coder_parameter.memory_watchdog_mode.value
+    ["observe", "enforce", "enforce-all"], data.coder_parameter.memory_watchdog_mode.value
   ) ? data.coder_parameter.memory_watchdog_mode.value : "observe"
 
   validated_system_packages = (data.coder_parameter.system_packages.value != "") ? [