Calibrate memory watchdog prlimit ceilings and decide observe -> enforce rollout

[ppat]

Context

The workspace pod (8 GiB memory limit) is repeatedly OOMKilled by VS Code Remote Server
memory growth. Because /sys/fs/cgroup/memory.oom.group = 1, a cgroup OOM kills every
process in the container as a group — the IDE, all tmux sessions, and all parallel agent
runs die together. No cgroup-level limit is reachable without privileged: true
(/sys/fs/cgroup is ro, cgroup.subtree_control is empty, cgroup namespace is private,
uid 10001, CapEff = 0), which DESIGN.md's "Unprivileged by default" stance rules out.

PR #859 ships a userspace watchdog (script-memory-watchdog.sh, scripts.tf, the Memory Headroom agent metadata, and DESIGN.md/CLAUDE.md documentation of why a userspace watchdog
exists at all) in observe-only mode: it measures headroom, logs candidate actions, and
sets no prlimit ceilings and sends no signals.

What headroom means here

memory.current reads misleadingly high even on an idle container, so the watchdog computes:

U = anon + shmem + unevictable + slab_unreclaimable + kernel_stack + pagetables + percpu + sock
H = memory.max - U

Measured at rest: memory.current 97% of limit, coder stat mem 63%, true U 23%,
headroom H 6.12 GiB, memory.pressure full avg10 0.00. A naive memory.current > 85%
trigger would fire permanently on an idle container — U gets it right by roughly a factor
of four.

What's open

The proposed soft RLIMIT_DATA ceilings (server-main.js 1.5 GiB, extensionHost 3 GiB,
tsserver 3.5 GiB, other language servers 1 GiB, fileWatcher 1 GiB, ptyHost never limited)
are a starting proposal derived from role and the 8 GiB budget, not from measurement.
They are not yet agreed and nothing sets them today. Too low silently kills a healthy
extension host mid-edit; too high makes the mechanism inert.

Acceptance criteria

• Collect at least a week of L0 calibration data (U and its components, logged by the
  observe-only watchdog) with the workspace under normal use.
• Agree L2/L3 prlimit thresholds from the observed distribution rather than the
  starting proposal.
• Decide and execute the observe -> enforce transition (the mode switch introduced in
  PR feat: add observe-only memory watchdog to the workspace template #859).
• Re-run the test_mode negative test: drive headroom below the L4 floor from outside
  the --type=ptyHost subtree and confirm the VS Code Remote Server tree dies while a
  coder ssh-attached tmux session and the coder agent process survive, with
  memory.events.oom_group_kill not incrementing.

Related

• PR feat: add observe-only memory watchdog to the workspace template #859 (this repo) — observe-only watchdog implementation.
• ppat/homelab-ops-kubernetes-apps — Coder ServiceMonitor + node-exporter pgscan_*/
  pgsteal_* fields, needed so headroom/reclaim pressure is visible outside the pod, not
  just in the watchdog's own log.

[ppat]

Closing as superseded. Four of the five premises this issue rests on are no longer true.

The blast radius was fixed at the kubelet layer, not in the pod. singleProcessOOMKill: true
(KubeletConfiguration) sets memory.oom.group = 0, so a cgroup OOM now kills only the offending
process instead of every process in the container. Verified live: oom.group=0 in a rebuilt
workspace. That removes the entire justification for a userspace watchdog as spike defense — the
kernel does the job, with none of the polling fragility.

The attribution was wrong. Kernel oom-kill:constraint=CONSTRAINT_MEMCG records over a 30-day
window name task=claude for 5 of 7 kills and task=node for 2. None were a VS Code server
process. The whole VS Code tree is ~727 MB PSS when fresh, against spikes measured at 70-220 MB/s,
so shedding all of it buys 5-10 seconds. When the ladder finally saw a real spike in live testing
it climbed correctly and logged no-candidates tier=L3 tree=0 eligible=0 — the runaway was not in
the tree it manages.

The proposed ceilings were computed against the wrong quantity. RLIMIT_DATA accounts VmData,
not RSS. Measured live: extension host 1004 MB VmData against 497 MB resident; a derived
file-watcher ceiling of 512 MB sat below what an idle file watcher already held, so enforce mode
would have killed it on its next allocation, permanently. Separately, RLIMIT_DATA on claude
produces panic: abort() / SIGSEGV rather than a catchable error, and idle sessions hold ~6.5 GB
VmData at ~0.9 GB RSS — no value is both safe at rest and meaningful during a spike.

Week-long calibration on the live workspace will not happen, so acceptance criteria that depend
on it are not executable as written.

What survives, and is worth keeping: the headroom measurement. memory.current reads 97% on an
idle container while true unreclaimable U is 23%; coder stat mem reads 63%. Any threshold built
on the stock readings fires permanently at rest. That reasoning is recorded in DESIGN.md and should
stay there regardless of what happens to the watchdog.

What replaces this issue: the watchdog is being re-aimed from spike defense to bounding slow
drift in the standing population — different targets (MCP servers and language servers, including
processes outside ~/.vscode-server/ that the tree-scoped selection cannot currently see),
different quantity (RSS rather than VmData), and different enforcement
(kill-and-let-the-supervisor-restart rather than RLIMIT_DATA, since every one of those components
is designed to be restarted). Budget calibration is still real work, but for entirely different
numbers, and it belongs with that change rather than here.

Evidence contributed upstream: anthropics/claude-code#81343.