[BUG] Spawned process (without a TTY) blocks forever when nobody streams its output #455

Describe the bug

A process started with POST /process/spawn (without allocate_tty) stops making progress once its unread output fills the 256-event output queue and the pipe buffer, unless a client is reading GET /process/{process_id}/stdout/stream.

The stdout/stderr readers send each chunk (up to 4 KiB) to outCh, which holds 256 events, with a blocking send:

With no stream client, the readers block once outCh is full, the pipe fills behind them, and the process blocks on its next write. It stays running until something reads the stream. Spawn disables scale-to-zero until the process exits, so the VM also stays up for as long as the process is blocked:

The process execution docs recommend spawn for agents, servers, watchers and other long-running processes, and document status polling as a way to check on one. A caller that only polls hits this as soon as the process writes more than the queue and the pipe can hold.

To Reproduce

1. POST /process/spawn with {"command": "sh", "args": ["-c", "i=0; while [ $i -lt 20000 ]; do echo line-$i; i=$((i+1)); done"]} and don't open the stream.
2. Poll GET /process/{process_id}/status. It is still running after 5 s.

I ran this against kernel-images-api built from main (70d2dba) for linux/arm64, running in a Debian container (stub ffmpeg and a fake DevTools log line so it starts without Chromium) and called over HTTP from a second container on an internal Docker network. I couldn't run the published image because it is amd64-only and my machine is arm64. A Go test calling the handlers directly on Linux shows the same result in 5 of 5 runs.

Expected behavior

A non-TTY process whose output nobody is reading keeps running and exits normally. A stream opened later still gets output and the exit event. (A TTY spawn with no /attach client also blocks once the PTY buffer fills; that's a separate case and not covered here.)

Why the obvious fix isn't enough

The obvious fix is to stop blocking when no stream is open: if the queue is full and nobody is streaming, drop an event instead. I tried that locally, dropping the oldest queued event while no stream is open and keeping today's blocking while one is. It fixes the freeze: the reproduction above exits in under 0.1 s.

But it also drops output in the normal spawn-then-stream pattern, because between the spawn call returning and the stream request arriving, no stream is open yet. In a Go test on Linux, a loop printing 3,000 lines with the stream opened 5, 20 or 100 ms after spawn missed the first line in 10 of 10 runs at each delay, losing on average about 2,000 (5 ms) to 2,600 (20 and 100 ms) of the 3,000 lines per run. With the stream opened immediately (0 ms), no run lost anything. The prototype was on top of #451; #451 alone, which keeps the blocking send, lost nothing at any of those delays (40 of 40 runs). Over the hosted API the gap is at least one client round trip, and a client that reopens the stream would hit the same gap.

Proposal: drop only after a grace period

• If the queue is full and no stream is open, wait up to N seconds (say 5) for a stream to open or for room in the queue, as today.
• If N seconds pass with no stream, start dropping, and keep dropping until a stream opens. A process nobody reads is then held back for at most N seconds before it runs freely. While the process is running, nothing is dropped as long as a stream opens within N seconds of the queue filling. (After the process exits, fix(process): don't drop process output and the exit event from streams #451 still closes the pipes after 2 s, so a stream opened later than that can miss what was left in them.)
• While a stream is open, keep today's behavior: a full queue waits for it.
• Never drop the exit event. With fix(process): don't drop process output and the exit event from streams #451 nothing is queued after it, so when the queue is full and no stream is open it can make room by dropping an output event.

On which end to drop: dropping the oldest keeps the most recent output, which is usually the useful end of a log. StartFsWatch takes the other approach and drops the newest event when its buffer is full, "preferring liveness over completeness":

Either works with the grace period. Other options include bounding the queue by bytes instead of events (so short lines don't fill it after 256 writes; on its own that doesn't remove the freeze), or dropping immediately and accepting the loss above. I'm happy to send a PR for whichever approach you prefer.

Screenshots

N/A

Desktop (please complete the following information):

• OS: server built from main (70d2dba) for linux/arm64, run in a Debian container (Docker on macOS)
• Browser: N/A (process API only)
• Version: main at 70d2dba

Additional context

The proposal builds on #451 (fix for #450), which queues the exit event after the output readers finish. #451 doesn't change the freeze: the reproduction above still blocks on its branch.