fix(run_code): enforce the timeout in the runner, report full stderr, and probe cores on macOS

[earakely-scale]

Summary

run_code ran its script under GNU timeout, which macOS doesn't ship, so every run_code step on a macOS host failed with exit 127 before the script started. This makes the runner enforce the limit itself, on every target.

• Timeout in the runner. run_code_runner.py takes the limit as a fourth argument and supervises. It reruns itself without that argument in a new session, which gives the script its own process group, and waits up to the limit.
  • At the limit the group gets SIGTERM, then SIGKILL 5 s later, and the runner exits 124. That's the status timeout gave, so the "timed out after Ns" message is unchanged.
  • SIGTERM, SIGINT and SIGHUP sent to the runner are forwarded to the group, which is SIGKILLed after the same grace period. As under GNU timeout, they're caught before the script starts, so the script gets their defaults even when the runner was started with them ignored (as a background job's SIGINT is).
  • Nothing from the run dir runs in the supervisor. The run dir is sys.path[0] and holds the artifact's files, so only the supervising branch imports signal, subprocess and time, and only after taking the run dir off its path. The script's own process imports what the runner always did, so a script that has its own signal.py still gets it.
  • It exits with the script's status, or 128+N when signal N killed it. Stdlib only, Python 3.9 compatible.
• The end of stderr. A failed step reports up to the last 1 MB of the script's stderr, instead of its first 1,000 characters, which cut off the end, where the exception is. Past 1 MB, the error says how many bytes before the kept part were cut. The same applies to failures reading the result file.
• Core count on macOS. MultiEnv's load-concurrency probe becomes nproc 2>/dev/null || getconf _NPROCESSORS_ONLN. It used to warn on every macOS load and fall back to loading everything at once.

What changes for a run

	Before	After
macOS host	exit 127	runs
Linux: a script finishes, crashes, or hits the limit	result, "exited N: …", or "timed out" (124)	the same
A script that ignores SIGTERM at the limit	ran on past it (timeout without -k only sends SIGTERM)	SIGKILLed 5 s later
Processes the script started, at the limit	timeout signalled its group	the same, then SIGKILL after the grace
Processes left running after a normal finish	untouched	untouched
A script killed by a signal	the provider's encoding (-N from the local provider)	128+N everywhere
Error text on a crash	the first 1,000 characters of stderr	up to the last 1 MB of stderr

Why 1 MB of stderr

The error is stored whole, twice, in the instance's Mongo document: in failed_steps, and in the task's failure record. It's also raised to Temporal as the activity failure. Uncapped, a script that wrote about 8 MB of stderr before failing could exceed Mongo's 16 MB document limit, or Temporal's payload limit. In chaos runs, 50 MB of stderr produced a 52,428,855-character error. The last 1 MB keeps any realistic traceback whole, and keeps both copies well under both limits.

Tests

• The runner, run directly:
  • a result is written;
  • the exit status and stderr pass through;
  • a signal death gives 128+N;
  • the limit stops the script and the sleep it started (124);
  • a script that ignores SIGTERM is SIGKILLed after the grace;
  • SIGTERM, SIGINT and SIGHUP sent to the runner stop the script and its child (128+N), with the runner started with SIGINT and SIGHUP ignored;
  • a forwarded signal that the script ignores is followed by SIGKILL;
  • artifact files named signal.py, subprocess.py and time.py never run in the supervisor, with the run dir reached through a symlink, as macOS's /tmp is;
  • a script still imports its own signal.py;
  • the source parses as Python 3.9.
  • These tests stage runner.py inside the run dir, as run_code does.
• Through run_code, with the real runner:
  • a script past its timeout reports "timed out after 1s";
  • a crash reports all 5,000+ characters of stderr, through to ValueError: the cause;
  • past the cap, the error keeps the end and says how many bytes were cut, counting a character cut in two.
  • The test fake no longer strips timeout, which it only did because macOS has none.
• Core probe: it reads this host's cores, and falls back to getconf when nproc isn't on PATH.
• Unit tier: 8,302 passed on macOS.
• By hand, under macOS's /usr/bin/python3 (3.9.6): success, a timeout at 1 s, and a SIGTERM-ignoring script killed at 6 s.
• tst/integration/task_step/test_run_code_named_sandbox_local.py: run_code now passes on macOS. The VM-mode case still fails at a later step, collect_artifacts, which uses GNU stat -c and find -printf. That's separate and not in this PR.

Rollout

bump-version: this is on the hot path of every run_code step. Chaos runs on Linux and macOS VMs, plus live modal_vm and beta_scale runs, are in one comment below. Each runs the same battery on the old and new code.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no new blocking issue was found.

Summary

The PR replaces GNU timeout with a Python supervisor and adds a macOS core-count fallback. Since the previous review, it also:

• The Python runner stops scripts when their time limit expires.
• Failed run_code errors keep the end of stderr.
• MultiEnv can find core counts when nproc is missing.

No new actionable issues were found. The earlier import-shadowing finding is fixed: the supervisor removes the run directory before importing its extra modules. That thread was unnumbered, so it has no previousFindings entry.

earakely-scale explicitly acknowledged and deferred the oversized-stderr risk in the PR description. The current code now limits each reported stderr tail.

Tests were inspected, not run.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[run_code] --> B[Python supervisor]
  B --> C[Script in a new process group]
  C --> D{Script finishes?}
  D -->|Yes| E[Return script status]
  D -->|Timeout| F[Send SIGTERM]
  F --> G[Send SIGKILL after exit or grace period]
  G --> H[Return 124]
  E --> I[Report failed stderr with bounded tail]

Loading

_{Reviews (4) · Last reviewed commit: "fix(run_code): keep the last 1 MB of a f..." · Reviewed by Greptile}

[earakely-scale]

Chaos and live verification

The same battery went through the old path (agentenv-framework 0.9.1331, GNU timeout) and this PR (f0b76ff; the stderr cases again on 36d016a) on these hosts:

• Linux: Ubuntu 24.04 on an m5.2xlarge, with Python 3.12.3 and GNU timeout 9.4.
• macOS 15.8 on a mac-m4.metal.
  • Old: run with Homebrew's coreutils 9.12 installed.
  • New: run on a stock Mac, with coreutils removed, under Python 3.9.6 (Xcode CLT) and 3.14.7 (Homebrew).
• Live hosts: modal_vm and beta_scale VMs (Python 3.10.12), and plain Modal containers (Python 3.12.12), driven from a laptop with the sdk's dev config, pinned to the worker's versions (sdk 0.34.4).

Each case ran on Linux and macOS in local VM mode and container mode, and on all three live hosts.

The same on both:

• Success with a result, exit 3 with its stderr, and 50 MB of stdout.
• Limits of 1, 2 and 5 s against scripts sleeping 10× as long: 90 of 90 trials per host gave 124, each within 0.15 s of the limit, with nothing left behind.
• 20 children at the limit: none left.
• A setsid daemon escapes both paths, whether the script finished or timed out (3 of 3).
• Four run_code steps timing out at once: all four report "timed out after 2s".
• SIGTERM or SIGINT sent to agent-env run mid-script, 15 trials each: 143 or 130 within 0.5 s, torn down, nothing left.
• The CoopBench bundle's two run_code steps both ran. The task completed 18/18 on the stock Mac and on Linux, scoring 0, its real hard-gate grade.

Different, as intended:

	Old	New
Stock macOS	every run_code fails: [Errno 2] No such file or directory: 'timeout'	runs
A script that ignores SIGTERM at the limit	runs on until the harness's 45 s cap; in a container it kept running after the run was cancelled	stopped at the limit plus the grace: 7.0–7.9 s locally, 11 s on modal_vm
A 20 KB traceback	the first 1,049 characters, cut mid-log	all ~20,700, ending with ZeroDivisionError
50 MB of stderr, then exit 1	1,050 characters	the last 1 MB (36d016a), behind [first 51,380,229 bytes of stderr cut]: a 1,048,665-character error on every host, live ones included. Before the cap it was 52,428,855 characters
A script killed by a signal	−N	128+N
A trivial script's run	0.04 s	0.09 s on Linux, 0.05 s on the Mac (one more interpreter start)

Found and fixed along the way:

• 546113b (Greptile): artifact files named after the supervisor's imports could run in it, and a script that imported its own signal.py got the stdlib's.
• f0b76ff (chaos): started as a background job, which ignores SIGINT, the runner let the script inherit the ignored SIGINT and SIGHUP. So a forwarded one did nothing until the SIGKILL 5 s later (137). The handlers are now set before the script starts, as GNU timeout does. Under the same conditions: SIGINT gives 130 (15 of 15) and SIGHUP 129 (5 of 5), in 0.06 s at most, on Linux and the Mac.

Core probe: the stock-Mac CoopBench run logged no core-count warning.

Not in this PR: on a macOS VM-mode host, collect_artifacts uses GNU stat -c and find -printf.