Audit smoke, soak, profiling, and benchmark harnesses for account daemon runtime isolation

Follow-up to #1691 and #1695.

PR #1695 isolates the focused security, watchdog, worker, and hook entry points that reproduced the account daemon shutdown. The longer-running smoke, soak, profiling, and benchmark harnesses have different lifecycle and measurement contracts, so they should be audited separately instead of expanding that fix.

Scope

scripts/smoke-local.sh and scripts/smoke-test.sh
scripts/soak-legs.sh and scripts/soak-test.sh
scripts/memlab.sh, scripts/memlab-drive.py, and scripts/memlab-report.py
scripts/benchmark-index.sh, scripts/benchmark-search-graph.sh, and scripts/clone-bench-repos.sh

Invariants to check

An inherited CBM_RUNTIME_DIR or CBM_CACHE_DIR is never reused for a product process started by a harness.
Concurrent harness runs receive different private roots.
Unix roots are owner-only. Windows runs use the repository's protected temp-root helper rather than duplicating ACL logic.
Install, update, uninstall, restart, and daemon cleanup stay inside the harness runtime.
A sentinel client in account runtime A remains connected with the same daemon PID and generation, and runtime A records no activation_shutdown.
A harness stops its own child processes before cleaning its runtime. If its private daemon remains active, cleanup leaves the root for diagnosis.
Runtime isolation does not alter benchmark coverage or invalidate timing comparisons. Any performance claim includes comparable setup, cache, and total wall-clock measurements.

Expected output

For each entry point, record whether it already satisfies these invariants. Add a focused fail-before reproduction for every confirmed gap, then submit scoped fixes separately from #1695.

Thank you for separating this audit from the focused fix in #1695. The boundary is correct: smoke, soak, profiling, and benchmark harnesses have lifecycle and measurement contracts that should not be folded into the security-script repair.

Please keep this issue as the audit ledger. Each confirmed isolation failure should have its own deterministic reproduction and focused fix, especially where changing setup could affect benchmark comparability. That gives every harness change an attributable reason and prevents a broad test-infrastructure sweep from becoming one unreviewable PR.

Audit ledger — account daemon runtime isolation

Static audit of every entry point in the scope at 0b515554 / 339b3f40 (2026-09-13). No harness was executed against a shared account; runtime reproductions are the fixture-driven contract tests linked below.

Mechanism. The daemon rendezvous is selected by CBM_RUNTIME_DIR alone: src/daemon/bootstrap.c:231-241 ("CBM_CACHE_DIR does not help either — it moves the cache, never the rendezvous"), the rendezvous key is a constant (src/daemon/service.c:144-161), the default parent is hard-coded /private/tmp (src/daemon/ipc.c:943) and on Windows SHGetFolderPathW(CSIDL_LOCAL_APPDATA) (src/daemon/ipc.c:5164) — the known-folder API, not %LOCALAPPDATA%. A HOME/XDG/TMPDIR/APPDATA/LOCALAPPDATA/CBM_CACHE_DIR sandbox therefore does not isolate the daemon. None of the ten entry points sourced scripts/test-runtime.sh from #1695, and no workflow or venue script sets CBM_RUNTIME_DIR.

Two failure shapes on a shared rendezvous: (1) daemon stop (src/main.c:2509-2537) reaches whichever daemon is active, regardless of build, and is refused only while committed clients exist; (2) a client arriving with another CBM_CACHE_DIR is refused with CBM_DAEMON_HELLO_CACHE_CONFLICT (src/daemon/service.c:220-227), so the first process to arrive owns the account.

Entry point	Own runtime before	Finding	Verdict	Reproduction / fix
`scripts/smoke-test.sh`	none (no `CBM_RUNTIME_DIR`, no own `CBM_CACHE_DIR`)	`retire_account_daemon()` issues `daemon stop` on the shared rendezvous from 7 call sites; its own comment reads "Retire the shared account daemon"	confirmed	`tests/test_smoke_runtime_isolation_contract.sh` — #2199
`scripts/smoke-local.sh`	`CBM_CACHE_DIR` + HOME/XDG/TMPDIR/APPDATA/LOCALAPPDATA sandbox	none of these move the rendezvous; an inherited `CBM_RUNTIME_DIR` was passed through	confirmed — fixed in `smoke-test.sh`, the process that starts every product process; wrappers unchanged	same
`scripts/soak-legs.sh`	none; starts no product process	passes the environment through to `soak-test.sh`	inherits	—
`scripts/soak-test.sh`	`CBM_CACHE_DIR` only	comment claimed "Isolate daemon coordination from interactive CBM sessions"; the run asserts that a session crash and the final shutdown stop "the shared daemon"	confirmed	`tests/test_soak_runtime_isolation_contract.sh` — #2196
`scripts/memlab.sh`	`CBM_CACHE_DIR` only	profiled process joins the account daemon; `cleanup()` removed the cache without stopping any daemon	confirmed	`tests/test_memlab_runtime_isolation_contract.sh` — #2197
`scripts/memlab-drive.py`	inherits (`Popen` without `env=`)	stops its own frontend; no daemon handling of its own	inherits	pass-through pinned by the memlab contract
`scripts/memlab-report.py`	n/a	starts no process	clean	—
`scripts/benchmark-index.sh`	none at all	`cli index_repository` indexed the benchmark repository into the operator's live store; the timing shared the operator's daemon	confirmed (+ comparability)	`tests/test_benchmark_runtime_isolation_contract.sh` — #2198
`scripts/benchmark-search-graph.sh`	none at all	8 frontend launches against a project that only exists in the operator's live store	confirmed (+ comparability)	same
`scripts/clone-bench-repos.sh`	n/a	`git clone` only; starts no product process	clean	—

Against the invariants in the issue body:

inherited CBM_RUNTIME_DIR/CBM_CACHE_DIR never reused — violated by every entry point that starts the product;
concurrent runs get distinct private roots — violated (constant key, default parent; a second run with another cache is refused);
Unix roots owner-only — violated (mktemp -d without chmod 700); Windows via the shared protected-root helper — soak and memlab still carry their own icacls stamping (left in place by these PRs: tests/test_soak_daemon_recovery_contract.sh pins those lines; deduplicating through scripts/ci/new-protected-temp-root.ps1 is a separate change if wanted);
install/update/uninstall/restart/cleanup inside the harness runtime — violated by the smoke;
harness stops its own children before cleaning its runtime — violated by memlab;
sentinel client in runtime A unaffected — not verified at runtime (static audit).

Fixes. Each harness sources scripts/test-runtime.sh, calls cbm_test_runtime_init and runs cbm_test_runtime_cleanup "$BINARY" from its EXIT trap, so the daemon it starts is stopped before its root goes and the root is kept for diagnosis when it will not stop. Each PR carries a fail-before contract test that drives the harness with an environment-probe fixture; no daemon is started by the tests.

Benchmark comparability. benchmark-index.sh now starts its private daemon before timing and records setup-time.txt and total-time.txt next to index-time.txt, so activation stays attributable and the total remains comparable with runs that paid activation inside the index timing. benchmark-search-graph.sh takes a repository path and indexes into the private cache before timing the queries against a warm daemon. docs/EVALUATION_PLAN.md §7 assumes the index stays in the live store for its graph phase; that assumption no longer holds and is called out in #2198.

Adjacent, outside the listed scope, same defect: scripts/smoke-invariants.sh (no isolation at all) and test-infrastructure/vm/vm-smoke.sh (mirrors smoke-local.sh; covered by the smoke-test.sh fix).

fix(test): isolate soak harness daemon runtime

fix(test): isolate memlab harness daemon runtime

fix(test): isolate smoke harness daemon runtime

fix(test): arm smoke runtime cleanup at init, retire the daemon first

fix(test): isolate benchmark harness daemon runtimes

fix(test): hand the benchmark index to the caller on request

fix(test): isolate soak harness daemon runtime

fix(test): isolate memlab harness daemon runtime

fix(test): isolate smoke harness daemon runtime

fix(test): arm smoke runtime cleanup at init, retire the daemon first

fix(test): isolate soak harness daemon runtime

fix(test): isolate memlab harness daemon runtime

fix(test): isolate smoke harness daemon runtime

fix(test): arm smoke runtime cleanup at init, retire the daemon first

fix(test): isolate benchmark harness daemon runtimes

fix(test): hand the benchmark index to the caller on request

fix(test): isolate benchmark harness daemon runtimes

fix(test): hand the benchmark index to the caller on request

docs(eval-plan): stop clearing the live cache before an isolated run

Merge origin/main into fix/1696-bench-runtime-isolation

Navigation Menu

Repository navigation

Audit smoke, soak, profiling, and benchmark harnesses for account daemon runtime isolation #1696

Scope

Invariants to check

Expected output

Audit ledger — account daemon runtime isolation

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Audit smoke, soak, profiling, and benchmark harnesses for account daemon runtime isolation #1696

Description

Scope

Invariants to check

Expected output

Activity

DeusData commented on Aug 18, 2026

astandrik commented on Sep 13, 2026

Audit ledger — account daemon runtime isolation

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions