Skip to content

CPU backend: SSD streaming for routed experts (safe CPU inference on macOS)#508

Open
Tatlatat wants to merge 8 commits into
antirez:mainfrom
Tatlatat:cpu-ssd-streaming
Open

CPU backend: SSD streaming for routed experts (safe CPU inference on macOS)#508
Tatlatat wants to merge 8 commits into
antirez:mainfrom
Tatlatat:cpu-ssd-streaming

Conversation

@Tatlatat

@Tatlatat Tatlatat commented Jul 6, 2026

Copy link
Copy Markdown

The README says the macOS CPU path "will crash the kernel … Help us, if you have the guts." This PR is an attempt at that, from someone whose M4 Max hit the watchdog panic in #483.

This does not fix Apple's kernel bug — it keeps ds4 out of the regime that triggers it. With --cpu --ssd-streaming, routed expert weights are pread() into a bounded LRU cache of heap slabs instead of being faulted through the 81–98 GB whole-file mmap. Non-routed weights (attention, shared expert, embeddings, output) stay on the existing MAP_PRIVATE mmap — they are small and stay hot. The kernel never has to stream tens of GB of file-backed pages under concurrent fault pressure, which is what both observed panic signatures grow out of:

  1. watchdog starvation (I've try to run it on MacBook Pro, the machine crashed. #483 and my own machine): 12 CPU threads faulting through the huge mapping saturate kernel_task's pageout/compressor path until watchdogd misses its 90 s check-in;
  2. the VM accounting panic mentioned in the model_open comment — consistent with VM map entry / zone exhaustion under the same load (XNU has no vm.max_map_count-style errno path; exhaustion ends in jetsam-then-panic).

For what it's worth: llama.cpp maps models the same way ds4's Metal path does (MAP_SHARED + WILLNEED) and its users report the same whole-machine crashes in the same model≫RAM MoE regime (ggml-org/llama.cpp#19825, #3029). The trigger is the access pattern, not any specific mmap flag — which is why this PR bounds residency instead of tweaking flags.

Design

  • One seam. All CPU routed-expert access already flows through tensor_expert_bytes(); when streaming is enabled it returns a pointer into the cache slab instead of map + offset. The matvec code is untouched and unaware.
  • Per-backend implementation, zero GPU changes. No edits to ds4_metal.m, ds4_cuda.cu, ds4_rocm.cu, or ds4_ssd.c/h (CUDA already has its own streaming implementation; this follows that pattern — make cpu builds don't compile the Metal file, so sharing its code isn't possible). ds4_backend_supports_ssd_streaming() is untouched; the CPU allowance is a sibling branch at the gate.
  • Thread safety. The token-parallel decode path fetches from pool workers, so fetch runs under a single mutex; entry lifetime is guaranteed by pinning (entries touched in the current layer call can't be evicted), with the sequence only advanced by the orchestrator. A 4-thread eviction-stress self-test runs clean under ThreadSanitizer.
  • Bounded by construction. Cache budget = explicit flag or 25% of physical RAM, floored so one full layer of experts always fits (which also makes the "budget too small" abort unreachable in normal decode/prefill). If a routed tensor ever reached the seam unregistered, ds4 dies loudly rather than silently falling back to the mmap.
  • No more whole-file WILLNEED. With streaming on, only non-routed tensor ranges get the prefetch hint.
  • --ssd-streaming-preload-experts is ignored on CPU (warned); on macOS, --cpu without --ssd-streaming now prints an advisory warning (kept advisory on purpose — easy to drop or harden, your call).

Testing (M4 Max 128 GB, macOS 26.5.1, DeepSeek-V4 Flash IQ2XXS 81 GiB — 43 MoE layers × 256 experts)

  • make (Metal), make cpu, make test (Metal): clean; the branch adds three self-tests (--cpu-stream-cache/-fetch/-parallel, no model needed).
  • Official logprob vectors on --cpu --ssd-streaming: 3 of 4 executable cases pass exactly (16 GiB cache, 8m19s; identical results at a 3 GiB cache forcing ~24× eviction pressure). The one failing case (short_code_completion) fails byte-for-byte identically on unmodified Metal with the same file — my test model is a community-modified ("abliterated") quant, and the divergence tracks the weights, not the backend (inferred from the cross-backend isolation; --metal-tensor-equivalence passes strictly). Happy to rerun against a pristine quant if you want that in the record.
  • The point of the exercise: 500-token generation, physical footprint flat at 16.6 GiB against the 81 GiB model, swapfiles 32→32, 6.57 t/s decode, clean exit — on the same machine and model file that previously died with the I've try to run it on MacBook Pro, the machine crashed. #483 watchdog panic on the plain CPU path. Peak footprint with an 8 GiB cache: 9.18 GiB.
  • Floor-budget torture: explicit cache of 1 expert gets raised to the 1.69 GiB floor (with a notice); 2-token run does 1,842 misses / 1,586 evictions / 12.14 GiB of preads and still decodes correctly.

CPU stays a reference/debug path — this makes it safe, not fast.

Full evidence bundle (commands, per-case tables, memory samples)

3. Correctness evidence

3.1 Self-tests (unit level, no model required)

From Tasks 1-3, re-confirmed in this task's build-matrix run (§6):

$ ./ds4_test --cpu-stream-cache
cpu-stream cache policy: OK
$ ./ds4_test --cpu-stream-fetch
cpu-stream fetch/install: OK
$ ./ds4_test --cpu-stream-parallel
cpu-stream concurrent fetch stress: OK
  • --cpu-stream-cache: victim selection (lowest use_count, LRU tie-break), pinning semantics, eviction bookkeeping, and the all-pinned/no-victim case.
  • --cpu-stream-fetch: pread-based fetch/install against a synthetic temp file with a byte-patterned fixture, including hit/evict/refetch.
  • --cpu-stream-parallel: 4-thread, 50-round, 100-iterations-per-thread concurrent fetch stress against a 5-triple budget forcing repeated cross-round eviction; asserts zero data corruption and that eviction was actually exercised (evictions > 0). Additionally run under ThreadSanitizer (-fsanitize=thread, separate throwaway build) with zero TSan diagnostics — confirms the g_cpu_stream_lock mutex added in 5610610 fully serializes the shared cache state with no unprotected access path. Also run 20x back-to-back with no flakiness (Task 3).

3.2 Official logprob-vector fixture (tests/test-vectors/official.vec, 5 cases)

Run A — CPU + SSD-streaming, 16 GiB expert cache (2,427 experts cached of 11,008 total):

DS4_TEST_MODEL="$MODEL" DS4_TEST_BACKEND=cpu DS4_TEST_SSD_STREAMING=1 \
DS4_TEST_SSD_STREAMING_CACHE_GB=16 caffeinate -i ./ds4_test --logprob-vectors
Case ctx steps Result
short_italian_fact 16384 4 PASS
short_code_completion 4096 4 FAIL (token mismatch every step)
short_reasoning_plain 4096 1 PASS
long_memory_archive 16384 4 SKIPPED (pre-existing API/official graph mismatch, unrelated to backend)
long_code_audit 16384 4 PASS

3 of 4 executable cases pass exactly. Wall clock 8m19s.

Disambiguation (run immediately per the controller's anti-tampering protocol, since the failure was a hard token-bytes mismatch, not a logprob-delta-over-tolerance): the same 5-vector fixture run on Metal, no CPU/streaming env, same model file:

DS4_TEST_MODEL="$MODEL" ./ds4_test --logprob-vectors

An unrelated pre-existing ds4-server --metal process (PID 5634, not started by this work) was holding the GPU during this run and caused spurious kIOGPUCommandBufferCallbackErrorOutOfMemory failures on 2 of 5 cases (short_italian_fact, long_code_audit — both already passing on CPU, so this is not a concern). Crucially, short_code_completion — the one case that failed on CPU — ran on Metal with zero interleaved OOM errors, and its failure signature on Metal is byte-for-byte identical to the CPU failure:

ds4-test: vector short_code_completion step 0 selected token mismatch
ds4-test: vector short_code_completion step 1 selected token mismatch
ds4-test: vector short_code_completion step 1 official top token missing locally
ds4-test: vector short_code_completion step 2 selected token mismatch
ds4-test: vector short_code_completion step 2 official top token missing locally
ds4-test: vector short_code_completion step 3 selected token mismatch
ds4-test: vector short_code_completion step 3 official top token missing locally

Verdict: Metal (unmodified by this branch) fails identically to CPU+streaming on this one case → the divergence is caused by the abliterated model's weights differing from the official DeepSeek API's weights (inferred from cross-backend isolation; official non-abliterated weights were not available to test directly), not by the CPU/SSD-streaming code. This is corroborated independently by Run C's metal-tensor-equivalence subtest (§3.4), a strict local logits comparison (not against the official API) that passes cleanly on short_code_completion — Metal's own numerics are self-consistent; only the comparison against the official (non-abliterated) API vector diverges.

3.3 Eviction-stress rerun (Run B — cache_gb=3)

Amended pass criterion (set by the controller after the disambiguation cleared the code): identical failure signature to Run A, no new failures, no crashes, no cache death.

DS4_TEST_MODEL="$MODEL" DS4_TEST_BACKEND=cpu DS4_TEST_SSD_STREAMING=1 \
DS4_TEST_SSD_STREAMING_CACHE_GB=3 caffeinate -i ./ds4_test --logprob-vectors
  • Cache budget resolves to 3.00 GiB / 6.75 MiB per expert = 455 experts cached against 11,008 total routed experts (43 layers x 256 experts) — heavy forced eviction, vs. Run A's 2,427-expert budget.
  • Result: PASS. Failure signature extracted from the log is byte-for-byte identical to Run A: same 3 cases pass (short_italian_fact, short_reasoning_plain, long_code_audit), only short_code_completion fails with the exact same 7 assertions, long_memory_archive skips the same way. Grep for budget too small|abort|crash|signal|panic|Insufficient over the whole log: zero hits.
  • Wall clock 8m01s (not measurably slower than Run A's 8m19s despite 5x smaller cache — OS page cache was warm from prior runs on the same model file).

Conclusion: the CPU SSD-streaming cache survives heavy eviction (455 of 11,008 experts resident) with bit-identical decode behavior to a roomy 2,427-expert budget. The eviction path does not perturb correctness.

3.4 Full Metal regression (Run C)

DS4_TEST_MODEL="$MODEL" caffeinate -i make test — 18 subtests, 15 executed (3 skipped by design: no MTP head, no DS4_TEST_SSD_STREAMING for one CPU-only subtest). Only 2 of 15 fail, and both are the same short_code_completion root cause described above:

Subtest Result
long-context OK
tool-call-quality OK
think-tool-recovery OK
logprob-vectors ERR (model-caused, see §3.2)
metal-ssd-streaming-cache-pressure ERR (same root cause, restricted to short_code_completion)
local-golden-vectors OK
metal-short-prefill OK
metal-kernels OK
metal-tensor-equivalence OK (strict local logits check, all 5 cases incl. short_code_completion: top1_mismatch=0 min_top5_overlap=5/5 min_overlap=20/20 worst_rank_delta=0 worst_rms=0)
streaming-decode-prefill-correctness OK (skipped, no DS4_TEST_SSD_STREAMING)
mtp-verify-depth OK (skipped, no MTP head)
server OK
cpu-stream-cache OK
cpu-stream-fetch OK
cpu-stream-parallel OK

Wall clock 5m36s. make test exits nonzero (Error 1) purely because of the model-caused divergence — not touched/worked around, per the controller's explicit anti-tampering instruction.

4. Safety / mechanism evidence

4.1 The problem this branch makes safe (the kernel bug itself is Apple's to fix)

Before this branch, running the CPU backend on macOS with a model too large to fully fit resident (e.g. this 81 GiB model on a machine with 128 GB RAM, but with other memory pressure — including the concurrent unrelated ds4-server process observed on this very machine, which alone holds the full model Metal-resident plus a 60 GB on-disk KV cache) could fault the entire GGUF through a plain mmap. On current macOS/XNU kernels this has been observed to panic the kernel outright — this exact failure mode is documented in the repo's own README ("current macOS versions have a bug in the virtual memory implementation that will crash the kernel if you try to run the CPU code... each time you have to restart the computer") and corresponds to upstream issue #483, tracked with two independently-identified panic mechanisms against this project's own panic logs and the XNU source (xnu-12377.121.6): (1) watchdog starvation — 12 CPU threads all faulting pages from the same huge mmap saturate the kernel's page-fault path badly enough that watchdogd cannot check in within its timeout and the kernel panics; (2) VM map entry / zone exhaustion — the huge single mmap fragments into enough VM map entries under heavy concurrent fault pressure to exhaust a kernel zone. Both mechanisms are avoided by never mmap-faulting the routed-expert region at all.

4.2 The fix's mechanism

--cpu --ssd-streaming on this branch never faults routed-expert weight ranges through mmap. Instead: cpu_stream_init reads expert geometry (offsets, sizes, counts) straight from the GGUF tensor directory (no page faults — this is metadata only); non-routed weights are still selectively WILLNEED-hinted (small, safe); routed-expert weight slabs are pread() directly into a bounded LRU+hotness cache of heap-allocated buffers, sized by an explicit flag or a default of 25% of physical RAM (always floored so one full layer's worth of experts fits). The cache is bounded by construction — its heap footprint cannot exceed the configured budget regardless of model size or generation length, because every fetch either hits an existing entry or evicts before installing a new one.

4.3 Smoke test (Task 4, prior work — cited here for continuity)

8-token generation, 8 GiB cache, same 81 GB model:

caffeinate -i ./ds4 --cpu --ssd-streaming --ssd-streaming-cache-experts 8GB -m "$MODEL" -p "The capital of France is" -n 8

Peak RSS 17.77 GiB, peak footprint 9.18 GiB — both far below the 81 GB model file. (RSS includes resident non-routed mmap pages and page-cache attribution; peak physical footprint is the bounded metric to compare against the cache budget.) Exit code 0, no crash, no panic. First-ever safe CPU run of this specific model on this machine (the old unprotected CPU mmap path is documented in-code as having triggered the kernel-level VM panic previously).

4.4 New centerpiece evidence: 500-token bounded-footprint run (this task)

caffeinate -i ./ds4 --cpu --ssd-streaming --ssd-streaming-cache-experts 16GB \
  -m "$MODEL" -p "Write a long story about a lighthouse keeper." -n 500

Run concurrently with an independent memory-monitor loop sampling vmmap --summary <pid> | grep -i "physical footprint" and sysctl vm.swapusage every ~30s.

Result: completed in full, exit clean, ~505 tokens generated (word-count-estimated; the run hit the requested -n 500 cap mid-sentence, which is the expected/correct behavior for a token-budget cutoff, not a crash). Total wall clock was under 2 minutes — much faster than the tens-of-minutes estimate in the task brief, because the OS page cache for this model's non-routed weights and popular experts was already warm from the Task 4/5 runs earlier in this same session on the same machine/model.

Reported throughput (from ds4's own output):

ds4: prefill: 2.74 t/s, generation: 6.57 t/s

Physical-footprint samples (only 3 fell within the run's short duration before the monitor's own exit-detection stopped it — the plateau is nonetheless clear and consistent with the 16 GiB cache budget plus fixed overhead):

Timestamp Physical footprint Physical footprint (peak)
22:52:59 16.6 GiB 16.6 GiB
22:53:31 16.6 GiB 16.6 GiB
22:54:04 16.6 GiB 16.6 GiB
22:54:36 (process exited; monitor self-terminated)

Footprint is flat/bounded across the entire run — no monotonic growth, no drift — consistent with the streaming cache never exceeding its configured 16 GiB budget regardless of how many of the 11,008 routed experts get touched over the course of 500 generated tokens.

Swapfile count (ls /System/Volumes/VM/ | grep -c swapfile): 32 before, 32 after — zero new swapfiles.

vm.swapusage was already near-saturated throughout (total = 32768.00M used = 32154.88M free = 613.12M) — this is a pre-existing condition from the unrelated, still-running ds4-server --metal process (PID 5634) on this shared machine (that process alone holds the 81 GB model Metal-resident plus a 60 GB on-disk KV cache directory), not something this run caused: swap usage was static across all samples during our run (32154.88M in every single sample, byte-for-byte), meaning our streaming run made zero net contribution to swap pressure.

Exit status: 0 (process ran to completion and exited on its own; confirmed via pgrep returning no match for the ds4 PID immediately after the log's final generation: stats line appeared, with no error/panic/signal text anywhere in the log).

Win condition met: footprint bounded/plateaued (not monotonic growth), no swapfile explosion, clean exit, throughput numbers printed by ds4 itself.

4.5 Speed numbers (transparency only — CPU is a reference path, not the production target)

Run Cache Tokens/steps Duration Throughput
Task 4 smoke 8 GiB 8 tokens 5.96s (real) prefill 4.52 t/s, generation 3.79 t/s
Task 5 Run A (vectors) 16 GiB 5 short/medium cases, 1-4 steps each 8m19s n/a (correctness run, not throughput-focused)
Task 5 Run B (vectors, stress) 3 GiB same as Run A 8m01s n/a
This task: 500-token generation 16 GiB ~500 tokens < 2 min wall clock prefill 2.74 t/s, generation 6.57 t/s

No separate ds4-bench CPU run was performed — the CLI's own printed throughput from the generation run above is the reported number, per the task's "no separate bench needed for CPU" instruction.

5. Metal non-regression

Metal before/after ds4-bench sweep: SKIPPED. An unrelated user process ds4-server --metal (PID 5634) was confirmed running and holding the GPU on this machine throughout this task (pgrep -fl ds4-server returns it; started well before this task, serving on port 8081 with a 100k-token context and a 60 GB on-disk KV cache directory). Running a Metal bench sweep concurrently would produce contaminated, non-comparable numbers (this exact contamination was observed and documented in Task 5's disambiguation run — spurious kIOGPUCommandBufferCallbackErrorOutOfMemory errors). The unrelated server was not killed (out of scope; it is another user's/process's workload on a shared machine).

Metal non-regression is instead evidenced by:

(a) Zero Metal source files touched.

$ git diff --stat main...HEAD
 README.md        |  14 +-
 ds4.c            | 614 +++...
 ds4_help.c       |   6 +-
 tests/ds4_test.c |  47 ++++-
 4 files changed, 666 insertions(+), 15 deletions(-)

$ git diff --name-only main...HEAD | grep -iE "metal|cuda|rocm"
(no output — zero matches)

ds4_metal.m, ds4_cuda.cu, ds4_rocm.cu, and ds4_ssd.c/h are all untouched by this branch, confirmed both by the diffstat (they don't appear at all) and by an explicit grep for their names/extensions across the changed-file list. The one shared-code touchpoint, ds4_backend_supports_ssd_streaming, was read but not modified — the CPU allowance was added as a sibling branch in the calling gate, not a change to that predicate function, so GPU-backend auto-cache logic is provably unreachable from this diff.

(b) Full make test pass (Task 5 Run C, §3.4 above). 15 of 15 executed subtests pass except the two instances of the single model-caused short_code_completion divergence, which is independently confirmed to reproduce identically on Metal with zero CPU/streaming code involved. Every Metal-specific subtest (metal-short-prefill, metal-kernels, metal-tensor-equivalence, local-golden-vectors, tool-call-quality, think-tool-recovery, long-context, server) passes cleanly. metal-tensor-equivalence in particular is a strict local-reference logits comparison (not an official-API comparison) and is clean across all 5 test-vector cases, positively confirming Metal's numerics are unperturbed.

Between (a) and (b), Metal non-regression is established without needing a live bench run on a GPU that is not actually free on this machine right now.

6. Build matrix

All three builds re-run clean from scratch in this task, on HEAD 340a7aa:

$ make clean && make            # Metal (default) build
... (0 warnings, 0 errors)
cc ... -o ds4 ds4_cli.o ds4_help.o linenoise.o ds4.o ds4_distributed.o ds4_ssd.o ds4_metal.o -lm -pthread -framework Foundation -framework Metal
cc ... -o ds4-server ...
cc ... -o ds4-bench ...
cc ... -o ds4-eval ...
cc ... -o ds4-agent ...

$ make clean && make cpu        # CPU-only (DS4_NO_GPU) build
8 warnings generated.           # pre-existing, identical count/content to the pre-branch baseline
                                 # (unused-function warnings on GPU-only helpers compiled out
                                 #  under -DDS4_NO_GPU, e.g. ds4_engine_configure_streaming_auto_cache,
                                 #  ds4_engine_preload_pro_q4_expert_tables — none relate to cpu_stream code)
cc ... -o ds4 ds4_cli_cpu.o ds4_help.o linenoise.o ds4_cpu.o ds4_distributed.o ds4_ssd.o -lm -pthread
cc ... -o ds4-server ...
cc ... -o ds4-bench ...
cc ... -o ds4-eval ...
cc ... -o ds4-agent ...

$ make clean && make && make ds4_test   # test binary, built against the Metal object set
cc ... -c -o ds4_test.o tests/ds4_test.c
cc ... -o ds4_test ds4_test.o ds4_help.o ds4_kvstore.o rax.o ds4.o ds4_distributed.o ds4_ssd.o ds4_metal.o -lm -pthread -framework Foundation -framework Metal

Self-tests re-confirmed against this exact fresh build:

$ ./ds4_test --cpu-stream-cache && ./ds4_test --cpu-stream-fetch && ./ds4_test --cpu-stream-parallel
cpu-stream cache policy: OK
cpu-stream fetch/install: OK
cpu-stream concurrent fetch stress: OK

The 8 make cpu warnings were separately verified in Task 4 (via git stash + rebuild) to be present identically on pre-branch HEAD — not introduced by this branch.

7. Concerns / caveats carried into this evidence bundle

  1. Model is a community "abliterated" variant. Its greedy-decode outputs diverge from the official DeepSeek API on one of five official test-vector cases (short_code_completion). This is fully isolated to the model weights via the Metal disambiguation (§3.2) and is not attributable to this branch.
  2. long_memory_archive is skipped by the test harness itself (API/official graph mismatch) in every run, pre-existing and unrelated to this branch — reduces effective official-vector coverage to 4 of 5 cases per run.
  3. Metal bench sweep was not run due to a genuinely occupied GPU on this shared machine (unrelated ds4-server --metal, PID 5634, left running). Non-regression is argued from the diffstat + make test, not a live A/B bench; a maintainer with a free GPU can run the ds4-bench sweep from the commands appendix below to get an additional numeric comparison if desired.
  4. The 500-token run finished much faster than estimated (under 2 minutes vs. an estimated 30-90 minutes) because the OS page cache was warm from earlier same-session runs on the same model file. This means the footprint monitor only captured 3 samples 30s apart rather than the dozens originally anticipated — however, all 3 samples plus the smoke test's independent RSS/footprint numbers (§4.3) are mutually consistent with a flat, bounded cache, and no sign of growth appeared at any point.
  5. --ssd-streaming-cache-bytes (raw byte budget) and the preload-warning path were not exercised end-to-end in any task (only --ssd-streaming-cache-experts was used); the preload-ignored-on-CPU fprintf was verified by code inspection only. Considered low-risk (single unconditional print, no control-flow effect).
  6. This machine runs a long-lived, unrelated ds4-server --metal process that holds significant GPU/unified-memory/swap pressure throughout this entire evidence-gathering session; it was never started or stopped by this work, and its presence is called out wherever it could plausibly have contaminated a result (Metal disambiguation OOMs in Task 5, and the pre-saturated vm.swapusage baseline in §4.4).

8. Commands appendix (verbatim, for reproduction)

Self-tests:

./ds4_test --cpu-stream-cache
./ds4_test --cpu-stream-fetch
./ds4_test --cpu-stream-parallel

Task 4 smoke test:

caffeinate -i ./ds4 --cpu --ssd-streaming --ssd-streaming-cache-experts 8GB \
  -m "$MODEL" -p "The capital of France is" -n 8

(wrapped in /usr/bin/time -l for RSS/footprint capture)

Task 5 Run A (official logprob vectors, CPU+streaming, 16 GiB cache):

DS4_TEST_MODEL="$MODEL" DS4_TEST_BACKEND=cpu DS4_TEST_SSD_STREAMING=1 \
DS4_TEST_SSD_STREAMING_CACHE_GB=16 caffeinate -i ./ds4_test --logprob-vectors

Task 5 disambiguation (same vectors, Metal, no streaming env):

DS4_TEST_MODEL="$MODEL" ./ds4_test --logprob-vectors

Task 5 Run B (eviction stress, 3 GiB cache):

DS4_TEST_MODEL="$MODEL" DS4_TEST_BACKEND=cpu DS4_TEST_SSD_STREAMING=1 \
DS4_TEST_SSD_STREAMING_CACHE_GB=3 caffeinate -i ./ds4_test --logprob-vectors

Task 5 Run C (full Metal regression):

DS4_TEST_MODEL="$MODEL" caffeinate -i make test

This task's centerpiece 500-token bounded-footprint run:

MODEL=/Users/tatlatat/.claude/codex-fleet-local-model/scratchpad/ds4-build/gguf/cyberneurova-DeepSeek-V4-Flash-abliterated-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-aligned.gguf

caffeinate -i ./ds4 --cpu --ssd-streaming --ssd-streaming-cache-experts 16GB \
  -m "$MODEL" -p "Write a long story about a lighthouse keeper." -n 500 \
  > /tmp/task7-gen.log 2>&1 &

# concurrently, sampled every ~30s:
PID=<ds4 pid>
vmmap --summary $PID | grep -i "physical footprint"
sysctl vm.swapusage

# swapfile count before/after:
ls /System/Volumes/VM/ | grep -c swapfile

Environment check (run before any GPU-affecting step):

pgrep -fl ds4-server

Build matrix:

make clean && make            # Metal (default)
make clean && make cpu        # CPU-only (DS4_NO_GPU)
make clean && make && make ds4_test

Git evidence:

git log --oneline main..HEAD
git diff --stat main...HEAD
git diff --name-only main...HEAD | grep -iE "metal|cuda|rocm"   # expect: no output

🤖 Generated with Claude Code

https://claude.ai/code/session_01A4PSPFYh6oBLsLWZsu3jB2

Tatlatat and others added 8 commits July 5, 2026 21:10
Implements the CPU SSD streaming expert cache infrastructure:
- cpu_stream_entry: per-expert cache slot with LRU timestamp, hotness counter, and pinning
- cpu_stream_state: global cache state with per-layer geometry and statistics
- cpu_stream_pick_victim(): selects lowest use_count entries for eviction, oldest as tie-break
- cpu_stream_evict_one(): frees buffers, updates accounting, marks invalid
- cpu_stream_layer_begin(): increments sequence counter for pinning in current layer call

Pinning prevents eviction of entries touched during the current layer call (pin_seq == seq).
Unpinned entries (pin_seq != seq or pin_seq == 0 initially) are candidates for eviction.

Unit test verifies victim selection policy and eviction behavior without requiring a model.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A4PSPFYh6oBLsLWZsu3jB2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant