CPU backend: SSD streaming for routed experts (safe CPU inference on macOS)#508
Open
Tatlatat wants to merge 8 commits into
Open
CPU backend: SSD streaming for routed experts (safe CPU inference on macOS)#508Tatlatat wants to merge 8 commits into
Tatlatat wants to merge 8 commits into
Conversation
Implements the CPU SSD streaming expert cache infrastructure: - cpu_stream_entry: per-expert cache slot with LRU timestamp, hotness counter, and pinning - cpu_stream_state: global cache state with per-layer geometry and statistics - cpu_stream_pick_victim(): selects lowest use_count entries for eviction, oldest as tie-break - cpu_stream_evict_one(): frees buffers, updates accounting, marks invalid - cpu_stream_layer_begin(): increments sequence counter for pinning in current layer call Pinning prevents eviction of entries touched during the current layer call (pin_seq == seq). Unpinned entries (pin_seq != seq or pin_seq == 0 initially) are candidates for eviction. Unit test verifies victim selection policy and eviction behavior without requiring a model. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A4PSPFYh6oBLsLWZsu3jB2
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A4PSPFYh6oBLsLWZsu3jB2
…t floor notice, macOS advisory
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The README says the macOS CPU path "will crash the kernel … Help us, if you have the guts." This PR is an attempt at that, from someone whose M4 Max hit the watchdog panic in #483.
This does not fix Apple's kernel bug — it keeps ds4 out of the regime that triggers it. With
--cpu --ssd-streaming, routed expert weights arepread()into a bounded LRU cache of heap slabs instead of being faulted through the 81–98 GB whole-file mmap. Non-routed weights (attention, shared expert, embeddings, output) stay on the existingMAP_PRIVATEmmap — they are small and stay hot. The kernel never has to stream tens of GB of file-backed pages under concurrent fault pressure, which is what both observed panic signatures grow out of:kernel_task's pageout/compressor path untilwatchdogdmisses its 90 s check-in;model_opencomment — consistent with VM map entry / zone exhaustion under the same load (XNU has novm.max_map_count-style errno path; exhaustion ends in jetsam-then-panic).For what it's worth: llama.cpp maps models the same way ds4's Metal path does (
MAP_SHARED+WILLNEED) and its users report the same whole-machine crashes in the same model≫RAM MoE regime (ggml-org/llama.cpp#19825, #3029). The trigger is the access pattern, not any specific mmap flag — which is why this PR bounds residency instead of tweaking flags.Design
tensor_expert_bytes(); when streaming is enabled it returns a pointer into the cache slab instead ofmap + offset. The matvec code is untouched and unaware.ds4_metal.m,ds4_cuda.cu,ds4_rocm.cu, ords4_ssd.c/h(CUDA already has its own streaming implementation; this follows that pattern —make cpubuilds don't compile the Metal file, so sharing its code isn't possible).ds4_backend_supports_ssd_streaming()is untouched; the CPU allowance is a sibling branch at the gate.WILLNEED. With streaming on, only non-routed tensor ranges get the prefetch hint.--ssd-streaming-preload-expertsis ignored on CPU (warned); on macOS,--cpuwithout--ssd-streamingnow prints an advisory warning (kept advisory on purpose — easy to drop or harden, your call).Testing (M4 Max 128 GB, macOS 26.5.1, DeepSeek-V4 Flash IQ2XXS 81 GiB — 43 MoE layers × 256 experts)
make(Metal),make cpu,make test(Metal): clean; the branch adds three self-tests (--cpu-stream-cache/-fetch/-parallel, no model needed).--cpu --ssd-streaming: 3 of 4 executable cases pass exactly (16 GiB cache, 8m19s; identical results at a 3 GiB cache forcing ~24× eviction pressure). The one failing case (short_code_completion) fails byte-for-byte identically on unmodified Metal with the same file — my test model is a community-modified ("abliterated") quant, and the divergence tracks the weights, not the backend (inferred from the cross-backend isolation;--metal-tensor-equivalencepasses strictly). Happy to rerun against a pristine quant if you want that in the record.CPU stays a reference/debug path — this makes it safe, not fast.
Full evidence bundle (commands, per-case tables, memory samples)
3. Correctness evidence
3.1 Self-tests (unit level, no model required)
From Tasks 1-3, re-confirmed in this task's build-matrix run (§6):
--cpu-stream-cache: victim selection (lowest use_count, LRU tie-break), pinning semantics, eviction bookkeeping, and the all-pinned/no-victim case.--cpu-stream-fetch: pread-based fetch/install against a synthetic temp file with a byte-patterned fixture, including hit/evict/refetch.--cpu-stream-parallel: 4-thread, 50-round, 100-iterations-per-thread concurrent fetch stress against a 5-triple budget forcing repeated cross-round eviction; asserts zero data corruption and that eviction was actually exercised (evictions > 0). Additionally run under ThreadSanitizer (-fsanitize=thread, separate throwaway build) with zero TSan diagnostics — confirms theg_cpu_stream_lockmutex added in5610610fully serializes the shared cache state with no unprotected access path. Also run 20x back-to-back with no flakiness (Task 3).3.2 Official logprob-vector fixture (
tests/test-vectors/official.vec, 5 cases)Run A — CPU + SSD-streaming, 16 GiB expert cache (2,427 experts cached of 11,008 total):
short_italian_factshort_code_completionshort_reasoning_plainlong_memory_archiveAPI/official graph mismatch, unrelated to backend)long_code_audit3 of 4 executable cases pass exactly. Wall clock 8m19s.
Disambiguation (run immediately per the controller's anti-tampering protocol, since the failure was a hard token-bytes mismatch, not a logprob-delta-over-tolerance): the same 5-vector fixture run on Metal, no CPU/streaming env, same model file:
An unrelated pre-existing
ds4-server --metalprocess (PID 5634, not started by this work) was holding the GPU during this run and caused spuriouskIOGPUCommandBufferCallbackErrorOutOfMemoryfailures on 2 of 5 cases (short_italian_fact,long_code_audit— both already passing on CPU, so this is not a concern). Crucially,short_code_completion— the one case that failed on CPU — ran on Metal with zero interleaved OOM errors, and its failure signature on Metal is byte-for-byte identical to the CPU failure:Verdict: Metal (unmodified by this branch) fails identically to CPU+streaming on this one case → the divergence is caused by the abliterated model's weights differing from the official DeepSeek API's weights (inferred from cross-backend isolation; official non-abliterated weights were not available to test directly), not by the CPU/SSD-streaming code. This is corroborated independently by Run C's
metal-tensor-equivalencesubtest (§3.4), a strict local logits comparison (not against the official API) that passes cleanly onshort_code_completion— Metal's own numerics are self-consistent; only the comparison against the official (non-abliterated) API vector diverges.3.3 Eviction-stress rerun (Run B — cache_gb=3)
Amended pass criterion (set by the controller after the disambiguation cleared the code): identical failure signature to Run A, no new failures, no crashes, no cache death.
short_italian_fact,short_reasoning_plain,long_code_audit), onlyshort_code_completionfails with the exact same 7 assertions,long_memory_archiveskips the same way. Grep forbudget too small|abort|crash|signal|panic|Insufficientover the whole log: zero hits.Conclusion: the CPU SSD-streaming cache survives heavy eviction (455 of 11,008 experts resident) with bit-identical decode behavior to a roomy 2,427-expert budget. The eviction path does not perturb correctness.
3.4 Full Metal regression (Run C)
DS4_TEST_MODEL="$MODEL" caffeinate -i make test— 18 subtests, 15 executed (3 skipped by design: no MTP head, noDS4_TEST_SSD_STREAMINGfor one CPU-only subtest). Only 2 of 15 fail, and both are the sameshort_code_completionroot cause described above:short_code_completion)short_code_completion:top1_mismatch=0 min_top5_overlap=5/5 min_overlap=20/20 worst_rank_delta=0 worst_rms=0)DS4_TEST_SSD_STREAMING)Wall clock 5m36s.
make testexits nonzero (Error 1) purely because of the model-caused divergence — not touched/worked around, per the controller's explicit anti-tampering instruction.4. Safety / mechanism evidence
4.1 The problem this branch makes safe (the kernel bug itself is Apple's to fix)
Before this branch, running the CPU backend on macOS with a model too large to fully fit resident (e.g. this 81 GiB model on a machine with 128 GB RAM, but with other memory pressure — including the concurrent unrelated
ds4-serverprocess observed on this very machine, which alone holds the full model Metal-resident plus a 60 GB on-disk KV cache) could fault the entire GGUF through a plainmmap. On current macOS/XNU kernels this has been observed to panic the kernel outright — this exact failure mode is documented in the repo's own README ("current macOS versions have a bug in the virtual memory implementation that will crash the kernel if you try to run the CPU code... each time you have to restart the computer") and corresponds to upstream issue #483, tracked with two independently-identified panic mechanisms against this project's own panic logs and the XNU source (xnu-12377.121.6): (1) watchdog starvation — 12 CPU threads all faulting pages from the same huge mmap saturate the kernel's page-fault path badly enough thatwatchdogdcannot check in within its timeout and the kernel panics; (2) VM map entry / zone exhaustion — the huge single mmap fragments into enough VM map entries under heavy concurrent fault pressure to exhaust a kernel zone. Both mechanisms are avoided by never mmap-faulting the routed-expert region at all.4.2 The fix's mechanism
--cpu --ssd-streamingon this branch never faults routed-expert weight ranges through mmap. Instead:cpu_stream_initreads expert geometry (offsets, sizes, counts) straight from the GGUF tensor directory (no page faults — this is metadata only); non-routed weights are still selectivelyWILLNEED-hinted (small, safe); routed-expert weight slabs arepread()directly into a bounded LRU+hotness cache of heap-allocated buffers, sized by an explicit flag or a default of 25% of physical RAM (always floored so one full layer's worth of experts fits). The cache is bounded by construction — its heap footprint cannot exceed the configured budget regardless of model size or generation length, because every fetch either hits an existing entry or evicts before installing a new one.4.3 Smoke test (Task 4, prior work — cited here for continuity)
8-token generation, 8 GiB cache, same 81 GB model:
Peak RSS 17.77 GiB, peak footprint 9.18 GiB — both far below the 81 GB model file. (RSS includes resident non-routed mmap pages and page-cache attribution; peak physical footprint is the bounded metric to compare against the cache budget.) Exit code 0, no crash, no panic. First-ever safe CPU run of this specific model on this machine (the old unprotected CPU mmap path is documented in-code as having triggered the kernel-level VM panic previously).
4.4 New centerpiece evidence: 500-token bounded-footprint run (this task)
Run concurrently with an independent memory-monitor loop sampling
vmmap --summary <pid> | grep -i "physical footprint"andsysctl vm.swapusageevery ~30s.Result: completed in full, exit clean, ~505 tokens generated (word-count-estimated; the run hit the requested
-n 500cap mid-sentence, which is the expected/correct behavior for a token-budget cutoff, not a crash). Total wall clock was under 2 minutes — much faster than the tens-of-minutes estimate in the task brief, because the OS page cache for this model's non-routed weights and popular experts was already warm from the Task 4/5 runs earlier in this same session on the same machine/model.Reported throughput (from ds4's own output):
Physical-footprint samples (only 3 fell within the run's short duration before the monitor's own exit-detection stopped it — the plateau is nonetheless clear and consistent with the 16 GiB cache budget plus fixed overhead):
Footprint is flat/bounded across the entire run — no monotonic growth, no drift — consistent with the streaming cache never exceeding its configured 16 GiB budget regardless of how many of the 11,008 routed experts get touched over the course of 500 generated tokens.
Swapfile count (
ls /System/Volumes/VM/ | grep -c swapfile): 32 before, 32 after — zero new swapfiles.vm.swapusagewas already near-saturated throughout (total = 32768.00M used = 32154.88M free = 613.12M) — this is a pre-existing condition from the unrelated, still-runningds4-server --metalprocess (PID 5634) on this shared machine (that process alone holds the 81 GB model Metal-resident plus a 60 GB on-disk KV cache directory), not something this run caused: swap usage was static across all samples during our run (32154.88Min every single sample, byte-for-byte), meaning our streaming run made zero net contribution to swap pressure.Exit status: 0 (process ran to completion and exited on its own; confirmed via
pgrepreturning no match for the ds4 PID immediately after the log's finalgeneration:stats line appeared, with no error/panic/signal text anywhere in the log).Win condition met: footprint bounded/plateaued (not monotonic growth), no swapfile explosion, clean exit, throughput numbers printed by ds4 itself.
4.5 Speed numbers (transparency only — CPU is a reference path, not the production target)
No separate
ds4-benchCPU run was performed — the CLI's own printed throughput from the generation run above is the reported number, per the task's "no separate bench needed for CPU" instruction.5. Metal non-regression
Metal before/after
ds4-benchsweep: SKIPPED. An unrelated user processds4-server --metal(PID 5634) was confirmed running and holding the GPU on this machine throughout this task (pgrep -fl ds4-serverreturns it; started well before this task, serving on port 8081 with a 100k-token context and a 60 GB on-disk KV cache directory). Running a Metal bench sweep concurrently would produce contaminated, non-comparable numbers (this exact contamination was observed and documented in Task 5's disambiguation run — spuriouskIOGPUCommandBufferCallbackErrorOutOfMemoryerrors). The unrelated server was not killed (out of scope; it is another user's/process's workload on a shared machine).Metal non-regression is instead evidenced by:
(a) Zero Metal source files touched.
ds4_metal.m,ds4_cuda.cu,ds4_rocm.cu, andds4_ssd.c/hare all untouched by this branch, confirmed both by the diffstat (they don't appear at all) and by an explicit grep for their names/extensions across the changed-file list. The one shared-code touchpoint,ds4_backend_supports_ssd_streaming, was read but not modified — the CPU allowance was added as a sibling branch in the calling gate, not a change to that predicate function, so GPU-backend auto-cache logic is provably unreachable from this diff.(b) Full
make testpass (Task 5 Run C, §3.4 above). 15 of 15 executed subtests pass except the two instances of the single model-causedshort_code_completiondivergence, which is independently confirmed to reproduce identically on Metal with zero CPU/streaming code involved. Every Metal-specific subtest (metal-short-prefill,metal-kernels,metal-tensor-equivalence,local-golden-vectors,tool-call-quality,think-tool-recovery,long-context,server) passes cleanly.metal-tensor-equivalencein particular is a strict local-reference logits comparison (not an official-API comparison) and is clean across all 5 test-vector cases, positively confirming Metal's numerics are unperturbed.Between (a) and (b), Metal non-regression is established without needing a live bench run on a GPU that is not actually free on this machine right now.
6. Build matrix
All three builds re-run clean from scratch in this task, on HEAD
340a7aa:Self-tests re-confirmed against this exact fresh build:
The 8
make cpuwarnings were separately verified in Task 4 (viagit stash+ rebuild) to be present identically on pre-branch HEAD — not introduced by this branch.7. Concerns / caveats carried into this evidence bundle
short_code_completion). This is fully isolated to the model weights via the Metal disambiguation (§3.2) and is not attributable to this branch.long_memory_archiveis skipped by the test harness itself (API/official graph mismatch) in every run, pre-existing and unrelated to this branch — reduces effective official-vector coverage to 4 of 5 cases per run.ds4-server --metal, PID 5634, left running). Non-regression is argued from the diffstat +make test, not a live A/B bench; a maintainer with a free GPU can run theds4-benchsweep from the commands appendix below to get an additional numeric comparison if desired.--ssd-streaming-cache-bytes(raw byte budget) and the preload-warning path were not exercised end-to-end in any task (only--ssd-streaming-cache-expertswas used); the preload-ignored-on-CPUfprintfwas verified by code inspection only. Considered low-risk (single unconditional print, no control-flow effect).ds4-server --metalprocess that holds significant GPU/unified-memory/swap pressure throughout this entire evidence-gathering session; it was never started or stopped by this work, and its presence is called out wherever it could plausibly have contaminated a result (Metal disambiguation OOMs in Task 5, and the pre-saturatedvm.swapusagebaseline in §4.4).8. Commands appendix (verbatim, for reproduction)
Self-tests:
Task 4 smoke test:
(wrapped in
/usr/bin/time -lfor RSS/footprint capture)Task 5 Run A (official logprob vectors, CPU+streaming, 16 GiB cache):
DS4_TEST_MODEL="$MODEL" DS4_TEST_BACKEND=cpu DS4_TEST_SSD_STREAMING=1 \ DS4_TEST_SSD_STREAMING_CACHE_GB=16 caffeinate -i ./ds4_test --logprob-vectorsTask 5 disambiguation (same vectors, Metal, no streaming env):
DS4_TEST_MODEL="$MODEL" ./ds4_test --logprob-vectorsTask 5 Run B (eviction stress, 3 GiB cache):
DS4_TEST_MODEL="$MODEL" DS4_TEST_BACKEND=cpu DS4_TEST_SSD_STREAMING=1 \ DS4_TEST_SSD_STREAMING_CACHE_GB=3 caffeinate -i ./ds4_test --logprob-vectorsTask 5 Run C (full Metal regression):
This task's centerpiece 500-token bounded-footprint run:
Environment check (run before any GPU-affecting step):
Build matrix:
Git evidence:
🤖 Generated with Claude Code
https://claude.ai/code/session_01A4PSPFYh6oBLsLWZsu3jB2