Home

engram-parser

Pure-Rust GGUF + Safetensors checkpoint/tensor substrate: format-independent tensor inventory, metadata, and raw payload access for ANN/SNN research.

Version: 0.3.0 | Edition: 2024 | MSRV: 1.97.1 | Dev toolchain: 1.99.0 | License: MIT OR Apache-2.0

An engram is a physical trace of memory. This crate opens frozen checkpoints and exposes tensors as raw byte buffers with shape/dtype metadata — no neural-network math. MoE per-expert extraction is a GGUF model-analysis specialization under analysis::moe, not the identity of the crate.

Default builds are GGUF + Checkpoint API with zero dependencies (fs::read into Vec<u8>). Two optional cargo features extend it:

• safetensors (off by default): upstream safetensors crate for canonical header validation, plus engram-owned manifests, discovery, and raw tensor payload access
• mmap (off by default): memmap2 — load_gguf_mmap / open_checkpoint_mmap and (with safetensors) borrowed mmap-backed Safetensors payloads

What It Does

• Checkpoint API: open_checkpoint → Checkpoint / AnyCheckpoint — inventory, metadata, raw tensor_bytes across GGUF and Safetensors
• Parse GGUF v3 (magic, header, KV metadata, tensor directory) → GgufLayout, with documented ParseLimits budgets
• Enumerate MoE experts and extract gate/up/down raw bytes via analysis::moe (compat re-exports under moe and crate root)
• Support stacked (ffn_*_exps.weight) and per-expert (ffn_*.E.weight) layouts
• Packed dequant to f32 for Q8_0, Q5_K, Q6_K, IQ3_M block layout
• Safetensors: inspect → SafetensorsManifest; open_safetensors_checkpoint / _mmap for bounded or borrowed payloads

What It Does Not Do

• No matmul, forward, routing, softmax
• No CUDA / GPU / SIMD
• Zero crate dependencies on the default path ([dependencies] empty; optional memmap2 / upstream safetensors under features)
• No tokenization or SNN dynamics
• No full model-family routing policy (see cortex-tensor)
• No Hugging Face config.json policy

Start Here

Page	Description
Getting Started	Install, Checkpoint, load, extract
Checkpoint	Format-independent open_checkpoint API
Project Structure	Modules and tree
Architecture	Ownership and non-goals
GGUF Parsing	load_gguf, limits, layout, dequant
Tensor and DType	Directory entries and types
MoE Extraction	analysis::moe — stacked vs per-expert
Safetensors	Feature-gated inspection + payload access
Errors	ParserError
Public API	Crate surface
Overview	Wiki map
Testing	Smoke, parity, golden, pilots
CI and Quality	GHA, Codecov, Qlty
MSRV	1.97.1 policy / 1.99.0 pin
Ecosystem	cortex-tensor, corinth-canal
Glossary	Terms

Quick Commands

cargo fmt --check
cargo clippy --all-targets --all-features --locked -- -D warnings
cargo build --all-features --locked
cargo test --all-features --locked

License

Dual-licensed under MIT or Apache-2.0 at your option.

---

Last updated: October 14, 2026 Updated by: Wiki Keeper (Grok Bot) Package tip reference: dbc46cf (main)