diff --git a/DEVLOG.md b/DEVLOG.md index 6734a1e..5aab79f 100644 --- a/DEVLOG.md +++ b/DEVLOG.md @@ -17,7 +17,7 @@ - Implemented vectorized sub-byte bit-packing runtimes in `quantbench.quant.packing`: - `int4_pack.py`: Vectorized bit-shift packing of two INT4 values (unsigned $[0, 15]$ or signed two's complement $[-8, 7]$) into a single `uint8` byte container: `byte = (v0 & 0x0F) | ((v1 & 0x0F) << 4)`. Full support for odd dimension length padding and exact shape restoration. - `ternary_pack.py`: BitNet 1.58b packing supporting both hardware-friendly 2-bit SIMD packing (4 weights/byte using biased offset $u = v + 1 \in \{0, 1, 2\}$) and theoretical maximal density Radix-3 base-3 packing (5 weights/byte via $3^5 = 243 < 256$). - - `sparse_pack.py`: NVIDIA Ampere/Ada/Hopper 2:4 structured sparsity layout. Automated row pruning, value condensation to $50\%$ storage ($M, N/2$), and 2-bit metadata column index packing into 4-bit nibbles ($M, N/8$). Decompression scatter preserves exact zeros and non-zero positions. + - `sparse_pack.py`: NVIDIA Ampere/Ada/Hopper 2:4 structured sparsity layout. Automated row pruning, value condensation to 50% storage ($M, N/2$), and 2-bit metadata column index packing into 4-bit nibbles ($M, N/8$). Decompression scatter preserves exact zeros and non-zero positions. - Added CLI command `pack-inspect` to visualize sub-byte bit layouts and theoretical memory compression directly from the terminal. - 22/22 unit tests passing with 93% test coverage in 2.18s. All Ruff checks passing. diff --git a/GUIDE.md b/GUIDE.md index d4e61e4..5f1e16e 100644 --- a/GUIDE.md +++ b/GUIDE.md @@ -59,7 +59,7 @@ Per-tensor quantization uses a single scale $s$ for the entire weight matrix. A Block-wise quantization slices the weight matrix into independent groups of size $G \in \{32, 64, 128\}$: * Each block $k$ computes its own independent scale $s_k$ and zero-point $z_k$. * **The Granularity Trade-off**: - * Smaller $G$ (e.g. 32): Higher precision, but scales consume $16 / 32 = 0.5$ bits/weight ($12.5\%$ metadata overhead). + * Smaller $G$ (e.g. 32): Higher precision, but scales consume $16 / 32 = 0.5$ bits/weight (12.5% metadata overhead). * Larger $G$ (e.g. 128): Lower metadata overhead ($16 / 128 = 0.125$ bits/weight), but slightly reduced ability to isolate local outliers. * In practice, **$G = 128$ or $G = 64$** represents the Pareto-optimal operating point for LLMs. diff --git a/README.md b/README.md index 3f6421b..c017a42 100644 --- a/README.md +++ b/README.md @@ -75,7 +75,7 @@ Bit-exact simulation of **E4M3FN** (higher mantissa precision, range $\pm 448$, Because hardware cannot address sub-byte boundaries, weights are packed into vectorized `uint8` containers: * **INT4**: Two 4-bit nibbles packed per byte: `(v0 & 0x0F) | ((v1 & 0x0F) << 4)` ($4\times$ memory compression vs FP16). * **BitNet Ternary**: Four $\{-1, 0, 1\}$ weights per byte using offset binary mapping ($8\times$ compression vs FP16). -* **2:4 Structured Sparsity**: Two non-zero values retained per 4-element block ($50\%$ storage) with 2-bit column positions packed into 4-bit metadata nibbles for Ampere/Ada/Hopper Sparse Tensor Cores ($2\times$ TFLOPS). +* **2:4 Structured Sparsity**: Two non-zero values retained per 4-element block (50% storage) with 2-bit column positions packed into 4-bit metadata nibbles for Ampere/Ada/Hopper Sparse Tensor Cores ($2\times$ TFLOPS). --- @@ -205,7 +205,7 @@ The repository includes runnable end-to-end scripts demonstrating drop-in infere * **[examples/01_quant_linear_inference.py](examples/01_quant_linear_inference.py)**: Demonstrates converting PyTorch `nn.Linear` layers into INT4 blockwise, BitNet 1.58b ternary, and Ampere 2:4 sparse `QuantLinear` modules, verifying forward pass execution and recursive transformer conversion with `lm_head` float exclusion. * **[examples/02_advanced_calibration.py](examples/02_advanced_calibration.py)**: Step-by-step demonstration of SmoothQuant outlier migration (proving exact mathematical continuous invariance), AWQ Frobenius grid search for salient channel protection, and GPTQ second-order Hessian Cholesky damping updates yielding $1.48\times$ error reduction. -* **[examples/03_servebench_roofline_eval.py](examples/03_servebench_roofline_eval.py)**: Bridges `quantbench` drop-in layers with `servebench`'s hardware roofline model (`HARDWARE_PROFILES`, RTX 3050 192 GB/s, T4 320 GB/s), calculating arithmetic intensity $I = \frac{2}{\text{bytes\_per\_param}}$ FLOPs/byte, decode token latency, and DRAM bandwidth traffic reduction (slashing memory traffic by $73.4\%$ for INT4). +* **[examples/03_servebench_roofline_eval.py](examples/03_servebench_roofline_eval.py)**: Bridges `quantbench` drop-in layers with `servebench`'s hardware roofline model (`HARDWARE_PROFILES`, RTX 3050 192 GB/s, T4 320 GB/s), calculating arithmetic intensity $I = \frac{2}{\text{bytes\_per\_param}}$ FLOPs/byte, decode token latency, and DRAM bandwidth traffic reduction (slashing memory traffic by 73.4% for INT4). ```bash uv run python examples/01_quant_linear_inference.py