Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
- Implemented vectorized sub-byte bit-packing runtimes in `quantbench.quant.packing`:
- `int4_pack.py`: Vectorized bit-shift packing of two INT4 values (unsigned $[0, 15]$ or signed two's complement $[-8, 7]$) into a single `uint8` byte container: `byte = (v0 & 0x0F) | ((v1 & 0x0F) << 4)`. Full support for odd dimension length padding and exact shape restoration.
- `ternary_pack.py`: BitNet 1.58b packing supporting both hardware-friendly 2-bit SIMD packing (4 weights/byte using biased offset $u = v + 1 \in \{0, 1, 2\}$) and theoretical maximal density Radix-3 base-3 packing (5 weights/byte via $3^5 = 243 < 256$).
- `sparse_pack.py`: NVIDIA Ampere/Ada/Hopper 2:4 structured sparsity layout. Automated row pruning, value condensation to $50\%$ storage ($M, N/2$), and 2-bit metadata column index packing into 4-bit nibbles ($M, N/8$). Decompression scatter preserves exact zeros and non-zero positions.
- `sparse_pack.py`: NVIDIA Ampere/Ada/Hopper 2:4 structured sparsity layout. Automated row pruning, value condensation to 50% storage ($M, N/2$), and 2-bit metadata column index packing into 4-bit nibbles ($M, N/8$). Decompression scatter preserves exact zeros and non-zero positions.
- Added CLI command `pack-inspect` to visualize sub-byte bit layouts and theoretical memory compression directly from the terminal.
- 22/22 unit tests passing with 93% test coverage in 2.18s. All Ruff checks passing.

Expand Down
2 changes: 1 addition & 1 deletion GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ Per-tensor quantization uses a single scale $s$ for the entire weight matrix. A
Block-wise quantization slices the weight matrix into independent groups of size $G \in \{32, 64, 128\}$:
* Each block $k$ computes its own independent scale $s_k$ and zero-point $z_k$.
* **The Granularity Trade-off**:
* Smaller $G$ (e.g. 32): Higher precision, but scales consume $16 / 32 = 0.5$ bits/weight ($12.5\%$ metadata overhead).
* Smaller $G$ (e.g. 32): Higher precision, but scales consume $16 / 32 = 0.5$ bits/weight (12.5% metadata overhead).
* Larger $G$ (e.g. 128): Lower metadata overhead ($16 / 128 = 0.125$ bits/weight), but slightly reduced ability to isolate local outliers.
* In practice, **$G = 128$ or $G = 64$** represents the Pareto-optimal operating point for LLMs.

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ Bit-exact simulation of **E4M3FN** (higher mantissa precision, range $\pm 448$,
Because hardware cannot address sub-byte boundaries, weights are packed into vectorized `uint8` containers:
* **INT4**: Two 4-bit nibbles packed per byte: `(v0 & 0x0F) | ((v1 & 0x0F) << 4)` ($4\times$ memory compression vs FP16).
* **BitNet Ternary**: Four $\{-1, 0, 1\}$ weights per byte using offset binary mapping ($8\times$ compression vs FP16).
* **2:4 Structured Sparsity**: Two non-zero values retained per 4-element block ($50\%$ storage) with 2-bit column positions packed into 4-bit metadata nibbles for Ampere/Ada/Hopper Sparse Tensor Cores ($2\times$ TFLOPS).
* **2:4 Structured Sparsity**: Two non-zero values retained per 4-element block (50% storage) with 2-bit column positions packed into 4-bit metadata nibbles for Ampere/Ada/Hopper Sparse Tensor Cores ($2\times$ TFLOPS).

---

Expand Down Expand Up @@ -205,7 +205,7 @@ The repository includes runnable end-to-end scripts demonstrating drop-in infere

* **[examples/01_quant_linear_inference.py](examples/01_quant_linear_inference.py)**: Demonstrates converting PyTorch `nn.Linear` layers into INT4 blockwise, BitNet 1.58b ternary, and Ampere 2:4 sparse `QuantLinear` modules, verifying forward pass execution and recursive transformer conversion with `lm_head` float exclusion.
* **[examples/02_advanced_calibration.py](examples/02_advanced_calibration.py)**: Step-by-step demonstration of SmoothQuant outlier migration (proving exact mathematical continuous invariance), AWQ Frobenius grid search for salient channel protection, and GPTQ second-order Hessian Cholesky damping updates yielding $1.48\times$ error reduction.
* **[examples/03_servebench_roofline_eval.py](examples/03_servebench_roofline_eval.py)**: Bridges `quantbench` drop-in layers with `servebench`'s hardware roofline model (`HARDWARE_PROFILES`, RTX 3050 192 GB/s, T4 320 GB/s), calculating arithmetic intensity $I = \frac{2}{\text{bytes\_per\_param}}$ FLOPs/byte, decode token latency, and DRAM bandwidth traffic reduction (slashing memory traffic by $73.4\%$ for INT4).
* **[examples/03_servebench_roofline_eval.py](examples/03_servebench_roofline_eval.py)**: Bridges `quantbench` drop-in layers with `servebench`'s hardware roofline model (`HARDWARE_PROFILES`, RTX 3050 192 GB/s, T4 320 GB/s), calculating arithmetic intensity $I = \frac{2}{\text{bytes\_per\_param}}$ FLOPs/byte, decode token latency, and DRAM bandwidth traffic reduction (slashing memory traffic by 73.4% for INT4).

```bash
uv run python examples/01_quant_linear_inference.py
Expand Down
Loading