You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support mixed-type quantization within a single quantization backend #15320
Other tools like ComfyUI and Unsloth already support mixed-type quants: different layers or modules of a model are quantized with different schemes or precisions. Some layers stay in high precision, and others drop to lower bit-widths. This usually gets a better quality/memory trade-off than quantizing every layer the same way.
Most of the quantization backends Diffusers integrates with already support this natively. Diffusers should let users use that capability.
Proposal
Allow mixed quantization types within a single backend (for example, different torchao or bnb configs per module/layer pattern), when the underlying backend framework supports it.
Out of scope for now: mixing backends in one checkpoint (for example, some layers bnb and others torchao). Not all backends are feature-complete, so this would be messy. We also haven't seen such checkpoints in practice.
Open questions
API design: how users specify per-module/per-layer quant configs (pattern-based mapping, modules_to_not_convert-style extension, etc.).
Serialization and loading of mixed-quant checkpoints, plus compatibility with existing pre-quantized checkpoints (for example, from the ComfyUI / Unsloth ecosystems).
Which backends support this today, and what's missing in each.
Benchmarks / guidance
As part of this work, produce:
Benchmarks (memory, latency) comparing uniform vs. mixed quantization on representative pipelines.
Qualitative comparisons of outputs.
Better-grounded heuristics for choosing which layers to keep in higher precision. Existing community guides give heuristics, but they're hit and miss.
Activity