Repository navigation
Replies: 1 comment
|
The answer depends on what those files actually are, so here is how to tell them apart and the loading path for each case. First, check the transformer's from diffusers import FluxTransformer2DModel # or whatever class config.json declares
transformer = FluxTransformer2DModel.from_pretrained("org/repo", subfolder="transformer")If the files are diffusers "variants", the naming convention is transformer = SomeModel.from_pretrained("org/repo", subfolder="transformer", variant="int4")This only works when the filename literally follows that convention — a file called just If
from diffusers import FluxTransformer2DModel
from transformers import BitsAndBytesConfig
import torch
quant = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16)
transformer = FluxTransformer2DModel.from_pretrained(
"org/repo", subfolder="transformer",
quantization_config=quant, torch_dtype=torch.bfloat16
)(
from safetensors import safe_open
with safe_open("int4.safetensors", framework="pt") as f:
keys = list(f.keys())
print(keys[:5], f.get_slice(keys[0]).get_dtype())Packed 4-bit dtypes (e.g. uint8 tensors with a About the Practical shortcut: if your goal is just "run this model in less VRAM", option 1 (quantize on load) is almost always the right answer — it works with the standard checkpoint and needs no knowledge of what the int4 file actually contains. |
Uh oh!
There was an error while loading. Please reload this page.
I have downloaded an entire repository of one model and there are many files under transformer folder.


What is the difference among them and I wonder how to load the int4.safetensors one as shown in the following figure if using the built-in 'from_pretrained' method.
If it is a '.pth' model as shown in the following,
then how to load it using the built-in 'from_pretrained' method or should we change with other method?
All reactions