Repository navigation
Description
Describe the bug
LTX-2.5 prompt embeddings are NaN or wrong (varying from run to run) from the second encode_prompt call on when the text encoder uses leaf_level group offloading with use_stream=True and offload_to_disk_path. use_stream=False, or offloading to CPU memory, gives identical results on every call.
The text encoder's rotary_emb (Gemma 4) has no parameters, only inv_freq buffers, and is called twice in a row per forward. With streams, the preceding group prefetches it once, before its first call, and it is offloaded after that call; nothing onloads it again before the second call. The synchronous fallback onload in GroupOffloadingHook.pre_forward only checks parameters, so a buffer-only group never takes it. Disk offloading replaces offloaded tensors with torch.empty_like(...), and the rotary embedding moves inv_freq to the input device itself, so the second call silently reads uninitialized memory. CPU-memory offloading keeps the real values, which hides the problem there.
I have a fix (include buffers in that fallback check, after parameters) with a regression test and can open a PR.
Reproduction
The repo is gated; this loads only the text encoder:
import tempfile
import torch
from diffusers import LTX2Pipeline
from diffusers.hooks import apply_group_offloading
pipe = LTX2Pipeline.from_pretrained(
"Lightricks/LTX-2.5-Diffusers", dtype=torch.bfloat16, transformer=None, vae=None, audio_vae=None, vocoder=None
)
apply_group_offloading(
pipe.text_encoder,
onload_device=torch.device("cuda"),
offload_device=torch.device("cpu"),
offload_type="leaf_level",
use_stream=True,
offload_to_disk_path=tempfile.mkdtemp(),
)
prompt = "A red fox walking through a snowy forest at dawn."
with torch.no_grad():
ref = pipe.encode_prompt(prompt, do_classifier_free_guidance=False, device="cuda")[0]
for i in range(2, 5):
emb = pipe.encode_prompt(prompt, do_classifier_free_guidance=False, device="cuda")[0]
print(f"call {i}: nan={emb.isnan().sum().item()} equal_to_call_1={torch.equal(emb, ref)}")Minimal case without the model (a buffer-only module called twice in a row):
import tempfile, torch
from diffusers.hooks import apply_group_offloading
class Scale(torch.nn.Module):
def __init__(self):
super().__init__()
self.register_buffer("s", torch.full((64,), 2.0))
def forward(self, x):
return x * self.s.to(x.device)
class M(torch.nn.Module):
def __init__(self):
super().__init__()
self.a = torch.nn.Linear(64, 64)
self.scale = Scale()
self.b = torch.nn.Linear(64, 64)
def forward(self, x):
return self.b(self.scale(self.scale(self.a(x))))
torch.manual_seed(0)
m = M().eval()
x = torch.randn(4, 64)
with torch.no_grad():
ref = m.to("cuda")(x.cuda())
m.to("cpu")
apply_group_offloading(m, onload_device=torch.device("cuda"), offload_device=torch.device("cpu"),
offload_type="leaf_level", use_stream=True, offload_to_disk_path=tempfile.mkdtemp())
print([torch.equal(m(x.cuda()), ref) for _ in range(3)]) # [True, False, False]Logs
# LTX-2.5 repro, two runs on main
call 2: nan=0 equal_to_call_1=False
call 3: nan=0 equal_to_call_1=False
call 4: nan=169082880 equal_to_call_1=False
call 2: nan=188743680 equal_to_call_1=False
call 3: nan=0 equal_to_call_1=False
call 4: nan=0 equal_to_call_1=FalseWith the fix all three print nan=0 equal_to_call_1=True. Also reproduces on 0.42.1 and with torch 2.16.0a0+rocm10.3.0a20261027.
System Info
- Diffusers version: 0.44.0.dev0 (main, e84bda1)
- Platform: Linux-7.0.0-42-generic-x86_64-with-glibc2.39
- Python version: 3.12.10
- PyTorch version (GPU?): 2.16.0a0+rocm10.3.0a20261115 (True)
- Huggingface_hub version: 1.33.0
- Transformers version: 5.21.0
- Accelerate version: 1.9.0
- Safetensors version: 0.8.0
- GPU: AMD Radeon AI PRO R9700
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no
Who can help?
No response
Activity