Skip to content

Group offloading with use_stream=True and offload_to_disk_path gives wrong output for a buffer-only module called twice in a row #15329

Open

Description

@klin2024

Describe the bug

LTX-2.5 prompt embeddings are NaN or wrong (varying from run to run) from the second encode_prompt call on when the text encoder uses leaf_level group offloading with use_stream=True and offload_to_disk_path. use_stream=False, or offloading to CPU memory, gives identical results on every call.

The text encoder's rotary_emb (Gemma 4) has no parameters, only inv_freq buffers, and is called twice in a row per forward. With streams, the preceding group prefetches it once, before its first call, and it is offloaded after that call; nothing onloads it again before the second call. The synchronous fallback onload in GroupOffloadingHook.pre_forward only checks parameters, so a buffer-only group never takes it. Disk offloading replaces offloaded tensors with torch.empty_like(...), and the rotary embedding moves inv_freq to the input device itself, so the second call silently reads uninitialized memory. CPU-memory offloading keeps the real values, which hides the problem there.

I have a fix (include buffers in that fallback check, after parameters) with a regression test and can open a PR.

Reproduction

The repo is gated; this loads only the text encoder:

import tempfile
import torch
from diffusers import LTX2Pipeline
from diffusers.hooks import apply_group_offloading

pipe = LTX2Pipeline.from_pretrained(
    "Lightricks/LTX-2.5-Diffusers", dtype=torch.bfloat16, transformer=None, vae=None, audio_vae=None, vocoder=None
)
apply_group_offloading(
    pipe.text_encoder,
    onload_device=torch.device("cuda"),
    offload_device=torch.device("cpu"),
    offload_type="leaf_level",
    use_stream=True,
    offload_to_disk_path=tempfile.mkdtemp(),
)

prompt = "A red fox walking through a snowy forest at dawn."
with torch.no_grad():
    ref = pipe.encode_prompt(prompt, do_classifier_free_guidance=False, device="cuda")[0]
    for i in range(2, 5):
        emb = pipe.encode_prompt(prompt, do_classifier_free_guidance=False, device="cuda")[0]
        print(f"call {i}: nan={emb.isnan().sum().item()} equal_to_call_1={torch.equal(emb, ref)}")

Minimal case without the model (a buffer-only module called twice in a row):

import tempfile, torch
from diffusers.hooks import apply_group_offloading

class Scale(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.register_buffer("s", torch.full((64,), 2.0))
    def forward(self, x):
        return x * self.s.to(x.device)

class M(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.a = torch.nn.Linear(64, 64)
        self.scale = Scale()
        self.b = torch.nn.Linear(64, 64)
    def forward(self, x):
        return self.b(self.scale(self.scale(self.a(x))))

torch.manual_seed(0)
m = M().eval()
x = torch.randn(4, 64)
with torch.no_grad():
    ref = m.to("cuda")(x.cuda())
    m.to("cpu")
    apply_group_offloading(m, onload_device=torch.device("cuda"), offload_device=torch.device("cpu"),
                           offload_type="leaf_level", use_stream=True, offload_to_disk_path=tempfile.mkdtemp())
    print([torch.equal(m(x.cuda()), ref) for _ in range(3)])  # [True, False, False]

Logs

# LTX-2.5 repro, two runs on main
call 2: nan=0 equal_to_call_1=False
call 3: nan=0 equal_to_call_1=False
call 4: nan=169082880 equal_to_call_1=False

call 2: nan=188743680 equal_to_call_1=False
call 3: nan=0 equal_to_call_1=False
call 4: nan=0 equal_to_call_1=False

With the fix all three print nan=0 equal_to_call_1=True. Also reproduces on 0.42.1 and with torch 2.16.0a0+rocm10.3.0a20261027.

System Info

  • Diffusers version: 0.44.0.dev0 (main, e84bda1)
  • Platform: Linux-7.0.0-42-generic-x86_64-with-glibc2.39
  • Python version: 3.12.10
  • PyTorch version (GPU?): 2.16.0a0+rocm10.3.0a20261115 (True)
  • Huggingface_hub version: 1.33.0
  • Transformers version: 5.21.0
  • Accelerate version: 1.9.0
  • Safetensors version: 0.8.0
  • GPU: AMD Radeon AI PRO R9700
  • Using GPU in script?: yes
  • Using distributed or parallel set-up in script?: no

Who can help?

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions