Skip to content

LTX-2.3 in float16: text-connector RMS norm overflows, pipeline silently returns an all-NaN video #15281

Open

Description

@AuthRan

Describe the bug

If you run LTX2Pipeline with diffusers/LTX-2.3-distilled-Diffusers in torch.float16, it finishes with no error or warning, but every output frame is NaN, so the exported mp4 is all black. I hit this in a real project on a Kaggle T4, where fp16 is the obvious choice because sm_75 has no native bf16.

Root cause, as far as I can tell: per_token_rms_norm in condition_embedder_ltx2.py computes mean(x**2) in the input dtype. The stacked Gemma-3 hidden states it normalizes reach |x| = 798,720 (measured), which is far above the fp16 max of 65,504:

  • 372 of the 192.7M embedding values turn into inf when cast to fp16, and they become NaN after the norm.
  • Any value above about 256 overflows x**2, so its whole (token, layer) slice is silently zeroed.

I put forward hooks on every module. In fp16, the first non-finite output is connectors.video_text_proj_in, which is the first module after the norm. In bf16, no module output anywhere (connectors, DiT, VAE) goes above 65,504: the peak is 45,824 in connectors.video_connector.transformer_blocks.7, and the DiT peaks at 23.8k.

per_layer_masked_mean_norm (the LTX-2.0 path) has the same pattern: its sum, mean, min and max all run in the input dtype. I haven't run LTX-2.0, though.

What I tried: I computed per_token_rms_norm in fp32, cast the result to fp16, and passed the embeddings in fp32. Same seed and settings in both runs:

bf16 fp16 + fp32 norm
pipe() wall time (T4, 8 steps, 576x320, 25 frames) 326.5 s 79.3 s
NaN in output no no
module outputs > 65,504 0 0
frame std / mean 0.254 / 0.38 0.127 / 0.49

The NaNs are gone and it runs 4.1x faster. But the frames come out washed out: it's the same scene layout, only greyish, with half the contrast of bf16. So there's at least one more fp16 precision problem further down the pipeline, and fp16 isn't usable yet even with this change.

Suggestion:

  1. Upcast to fp32 inside per_token_rms_norm and per_layer_masked_mean_norm (the usual RMSNorm practice) and cast back after normalizing. That removes the NaN.
  2. Until fp16 output is correct, add a logger.warning or a note in ltx2.md saying LTX-2.x needs bf16 or fp32.

I'm happy to open a PR for 1 and/or 2 once you confirm the approach. If it helps, I can also narrow down which component causes the washed-out output (for example by running the VAE in fp32).

Reproduction

Minimal version, runs on CPU with no download:

import torch
from diffusers.models.condition_embedders.condition_embedder_ltx2 import per_token_rms_norm

torch.manual_seed(0)
x = torch.randn(1, 4, 3840, 49)   # [batch, seq, caption_channels, layers], like the stacked Gemma-3 states
x[0, 1, 7, 10] = 300.0            # 300**2 > 65504
x[0, 2, 7, 10] = 798720.0         # the peak measured on the real embeddings
ref = per_token_rms_norm(x.double()).float()
out = per_token_rms_norm(x.half()).float()
for t in range(4):
    print(t, f"{ref[0, t].abs().max().item():.3f}", f"{out[0, t].abs().nan_to_num(-1).max().item():.3f}", out[0, t].isnan().any().item())
# token  fp64 ref  fp16    nan
# 0       4.612    4.613  False
# 1      60.688    4.715  False   <- the 300.0 entry's (token, layer) slice is zeroed
# 2      61.968    4.320  True    <- NaN
# 3       4.541    4.543  False

End-to-end run (T4, NF4 transformer + connectors, enable_model_cpu_offload(), VAE tiling, 8 distilled steps). It passes guidance_scale=1.0, audio_guidance_scale=1.0; the second one is needed because of #14649, otherwise a pipeline without a text encoder crashes on the negative prompt.

find_overflow.py (--dtype bf16 | fp16 | fp16fix)
"""Localize where LTX-2.3 overflows fp16. Run on a Kaggle T4 (needs diffusers from git).

  python find_overflow.py --dtype bf16   # works; logs modules whose |activation| > 65504 (fp16 max)
  python find_overflow.py --dtype fp16   # reproduces the bug; logs the FIRST module that emits inf/NaN

Paste both outputs into the issue. Same model + settings as the Navriti pipeline.
"""
import argparse, gc, os
os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
import numpy as np
import torch
import diffusers
from diffusers import BitsAndBytesConfig, LTX2Pipeline, LTX2VideoTransformer3DModel
from diffusers.pipelines.ltx2.connectors import LTX2TextConnectors
from diffusers.pipelines.ltx2.utils import DISTILLED_SIGMA_VALUES
from transformers import BitsAndBytesConfig as TBnb, Gemma3ForConditionalGeneration

REPO = "diffusers/LTX-2.3-distilled-Diffusers"
FP16_MAX = 65504.0
PROMPT = "A flat vector illustration of a padlock closing over a web browser window, clean shapes, soft colors."

ap = argparse.ArgumentParser()
ap.add_argument("--dtype", choices=["bf16", "fp16", "fp16fix"], required=True)
ap.add_argument("--nohooks", action="store_true")  # for clean timing
ap.add_argument("--frames", type=int, default=25)  # 8n+1; the bug did not depend on length
args = ap.parse_args()
dtype = torch.bfloat16 if args.dtype == "bf16" else torch.float16
if args.dtype == "fp16fix":  # candidate fix: RMS-norm the raw Gemma states in fp32, then cast to the model dtype
    import diffusers.models.condition_embedders.condition_embedder_ltx2 as C
    def per_token_rms_norm_fp32(x, eps=1e-6):
        x32 = x.float()
        return (x32 * torch.rsqrt(x32.pow(2).mean(dim=2, keepdim=True) + eps)).to(torch.float16)
    C.per_token_rms_norm = per_token_rms_norm_fp32
print(f"diffusers {diffusers.__version__} | torch {torch.__version__} | {torch.cuda.get_device_name()} | {dtype}")

# Phase 1: text encoding (always bf16, as in the real pipeline), then free the encoder.
te = Gemma3ForConditionalGeneration.from_pretrained(
    REPO, subfolder="text_encoder", torch_dtype=torch.bfloat16,
    quantization_config=TBnb(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16))
p = LTX2Pipeline.from_pretrained(REPO, text_encoder=te, transformer=None, vae=None, audio_vae=None,
                                 connectors=None, vocoder=None, torch_dtype=torch.bfloat16)
with torch.no_grad():
    pe, mask, _, _ = p.encode_prompt(prompt=PROMPT, do_classifier_free_guidance=False, device="cuda")
print(f"prompt_embeds max|x| = {pe.abs().max().item():.1f}")
pe16 = pe.to(torch.float16)
print(f"after .to(float16): inf={torch.isinf(pe16).sum().item()} of {pe16.numel()} | shape={tuple(pe.shape)}")
del pe16
del p, te; gc.collect(); torch.cuda.empty_cache()

# Phase 2: generation in the dtype under test.
nf4 = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=dtype, bnb_4bit_use_double_quant=True)
transformer = LTX2VideoTransformer3DModel.from_pretrained(REPO, subfolder="transformer", quantization_config=nf4, torch_dtype=dtype)
connectors = LTX2TextConnectors.from_pretrained(REPO, subfolder="connectors", quantization_config=nf4, torch_dtype=dtype)
pipe = LTX2Pipeline.from_pretrained(REPO, transformer=transformer, connectors=connectors, text_encoder=None, torch_dtype=dtype)
pipe.enable_model_cpu_offload()
pipe.vae.enable_tiling()

peak = {}        # module -> running max |output| (GPU tensor, no sync per call)
first_bad = []   # fp16 mode: first module whose output is non-finite
handles = []

def tensors(o):
    if torch.is_tensor(o): yield o
    elif isinstance(o, (tuple, list)):
        for x in o: yield from tensors(x)
    elif hasattr(o, "sample"): yield from tensors(o.sample)

def hook(name):
    def fn(_m, _inp, out):
        for t in tensors(out):
            if not t.is_floating_point() or t.numel() == 0: continue
            m = t.detach().abs().amax().float()
            peak[name] = torch.maximum(peak[name], m) if name in peak else m
            if dtype == torch.float16 and not first_bad and not torch.isfinite(t).all():
                first_bad.append(name)
                print(f"FIRST NON-FINITE OUTPUT: {name}")
    return fn

for comp_name in ((), ("connectors", "transformer", "vae"))[not args.nohooks]:
    for n, m in getattr(pipe, comp_name).named_modules():
        handles.append(m.register_forward_hook(hook(f"{comp_name}.{n}" if n else comp_name)))

import time
pe_in = pe.to("cuda", torch.float32) if args.dtype == "fp16fix" else pe.to("cuda", dtype)
torch.cuda.synchronize(); t0 = time.time()
with torch.no_grad():
    video, _ = pipe(prompt_embeds=pe_in, prompt_attention_mask=mask.to("cuda"),
                    width=576, height=320, num_frames=args.frames, frame_rate=24, num_inference_steps=8,
                    sigmas=DISTILLED_SIGMA_VALUES, guidance_scale=1.0, audio_guidance_scale=1.0,  # see diffusers#14649
                    generator=torch.Generator("cuda").manual_seed(0), output_type="np", return_dict=False)
torch.cuda.synchronize(); print(f"pipe() wall time: {time.time() - t0:.1f} s (hooks={not args.nohooks})")
for h in handles: h.remove()

v = np.asarray(video[0])
print(f"output: nan={np.isnan(v).any()} min={np.nanmin(v):.3f} max={np.nanmax(v):.3f}")
rows = sorted(((k, t.item()) for k, t in peak.items()), key=lambda kv: -kv[1] if np.isfinite(kv[1]) else -np.inf)
over = [(k, x) for k, x in rows if not np.isfinite(x) or x > FP16_MAX]
print(f"\nmodules with max|output| > fp16 max ({FP16_MAX:.0f}) or non-finite: {len(over)}")
for k, x in over[:30]: print(f"  {x:>14.1f}  {k}")
print("\ntop 10 overall:")
for k, x in rows[:10]: print(f"  {x:>14.1f}  {k}")

Logs

# --dtype fp16
prompt_embeds max|x| = 798720.0
after .to(float16): inf=372 of 192675840 | shape=(1, 1024, 188160)
FIRST NON-FINITE OUTPUT: connectors.video_text_proj_in
output: nan=True min=nan max=nan

# --dtype bf16
output: nan=False min=0.000 max=1.000
modules with max|output| > fp16 max (65504) or non-finite: 0

Context

During an ML internship I built a text-to-video explainer pipeline: LangGraph scene planning, then LTX-2.3 on a free Kaggle T4, then TTS and ffmpeg. The fp16 run "succeeded" end to end and still produced a 100% black video. I only noticed after measuring the pixel values, and I worked around it with bf16 plus a NaN guard. I came back to localize the cause. I used an AI coding assistant (Claude Code) to help write the diagnostic script; the runs and numbers come from my own Kaggle notebook.

System Info

- 🤗 Diffusers version: 0.43.0.dev0 (main @ 843f1743e259)
- Platform: Linux-6.18.55+-x86_64-with-glibc2.39 (Kaggle; diffusers-cli reports "Colab: Yes")
- Python version: 3.13.16
- PyTorch version (GPU?): 2.11.0+cu130 (True)
- Huggingface_hub version: 1.33.0
- Transformers version: 5.22.2
- Accelerate version: 1.15.0
- bitsandbytes version: 0.50.2
- Safetensors version: 0.8.0
- Accelerator: Tesla T4, 15360 MiB
- Using GPU in script?: yes, one T4
- Using distributed or parallel set-up in script?: no

Who can help?

@dg845 @DN6

Activity

  1. 1OF1Research commented on Nov 14, 2026

    @1OF1Research

    Independent measurement of the shipped function itself (diffusers 0.43.0, torch 2.14.0+cpu, CPU), importing per_token_rms_norm from condition_embedder_ltx2.py exactly as shipped:

    fp16 token (constant value = its RMS) shipped output fp32-upcast output
    100 1.0 1.0
    200 1.0 1.0
    255 1.0 1.0
    257 0.0 1.0
    300 0.0 1.0
    1000 0.0 1.0

    The boundary is exactly where the report puts it: a token's mean of squares crosses the fp16 maximum of 65,504 at RMS ≈ 255.94, the variance becomes inf, and rsqrt turns the whole token to zero — silently, with no error and no NaN at this site. In a mixed tensor (quiet tokens at 10, one token at 300), only the hot token is zeroed; the rest norm correctly. And it is not only a cliff edge: random tokens at RMS 240 already come out completely wrong (max difference 4.13 against the fp32-upcast form), because individual elements above ~256 overflow their squares and take the token with them.

    This matches the minimal repro in the report — the 300.0 entry's slice zeroed, the large entry NaN downstream — and the proposed fp32 upcast inside the norm returns the reference value in every row above, which gives the fix a direct regression target.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions