Repository navigation
Description
Describe the bug
If you run LTX2Pipeline with diffusers/LTX-2.3-distilled-Diffusers in torch.float16, it finishes with no error or warning, but every output frame is NaN, so the exported mp4 is all black. I hit this in a real project on a Kaggle T4, where fp16 is the obvious choice because sm_75 has no native bf16.
Root cause, as far as I can tell: per_token_rms_norm in condition_embedder_ltx2.py computes mean(x**2) in the input dtype. The stacked Gemma-3 hidden states it normalizes reach |x| = 798,720 (measured), which is far above the fp16 max of 65,504:
- 372 of the 192.7M embedding values turn into
infwhen cast to fp16, and they become NaN after the norm. - Any value above about 256 overflows
x**2, so its whole (token, layer) slice is silently zeroed.
I put forward hooks on every module. In fp16, the first non-finite output is connectors.video_text_proj_in, which is the first module after the norm. In bf16, no module output anywhere (connectors, DiT, VAE) goes above 65,504: the peak is 45,824 in connectors.video_connector.transformer_blocks.7, and the DiT peaks at 23.8k.
per_layer_masked_mean_norm (the LTX-2.0 path) has the same pattern: its sum, mean, min and max all run in the input dtype. I haven't run LTX-2.0, though.
What I tried: I computed per_token_rms_norm in fp32, cast the result to fp16, and passed the embeddings in fp32. Same seed and settings in both runs:
| bf16 | fp16 + fp32 norm | |
|---|---|---|
pipe() wall time (T4, 8 steps, 576x320, 25 frames) |
326.5 s | 79.3 s |
| NaN in output | no | no |
| module outputs > 65,504 | 0 | 0 |
| frame std / mean | 0.254 / 0.38 | 0.127 / 0.49 |
The NaNs are gone and it runs 4.1x faster. But the frames come out washed out: it's the same scene layout, only greyish, with half the contrast of bf16. So there's at least one more fp16 precision problem further down the pipeline, and fp16 isn't usable yet even with this change.
Suggestion:
- Upcast to fp32 inside
per_token_rms_normandper_layer_masked_mean_norm(the usual RMSNorm practice) and cast back after normalizing. That removes the NaN. - Until fp16 output is correct, add a
logger.warningor a note inltx2.mdsaying LTX-2.x needs bf16 or fp32.
I'm happy to open a PR for 1 and/or 2 once you confirm the approach. If it helps, I can also narrow down which component causes the washed-out output (for example by running the VAE in fp32).
Reproduction
Minimal version, runs on CPU with no download:
import torch
from diffusers.models.condition_embedders.condition_embedder_ltx2 import per_token_rms_norm
torch.manual_seed(0)
x = torch.randn(1, 4, 3840, 49) # [batch, seq, caption_channels, layers], like the stacked Gemma-3 states
x[0, 1, 7, 10] = 300.0 # 300**2 > 65504
x[0, 2, 7, 10] = 798720.0 # the peak measured on the real embeddings
ref = per_token_rms_norm(x.double()).float()
out = per_token_rms_norm(x.half()).float()
for t in range(4):
print(t, f"{ref[0, t].abs().max().item():.3f}", f"{out[0, t].abs().nan_to_num(-1).max().item():.3f}", out[0, t].isnan().any().item())
# token fp64 ref fp16 nan
# 0 4.612 4.613 False
# 1 60.688 4.715 False <- the 300.0 entry's (token, layer) slice is zeroed
# 2 61.968 4.320 True <- NaN
# 3 4.541 4.543 FalseEnd-to-end run (T4, NF4 transformer + connectors, enable_model_cpu_offload(), VAE tiling, 8 distilled steps). It passes guidance_scale=1.0, audio_guidance_scale=1.0; the second one is needed because of #14649, otherwise a pipeline without a text encoder crashes on the negative prompt.
find_overflow.py (--dtype bf16 | fp16 | fp16fix)
"""Localize where LTX-2.3 overflows fp16. Run on a Kaggle T4 (needs diffusers from git).
python find_overflow.py --dtype bf16 # works; logs modules whose |activation| > 65504 (fp16 max)
python find_overflow.py --dtype fp16 # reproduces the bug; logs the FIRST module that emits inf/NaN
Paste both outputs into the issue. Same model + settings as the Navriti pipeline.
"""
import argparse, gc, os
os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
import numpy as np
import torch
import diffusers
from diffusers import BitsAndBytesConfig, LTX2Pipeline, LTX2VideoTransformer3DModel
from diffusers.pipelines.ltx2.connectors import LTX2TextConnectors
from diffusers.pipelines.ltx2.utils import DISTILLED_SIGMA_VALUES
from transformers import BitsAndBytesConfig as TBnb, Gemma3ForConditionalGeneration
REPO = "diffusers/LTX-2.3-distilled-Diffusers"
FP16_MAX = 65504.0
PROMPT = "A flat vector illustration of a padlock closing over a web browser window, clean shapes, soft colors."
ap = argparse.ArgumentParser()
ap.add_argument("--dtype", choices=["bf16", "fp16", "fp16fix"], required=True)
ap.add_argument("--nohooks", action="store_true") # for clean timing
ap.add_argument("--frames", type=int, default=25) # 8n+1; the bug did not depend on length
args = ap.parse_args()
dtype = torch.bfloat16 if args.dtype == "bf16" else torch.float16
if args.dtype == "fp16fix": # candidate fix: RMS-norm the raw Gemma states in fp32, then cast to the model dtype
import diffusers.models.condition_embedders.condition_embedder_ltx2 as C
def per_token_rms_norm_fp32(x, eps=1e-6):
x32 = x.float()
return (x32 * torch.rsqrt(x32.pow(2).mean(dim=2, keepdim=True) + eps)).to(torch.float16)
C.per_token_rms_norm = per_token_rms_norm_fp32
print(f"diffusers {diffusers.__version__} | torch {torch.__version__} | {torch.cuda.get_device_name()} | {dtype}")
# Phase 1: text encoding (always bf16, as in the real pipeline), then free the encoder.
te = Gemma3ForConditionalGeneration.from_pretrained(
REPO, subfolder="text_encoder", torch_dtype=torch.bfloat16,
quantization_config=TBnb(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16))
p = LTX2Pipeline.from_pretrained(REPO, text_encoder=te, transformer=None, vae=None, audio_vae=None,
connectors=None, vocoder=None, torch_dtype=torch.bfloat16)
with torch.no_grad():
pe, mask, _, _ = p.encode_prompt(prompt=PROMPT, do_classifier_free_guidance=False, device="cuda")
print(f"prompt_embeds max|x| = {pe.abs().max().item():.1f}")
pe16 = pe.to(torch.float16)
print(f"after .to(float16): inf={torch.isinf(pe16).sum().item()} of {pe16.numel()} | shape={tuple(pe.shape)}")
del pe16
del p, te; gc.collect(); torch.cuda.empty_cache()
# Phase 2: generation in the dtype under test.
nf4 = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=dtype, bnb_4bit_use_double_quant=True)
transformer = LTX2VideoTransformer3DModel.from_pretrained(REPO, subfolder="transformer", quantization_config=nf4, torch_dtype=dtype)
connectors = LTX2TextConnectors.from_pretrained(REPO, subfolder="connectors", quantization_config=nf4, torch_dtype=dtype)
pipe = LTX2Pipeline.from_pretrained(REPO, transformer=transformer, connectors=connectors, text_encoder=None, torch_dtype=dtype)
pipe.enable_model_cpu_offload()
pipe.vae.enable_tiling()
peak = {} # module -> running max |output| (GPU tensor, no sync per call)
first_bad = [] # fp16 mode: first module whose output is non-finite
handles = []
def tensors(o):
if torch.is_tensor(o): yield o
elif isinstance(o, (tuple, list)):
for x in o: yield from tensors(x)
elif hasattr(o, "sample"): yield from tensors(o.sample)
def hook(name):
def fn(_m, _inp, out):
for t in tensors(out):
if not t.is_floating_point() or t.numel() == 0: continue
m = t.detach().abs().amax().float()
peak[name] = torch.maximum(peak[name], m) if name in peak else m
if dtype == torch.float16 and not first_bad and not torch.isfinite(t).all():
first_bad.append(name)
print(f"FIRST NON-FINITE OUTPUT: {name}")
return fn
for comp_name in ((), ("connectors", "transformer", "vae"))[not args.nohooks]:
for n, m in getattr(pipe, comp_name).named_modules():
handles.append(m.register_forward_hook(hook(f"{comp_name}.{n}" if n else comp_name)))
import time
pe_in = pe.to("cuda", torch.float32) if args.dtype == "fp16fix" else pe.to("cuda", dtype)
torch.cuda.synchronize(); t0 = time.time()
with torch.no_grad():
video, _ = pipe(prompt_embeds=pe_in, prompt_attention_mask=mask.to("cuda"),
width=576, height=320, num_frames=args.frames, frame_rate=24, num_inference_steps=8,
sigmas=DISTILLED_SIGMA_VALUES, guidance_scale=1.0, audio_guidance_scale=1.0, # see diffusers#14649
generator=torch.Generator("cuda").manual_seed(0), output_type="np", return_dict=False)
torch.cuda.synchronize(); print(f"pipe() wall time: {time.time() - t0:.1f} s (hooks={not args.nohooks})")
for h in handles: h.remove()
v = np.asarray(video[0])
print(f"output: nan={np.isnan(v).any()} min={np.nanmin(v):.3f} max={np.nanmax(v):.3f}")
rows = sorted(((k, t.item()) for k, t in peak.items()), key=lambda kv: -kv[1] if np.isfinite(kv[1]) else -np.inf)
over = [(k, x) for k, x in rows if not np.isfinite(x) or x > FP16_MAX]
print(f"\nmodules with max|output| > fp16 max ({FP16_MAX:.0f}) or non-finite: {len(over)}")
for k, x in over[:30]: print(f" {x:>14.1f} {k}")
print("\ntop 10 overall:")
for k, x in rows[:10]: print(f" {x:>14.1f} {k}")Logs
# --dtype fp16
prompt_embeds max|x| = 798720.0
after .to(float16): inf=372 of 192675840 | shape=(1, 1024, 188160)
FIRST NON-FINITE OUTPUT: connectors.video_text_proj_in
output: nan=True min=nan max=nan
# --dtype bf16
output: nan=False min=0.000 max=1.000
modules with max|output| > fp16 max (65504) or non-finite: 0Context
During an ML internship I built a text-to-video explainer pipeline: LangGraph scene planning, then LTX-2.3 on a free Kaggle T4, then TTS and ffmpeg. The fp16 run "succeeded" end to end and still produced a 100% black video. I only noticed after measuring the pixel values, and I worked around it with bf16 plus a NaN guard. I came back to localize the cause. I used an AI coding assistant (Claude Code) to help write the diagnostic script; the runs and numbers come from my own Kaggle notebook.
System Info
- 🤗 Diffusers version: 0.43.0.dev0 (main @ 843f1743e259)
- Platform: Linux-6.18.55+-x86_64-with-glibc2.39 (Kaggle; diffusers-cli reports "Colab: Yes")
- Python version: 3.13.16
- PyTorch version (GPU?): 2.11.0+cu130 (True)
- Huggingface_hub version: 1.33.0
- Transformers version: 5.22.2
- Accelerate version: 1.15.0
- bitsandbytes version: 0.50.2
- Safetensors version: 0.8.0
- Accelerator: Tesla T4, 15360 MiB
- Using GPU in script?: yes, one T4
- Using distributed or parallel set-up in script?: no
Activity
1OF1Research commentedon Nov 14, 2026
Independent measurement of the shipped function itself (diffusers 0.43.0, torch 2.14.0+cpu, CPU), importing
per_token_rms_normfromcondition_embedder_ltx2.pyexactly as shipped:The boundary is exactly where the report puts it: a token's mean of squares crosses the fp16 maximum of 65,504 at RMS ≈ 255.94, the variance becomes
inf, andrsqrtturns the whole token to zero — silently, with no error and no NaN at this site. In a mixed tensor (quiet tokens at 10, one token at 300), only the hot token is zeroed; the rest norm correctly. And it is not only a cliff edge: random tokens at RMS 240 already come out completely wrong (max difference 4.13 against the fp32-upcast form), because individual elements above ~256 overflow their squares and take the token with them.This matches the minimal repro in the report — the 300.0 entry's slice zeroed, the large entry NaN downstream — and the proposed fp32 upcast inside the norm returns the reference value in every row above, which gives the fix a direct regression target.