Repository navigation
Description
Describe the bug
With fp16 samples, DDPMScheduler.add_noise returns the clean sample unchanged at t=0 and t=1: the noise level sqrt(1 - alphas_cumprod[t]) is 0 there instead of 0.0100 and 0.0148. It casts alphas_cumprod to the sample dtype before the 1 - ..., and in fp16 alphas_cumprod[t] rounds to 1 at t=0-1 (t=0-9 in bf16), so a mixed-precision training step at those timesteps asks the model for noise that isn't in its input.
#9004 hit this and upcast in the two ControlNet scripts, but the scheduler still casts, and train_unconditional.py, train_text_to_image.py and others pass fp16/bf16 samples straight in. The same cast sits in 21 places in 13 files (DDPM add_noise/get_velocity, their 17 # Copied from copies, Euler get_velocity, Kandinsky img2img add_noise). Doing the math in the scheduler's dtype and casting only the result fixes all of them, and fp32/fp64 outputs are unchanged, bit for bit:
--- a/src/diffusers/schedulers/scheduling_ddpm.py
+++ b/src/diffusers/schedulers/scheduling_ddpm.py
@@ -588,19 +588,22 @@ class DDPMScheduler(SchedulerMixin, ConfigMixin):
`torch.Tensor`:
The noisy samples.
"""
- # Make sure alphas_cumprod and timestep have same device and dtype as original_samples
+ # Make sure alphas_cumprod and timestep have same device as original_samples
# Move the self.alphas_cumprod to device to avoid redundant CPU to GPU data movement
# for the subsequent add_noise calls
self.alphas_cumprod = self.alphas_cumprod.to(device=original_samples.device)
- alphas_cumprod = self.alphas_cumprod.to(dtype=original_samples.dtype)
+ # Compute the coefficients in at least the dtype of self.alphas_cumprod and only cast the results to the dtype
+ # of original_samples: in float16/bfloat16, alphas_cumprod rounds to 1 at the first timesteps
+ compute_dtype = torch.promote_types(self.alphas_cumprod.dtype, original_samples.dtype)
+ alphas_cumprod = self.alphas_cumprod.to(dtype=compute_dtype)
timesteps = timesteps.to(original_samples.device)
- sqrt_alpha_prod = alphas_cumprod[timesteps] ** 0.5
+ sqrt_alpha_prod = (alphas_cumprod[timesteps] ** 0.5).to(original_samples.dtype)
sqrt_alpha_prod = sqrt_alpha_prod.flatten()
while len(sqrt_alpha_prod.shape) < len(original_samples.shape):
sqrt_alpha_prod = sqrt_alpha_prod.unsqueeze(-1)
- sqrt_one_minus_alpha_prod = (1 - alphas_cumprod[timesteps]) ** 0.5
+ sqrt_one_minus_alpha_prod = ((1 - alphas_cumprod[timesteps]) ** 0.5).to(original_samples.dtype)
sqrt_one_minus_alpha_prod = sqrt_one_minus_alpha_prod.flatten()
while len(sqrt_one_minus_alpha_prod.shape) < len(original_samples.shape):
sqrt_one_minus_alpha_prod = sqrt_one_minus_alpha_prod.unsqueeze(-1)Would you take that as one PR across all 21 (through make fix-copies), or would you rather start with DDPM?
Reproduction
One step of examples/unconditional_image_generation/train_unconditional.py with --mixed_precision fp16 or bf16:
import torch
from diffusers import DDPMScheduler
# one training step of examples/unconditional_image_generation/train_unconditional.py with --mixed_precision fp16/bf16
noise_scheduler = DDPMScheduler(num_train_timesteps=1000, beta_schedule="linear", prediction_type="epsilon")
timesteps = torch.tensor([0, 1, 2, 10])
for weight_dtype in [torch.float16, torch.bfloat16]:
clean_images = (torch.rand(4, 3, 64, 64) * 2 - 1).to(weight_dtype)
noise = torch.randn(clean_images.shape, dtype=weight_dtype)
noisy_images = noise_scheduler.add_noise(clean_images, noise, timesteps)
print(weight_dtype, [torch.equal(noisy_images[i], clean_images[i]) for i in range(4)])
# torch.float16 [True, True, False, False] <- no noise added at t=0, 1
# torch.bfloat16 [True, True, True, False] <- no noise added at t=0, 1, 2and the noise level itself:
# mechanism check: add_noise(0, 1, t) is the noise level sqrt(1 - alphas_cumprod[t]) itself
for weight_dtype in [torch.float16, torch.bfloat16]:
zeros, ones = torch.zeros(4, dtype=weight_dtype), torch.ones(4, dtype=weight_dtype)
print(noise_scheduler.add_noise(zeros, ones, timesteps))
print((1 - noise_scheduler.alphas_cumprod[timesteps]) ** 0.5)
# tensor([0.0000, 0.0000, 0.0221, 0.0442], dtype=torch.float16)
# tensor([0.0000, 0.0000, 0.0000, 0.0625], dtype=torch.bfloat16)
# tensor([0.0100, 0.0148, 0.0190, 0.0468]) <- float32, what it should beOver all 1000 timesteps, 16 noise levels are more than 1% off in fp16 and 62 in bf16, on CPU and CUDA alike, with 0.43.0 and with main.
Logs
No response
System Info
- 🤗 Diffusers version: 0.43.0
- Platform: Linux-6.12.95-124.187.amzn2023.x86_64-x86_64-with-glibc2.34
- Running on Google Colab?: No
- Python version: 3.12.13
- PyTorch version (GPU?): 2.15.0+cu132 (True)
- Huggingface_hub version: 2.4.0
- Transformers version: 5.22.2
- Accelerate version: 1.15.0
- PEFT version: not installed
- Safetensors version: 0.8.0
- xFormers version: not installed
- Accelerator: NVIDIA A100-SXM4-80GB, 81920 MiB
- Using GPU in script?: No (CPU); the same output on one A100
- Using distributed or parallel set-up in script?: No
main at 27a5490 (2026-11-13) gives the same output.
Who can help?
No response
Activity
1OF1Research commentedon Nov 14, 2026
Independent measurement on the shipped release (diffusers 0.43.0, CPU). With a zero clean sample and unit noise,
DDPMScheduler.add_noisereturns the effective noise coefficient directly:The fp32 row matches the fp64 reference
sqrt(1 - alphas_cumprod)(0.01000083 at t=0, 0.01483041 at t=1), so the harness itself is sound: in fp16 the first two timesteps add literally no noise, and in bf16 the first ten add none, exactly as the cast-order mechanism in the report predicts. A training step at those timesteps is asking the model to predict noise that is not in its input.The proposed direction — compute the coefficients in the promoted dtype and cast only the results — also has the right safety property to check in review: fp32/fp64 outputs should stay bit-identical, which the fp32 row above gives a direct regression target for.
sunhouy commentedon Nov 15, 2026
Opened #15300 for this. Went with the promoted-dtype direction from the report: the schedule stays in its own dtype for the math and only
sqrt(alpha)/sqrt(1 - alpha)get cast to the sample dtype.Carried it through all the
# Copied fromcopies withcheck_copies.py --fix_and_overwrite(15 files, no other changes), plusEulerDiscreteScheduler.get_velocityand the Kandinsky img2imgadd_noise, which build/cast the schedule the same way. fp32/fp64 output is unchanged bit for bit.New tests are
test_add_noise_in_low_precision/test_get_velocity_in_low_precisionin the DDPM tests andtest_get_velocity_in_low_precisionin the Euler one; 3 failed before, all green after.Robertboy18 commentedon Nov 15, 2026
Thanks @1OF1Research for the measurement and @sunhouy for #15300! Keeping the schedule in its own dtype and only casting
sqrt(alpha)/sqrt(1 - alpha)to the sample dtype sounds right to me.