Skip to content

add_noise / get_velocity in DDPM-family schedulers add no noise at the first timesteps for fp16/bf16 samples #15277

Open

Description

@Robertboy18

Describe the bug

With fp16 samples, DDPMScheduler.add_noise returns the clean sample unchanged at t=0 and t=1: the noise level sqrt(1 - alphas_cumprod[t]) is 0 there instead of 0.0100 and 0.0148. It casts alphas_cumprod to the sample dtype before the 1 - ..., and in fp16 alphas_cumprod[t] rounds to 1 at t=0-1 (t=0-9 in bf16), so a mixed-precision training step at those timesteps asks the model for noise that isn't in its input.

#9004 hit this and upcast in the two ControlNet scripts, but the scheduler still casts, and train_unconditional.py, train_text_to_image.py and others pass fp16/bf16 samples straight in. The same cast sits in 21 places in 13 files (DDPM add_noise/get_velocity, their 17 # Copied from copies, Euler get_velocity, Kandinsky img2img add_noise). Doing the math in the scheduler's dtype and casting only the result fixes all of them, and fp32/fp64 outputs are unchanged, bit for bit:

--- a/src/diffusers/schedulers/scheduling_ddpm.py
+++ b/src/diffusers/schedulers/scheduling_ddpm.py
@@ -588,19 +588,22 @@ class DDPMScheduler(SchedulerMixin, ConfigMixin):
             `torch.Tensor`:
                 The noisy samples.
         """
-        # Make sure alphas_cumprod and timestep have same device and dtype as original_samples
+        # Make sure alphas_cumprod and timestep have same device as original_samples
         # Move the self.alphas_cumprod to device to avoid redundant CPU to GPU data movement
         # for the subsequent add_noise calls
         self.alphas_cumprod = self.alphas_cumprod.to(device=original_samples.device)
-        alphas_cumprod = self.alphas_cumprod.to(dtype=original_samples.dtype)
+        # Compute the coefficients in at least the dtype of self.alphas_cumprod and only cast the results to the dtype
+        # of original_samples: in float16/bfloat16, alphas_cumprod rounds to 1 at the first timesteps
+        compute_dtype = torch.promote_types(self.alphas_cumprod.dtype, original_samples.dtype)
+        alphas_cumprod = self.alphas_cumprod.to(dtype=compute_dtype)
         timesteps = timesteps.to(original_samples.device)
 
-        sqrt_alpha_prod = alphas_cumprod[timesteps] ** 0.5
+        sqrt_alpha_prod = (alphas_cumprod[timesteps] ** 0.5).to(original_samples.dtype)
         sqrt_alpha_prod = sqrt_alpha_prod.flatten()
         while len(sqrt_alpha_prod.shape) < len(original_samples.shape):
             sqrt_alpha_prod = sqrt_alpha_prod.unsqueeze(-1)
 
-        sqrt_one_minus_alpha_prod = (1 - alphas_cumprod[timesteps]) ** 0.5
+        sqrt_one_minus_alpha_prod = ((1 - alphas_cumprod[timesteps]) ** 0.5).to(original_samples.dtype)
         sqrt_one_minus_alpha_prod = sqrt_one_minus_alpha_prod.flatten()
         while len(sqrt_one_minus_alpha_prod.shape) < len(original_samples.shape):
             sqrt_one_minus_alpha_prod = sqrt_one_minus_alpha_prod.unsqueeze(-1)

Would you take that as one PR across all 21 (through make fix-copies), or would you rather start with DDPM?

Reproduction

One step of examples/unconditional_image_generation/train_unconditional.py with --mixed_precision fp16 or bf16:

import torch
from diffusers import DDPMScheduler

# one training step of examples/unconditional_image_generation/train_unconditional.py with --mixed_precision fp16/bf16
noise_scheduler = DDPMScheduler(num_train_timesteps=1000, beta_schedule="linear", prediction_type="epsilon")
timesteps = torch.tensor([0, 1, 2, 10])
for weight_dtype in [torch.float16, torch.bfloat16]:
    clean_images = (torch.rand(4, 3, 64, 64) * 2 - 1).to(weight_dtype)
    noise = torch.randn(clean_images.shape, dtype=weight_dtype)
    noisy_images = noise_scheduler.add_noise(clean_images, noise, timesteps)
    print(weight_dtype, [torch.equal(noisy_images[i], clean_images[i]) for i in range(4)])
# torch.float16 [True, True, False, False]   <- no noise added at t=0, 1
# torch.bfloat16 [True, True, True, False]   <- no noise added at t=0, 1, 2

and the noise level itself:

# mechanism check: add_noise(0, 1, t) is the noise level sqrt(1 - alphas_cumprod[t]) itself
for weight_dtype in [torch.float16, torch.bfloat16]:
    zeros, ones = torch.zeros(4, dtype=weight_dtype), torch.ones(4, dtype=weight_dtype)
    print(noise_scheduler.add_noise(zeros, ones, timesteps))
print((1 - noise_scheduler.alphas_cumprod[timesteps]) ** 0.5)
# tensor([0.0000, 0.0000, 0.0221, 0.0442], dtype=torch.float16)
# tensor([0.0000, 0.0000, 0.0000, 0.0625], dtype=torch.bfloat16)
# tensor([0.0100, 0.0148, 0.0190, 0.0468])   <- float32, what it should be

Over all 1000 timesteps, 16 noise levels are more than 1% off in fp16 and 62 in bf16, on CPU and CUDA alike, with 0.43.0 and with main.

Logs

No response

System Info

- 🤗 Diffusers version: 0.43.0
- Platform: Linux-6.12.95-124.187.amzn2023.x86_64-x86_64-with-glibc2.34
- Running on Google Colab?: No
- Python version: 3.12.13
- PyTorch version (GPU?): 2.15.0+cu132 (True)
- Huggingface_hub version: 2.4.0
- Transformers version: 5.22.2
- Accelerate version: 1.15.0
- PEFT version: not installed
- Safetensors version: 0.8.0
- xFormers version: not installed
- Accelerator: NVIDIA A100-SXM4-80GB, 81920 MiB
- Using GPU in script?: No (CPU); the same output on one A100
- Using distributed or parallel set-up in script?: No

main at 27a5490 (2026-11-13) gives the same output.

Who can help?

No response

Activity

  1. 1OF1Research commented on Nov 14, 2026

    @1OF1Research

    Independent measurement on the shipped release (diffusers 0.43.0, CPU). With a zero clean sample and unit noise, DDPMScheduler.add_noise returns the effective noise coefficient directly:

    dtype t=0 t=1 t=2 t=3–9
    fp16 0.0 0.0 0.0221 —
    bf16 0.0 0.0 0.0 all 0.0
    fp32 0.01000 0.01483 — —

    The fp32 row matches the fp64 reference sqrt(1 - alphas_cumprod) (0.01000083 at t=0, 0.01483041 at t=1), so the harness itself is sound: in fp16 the first two timesteps add literally no noise, and in bf16 the first ten add none, exactly as the cast-order mechanism in the report predicts. A training step at those timesteps is asking the model to predict noise that is not in its input.

    The proposed direction — compute the coefficients in the promoted dtype and cast only the results — also has the right safety property to check in review: fp32/fp64 outputs should stay bit-identical, which the fp32 row above gives a direct regression target for.

  2. sunhouy commented on Nov 15, 2026

    @sunhouy

    Opened #15300 for this. Went with the promoted-dtype direction from the report: the schedule stays in its own dtype for the math and only sqrt(alpha) / sqrt(1 - alpha) get cast to the sample dtype.

    Carried it through all the # Copied from copies with check_copies.py --fix_and_overwrite (15 files, no other changes), plus EulerDiscreteScheduler.get_velocity and the Kandinsky img2img add_noise, which build/cast the schedule the same way. fp32/fp64 output is unchanged bit for bit.

    New tests are test_add_noise_in_low_precision / test_get_velocity_in_low_precision in the DDPM tests and test_get_velocity_in_low_precision in the Euler one; 3 failed before, all green after.

  3. Robertboy18 commented on Nov 15, 2026

    @Robertboy18
    Author

    Thanks @1OF1Research for the measurement and @sunhouy for #15300! Keeping the schedule in its own dtype and only casting sqrt(alpha) / sqrt(1 - alpha) to the sample dtype sounds right to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions