Repository navigation
Description
Describe the bug
With one init image per prompt and num_images_per_prompt > 1, half of the outputs start from the other prompt's image, and nothing errors. With prompt=["a cat", "a dog"], image=[cat_img, dog_img] and num_images_per_prompt=2, the second "a cat" output starts from dog_img and the first "a dog" output starts from cat_img.
The cause is in StableDiffusionImg2ImgPipeline.prepare_latents. The prompt embeddings are ordered [a, a, b, b], but torch.cat([image] * n) and torch.cat([init_latents] * n) give [a, b, a, b], so rows 1 and 2 get the wrong image.
I confirmed this in SD, SDXL, Kolors and LCM img2img. The same code is copied into 7 more pipelines, which I didn't test. The tests miss it because test_num_images_per_prompt only checks the shape and test_inference_batch_single_identical only compares row 0.
By the way I already have a fix for this. It repeats each image with repeat_interleave when there's one image per prompt, so a single shared image gives the same results as before, and there's a test that fails on main and passes with it. I would love to open a PR if that works.
Reproduction
import torch
from diffusers import StableDiffusionImg2ImgPipeline
pipe = StableDiffusionImg2ImgPipeline.from_pretrained("hf-internal-testing/tiny-stable-diffusion-torch", safety_checker=None)
pipe.set_progress_bar_config(disable=True)
image_a = torch.rand(1, 3, 64, 64, generator=torch.Generator().manual_seed(1))
image_b = torch.rand(1, 3, 64, 64, generator=torch.Generator().manual_seed(2))
kwargs = dict(num_inference_steps=2, strength=0.5, output_type="pt")
def gen(seed):
return torch.Generator().manual_seed(seed)
# one image per prompt, two outputs per prompt -> expected order: (a, img a), (a, img a), (b, img b), (b, img b)
batched = pipe(["a cat", "a dog"], image=torch.cat([image_a, image_b]), num_images_per_prompt=2,
generator=[gen(i) for i in range(4)], **kwargs).images
for i, (prompt, image) in enumerate([("a cat", image_a), ("a cat", image_a), ("a dog", image_b), ("a dog", image_b)]):
other = image_b if image is image_a else image_a
own = pipe(prompt, image=image, generator=gen(i), **kwargs).images[0]
wrong = pipe(prompt, image=other, generator=gen(i), **kwargs).images[0]
print(f"output {i}: diff vs its own image {(batched[i] - own).abs().max():.2e}, "
f"vs the other prompt's image {(batched[i] - wrong).abs().max():.2e}")Logs
Outputs 1 and 2 match a single call that uses the other prompt's image:
output 0: diff vs its own image 1.04e-05, vs the other prompt's image 4.96e-01
output 1: diff vs its own image 5.31e-01, vs the other prompt's image 8.46e-06
output 2: diff vs its own image 6.56e-01, vs the other prompt's image 1.05e-05
output 3: diff vs its own image 1.08e-05, vs the other prompt's image 5.91e-01System Info
- Diffusers version: 0.42.0.dev0 (main)
- Platform: macOS-15.5-arm64-arm-64bit
- Running on Google Colab?: No
- Python version: 3.12.13
- PyTorch version (GPU?): 2.15.0 (False)
- Huggingface_hub version: 2.4.0
- Transformers version: 5.23.0.dev0
- Accelerate version: 1.15.0
- PEFT version: 0.21.2
- Safetensors version: 0.8.0
- xFormers version: not installed
- Accelerator: Apple M4
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
Activity
Arnav66692 commentedon Nov 3, 2026
By the way I already have a fix for this and would love to open a PR if that works
davrens101 commentedon Nov 13, 2026
I'll take this one — fix to interleave images per prompt () instead of concatenating (), plus a regression test.
davrens101 commentedon Nov 13, 2026
The repro and a verified fix are on my fork branch: https://github.com/davrens101/diffusers/tree/fix/img2img-prepare-latents-interleave. I closed my fork PR #1 since there's no maintainer ack on this issue yet — happy to open an upstream PR once a maintainer claims it.
1OF1Research commentedon Nov 14, 2026
Adding the pipeline census for the fix discussion, run against the shipped diffusers 0.43.0 wheel.
The report names SD, SDXL, Kolors and LCM as confirmed and 7 more pipelines as carrying the same code untested. The pattern to look for is the expansion inside
prepare_latents: whenbatch_size > init_latents.shape[0], the block repeatproduces
[a, b, a, b]against prompt embeddings ordered[a, a, b, b]. Reading the shipped code, that exact expansion is present in:stable_diffusion/pipeline_stable_diffusion_img2img.py(with a deprecation warning whose text describes the branch as duplicating images "to match the number of text prompts" — withnum_images_per_prompt > 1the mismatch it creates is against the expanded batch, not the prompt count)stable_diffusion_3/pipeline_stable_diffusion_3_img2img.pystable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.pyz_image/pipeline_z_image_img2img.pyandz_image/pipeline_z_image_inpaint.py(onimage_latents)stable_diffusion/pipeline_stable_diffusion_depth2img.pystable_diffusion/pipeline_stable_diffusion_instruct_pix2pix.pyand the SDXL instruct-pix2pix (onimage_latents)visualcloze/pipeline_visualcloze_generation.py(onimage_latents)The
torch.cat([latents] * 2)/* 3sites elsewhere are classifier-free-guidance doubling and are not this bug.Worth noting for whichever fix lands: several sibling img2img pipelines already expand with
repeat_interleaveinstead (Flux, ControlNet, Kandinsky, PAG, DeepFloyd IF img2img), which produces the[a, a, b, b]order the embeddings use — so the fix shape the forks chose matches the pattern the library itself uses elsewhere. A per-pipeline pairing test (batched output row vs the single call with its own image, as in the report's repro) would cover the whole set above, not just the four confirmed pipelines.Arnav66692 commentedon Nov 15, 2026
I filed this and have a fix ready, with a test that fails on main and passes with it. I'll open the PR for SD and SDXL img2img and the 9 pipelines that copy their
prepare_latents, and leave the rest of the census for follow-ups so this one stays easy to review.