Skip to content

img2img: outputs paired with the wrong init image #15176

Open

Description

@Arnav66692

Describe the bug

With one init image per prompt and num_images_per_prompt > 1, half of the outputs start from the other prompt's image, and nothing errors. With prompt=["a cat", "a dog"], image=[cat_img, dog_img] and num_images_per_prompt=2, the second "a cat" output starts from dog_img and the first "a dog" output starts from cat_img.

The cause is in StableDiffusionImg2ImgPipeline.prepare_latents. The prompt embeddings are ordered [a, a, b, b], but torch.cat([image] * n) and torch.cat([init_latents] * n) give [a, b, a, b], so rows 1 and 2 get the wrong image.

I confirmed this in SD, SDXL, Kolors and LCM img2img. The same code is copied into 7 more pipelines, which I didn't test. The tests miss it because test_num_images_per_prompt only checks the shape and test_inference_batch_single_identical only compares row 0.

By the way I already have a fix for this. It repeats each image with repeat_interleave when there's one image per prompt, so a single shared image gives the same results as before, and there's a test that fails on main and passes with it. I would love to open a PR if that works.

Reproduction

import torch
from diffusers import StableDiffusionImg2ImgPipeline

pipe = StableDiffusionImg2ImgPipeline.from_pretrained("hf-internal-testing/tiny-stable-diffusion-torch", safety_checker=None)
pipe.set_progress_bar_config(disable=True)

image_a = torch.rand(1, 3, 64, 64, generator=torch.Generator().manual_seed(1))
image_b = torch.rand(1, 3, 64, 64, generator=torch.Generator().manual_seed(2))
kwargs = dict(num_inference_steps=2, strength=0.5, output_type="pt")


def gen(seed):
    return torch.Generator().manual_seed(seed)


# one image per prompt, two outputs per prompt -> expected order: (a, img a), (a, img a), (b, img b), (b, img b)
batched = pipe(["a cat", "a dog"], image=torch.cat([image_a, image_b]), num_images_per_prompt=2,
               generator=[gen(i) for i in range(4)], **kwargs).images

for i, (prompt, image) in enumerate([("a cat", image_a), ("a cat", image_a), ("a dog", image_b), ("a dog", image_b)]):
    other = image_b if image is image_a else image_a
    own = pipe(prompt, image=image, generator=gen(i), **kwargs).images[0]
    wrong = pipe(prompt, image=other, generator=gen(i), **kwargs).images[0]
    print(f"output {i}: diff vs its own image {(batched[i] - own).abs().max():.2e}, "
          f"vs the other prompt's image {(batched[i] - wrong).abs().max():.2e}")

Logs

Outputs 1 and 2 match a single call that uses the other prompt's image:


output 0: diff vs its own image 1.04e-05, vs the other prompt's image 4.96e-01
output 1: diff vs its own image 5.31e-01, vs the other prompt's image 8.46e-06
output 2: diff vs its own image 6.56e-01, vs the other prompt's image 1.05e-05
output 3: diff vs its own image 1.08e-05, vs the other prompt's image 5.91e-01

System Info

- Diffusers version: 0.42.0.dev0 (main)
- Platform: macOS-15.5-arm64-arm-64bit
- Running on Google Colab?: No
- Python version: 3.12.13
- PyTorch version (GPU?): 2.15.0 (False)
- Huggingface_hub version: 2.4.0
- Transformers version: 5.23.0.dev0
- Accelerate version: 1.15.0
- PEFT version: 0.21.2
- Safetensors version: 0.8.0
- xFormers version: not installed
- Accelerator: Apple M4
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

Who can help?

@yiyixuxu @asomoza

Activity

  1. Arnav66692 commented on Nov 3, 2026

    @Arnav66692
    Author

    By the way I already have a fix for this and would love to open a PR if that works

  2. davrens101 commented on Nov 13, 2026

    @davrens101

    I'll take this one — fix to interleave images per prompt () instead of concatenating (), plus a regression test.

  3. davrens101 commented on Nov 13, 2026

    @davrens101

    The repro and a verified fix are on my fork branch: https://github.com/davrens101/diffusers/tree/fix/img2img-prepare-latents-interleave. I closed my fork PR #1 since there's no maintainer ack on this issue yet — happy to open an upstream PR once a maintainer claims it.

  4. 1OF1Research commented on Nov 14, 2026

    @1OF1Research

    Adding the pipeline census for the fix discussion, run against the shipped diffusers 0.43.0 wheel.

    The report names SD, SDXL, Kolors and LCM as confirmed and 7 more pipelines as carrying the same code untested. The pattern to look for is the expansion inside prepare_latents: when batch_size > init_latents.shape[0], the block repeat

    init_latents = torch.cat([init_latents] * additional_image_per_prompt, dim=0)

    produces [a, b, a, b] against prompt embeddings ordered [a, a, b, b]. Reading the shipped code, that exact expansion is present in:

    • stable_diffusion/pipeline_stable_diffusion_img2img.py (with a deprecation warning whose text describes the branch as duplicating images "to match the number of text prompts" — with num_images_per_prompt > 1 the mismatch it creates is against the expanded batch, not the prompt count)
    • stable_diffusion_3/pipeline_stable_diffusion_3_img2img.py
    • stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py
    • z_image/pipeline_z_image_img2img.py and z_image/pipeline_z_image_inpaint.py (on image_latents)
    • stable_diffusion/pipeline_stable_diffusion_depth2img.py
    • stable_diffusion/pipeline_stable_diffusion_instruct_pix2pix.py and the SDXL instruct-pix2pix (on image_latents)
    • visualcloze/pipeline_visualcloze_generation.py (on image_latents)

    The torch.cat([latents] * 2) / * 3 sites elsewhere are classifier-free-guidance doubling and are not this bug.

    Worth noting for whichever fix lands: several sibling img2img pipelines already expand with repeat_interleave instead (Flux, ControlNet, Kandinsky, PAG, DeepFloyd IF img2img), which produces the [a, a, b, b] order the embeddings use — so the fix shape the forks chose matches the pattern the library itself uses elsewhere. A per-pipeline pairing test (batched output row vs the single call with its own image, as in the report's repro) would cover the whole set above, not just the four confirmed pipelines.

  5. Arnav66692 commented on Nov 15, 2026

    @Arnav66692
    Author

    I filed this and have a fix ready, with a test that fails on main and passes with it. I'll open the PR for SD and SDXL img2img and the 9 pipelines that copy their prepare_latents, and leave the rest of the census for follow-ups so this one stays easy to review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions