Add IP-Adapter support to StableDiffusionXLInstructPix2PixPipeline

[vivi27x]

What does this PR do?

Adds IP-Adapter support to StableDiffusionXLInstructPix2PixPipeline, the only InstructPix2Pix pipeline that didn't have it - the SD version got it in #7820.

I hit this building a depth-aware image editing pipeline on top of CosXL Edit, which StableDiffusionXLInstructPix2PixPipeline is the only way to load in diffusers. I needed to combine an instruction edit with an IP-Adapter style reference, and with no IP-Adapter support on this pipeline I ended up wiring the adapter in by hand outside the library - which meant no set_ip_adapter_scale, no masks, and no multi-adapter support. The three-way CFG batch is the part that makes this pipeline different from the other SDXL ones, and it's what the standard two-way embedding helper can't produce.

Fixes #14761

Changes

• IPAdapterMixin on the pipeline, with optional image_encoder / feature_extractor components and image_encoder in model_cpu_offload_seq.
• encode_image and prepare_ip_adapter_image_embeds, both # Copied from their counterparts. prepare_ip_adapter_image_embeds is copied from StableDiffusionInstructPix2PixPipeline, not StableDiffusionPipeline, because InstructPix2Pix runs a three-way CFG batch.
• ip_adapter_image / ip_adapter_image_embeds in __call__ and check_inputs.
• IP-Adapter tests, plus image_encoder / feature_extractor in the dummy components.

On the CFG ordering

InstructPix2Pix expands the batch three ways as [text, image, uncond], and prompt_embeds is laid out as [prompt, negative, negative]. IP-Adapter conditioning is cross-attention conditioning like the prompt, so its embeddings follow the same layout - meaning IP-Adapter strength is scaled by guidance_scale, consistent with every other pipeline. I checked this by hooking unet.encoder_hid_proj: with CFG on, the projected hidden states come back as batch 3, chunk 0 carries the image embedding and chunks 1 and 2 are identical negatives.

I first ordered them [cond, cond, uncond], which puts IP-Adapter conditioning in the image_guidance_scale term instead. That's a real behaviour difference and it disagrees with the SD pipeline, so I dropped it. Happy to switch if you'd rather the edit pipelines behaved that way.

Tests

$ pytest tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_instruction_pix2pix.py -q
21 passed, 18 skipped, 57 warnings in 54.30s

The 18 skips are the accelerator-gated memory tests - no GPU on this machine.

Two tests in that module that weren't passing before now do: test_components_function (the new components are in get_dummy_components) and test_save_load_optional_components, which was skipped as "every optional component is needed to encode the prompt" and is now implemented over the droppable image_encoder / feature_extractor, matching test_stable_diffusion_xl_img2img.py.

Also green:

$ make quality
$ python utils/check_copies.py
$ python utils/check_dummies.py
$ python utils/check_support_list.py
$ python utils/check_forward_call_docstrings.py
$ make deps_table_check_updated

The shared IPAdapterTesterMixin only exercises pre-computed ip_adapter_image_embeds, so I ran the ip_adapter_image path by hand as well: a single PIL image, num_images_per_prompt=3, a batch of 2 prompts, CFG off, and two stacked adapters all produce correctly shaped output, and set_ip_adapter_scale(0.0) reproduces the no-adapter output exactly.

Notes

• The two new __init__ parameters sit before force_zeros_for_empty_prompt, matching StableDiffusionXLImg2ImgPipeline. That shifts positions for anyone constructing the pipeline positionally -- appending them at the end would avoid that but break consistency with the sibling pipelines. Happy to move them.
• docs/source/en/using-diffusers/ip_adapter.md doesn't keep a list of supported pipelines, so I didn't add a docs change there. Let me know if you'd like an example added.

Before submitting

• Did you use an AI agent (Claude Code, Codex, Cursor, etc.) to help with this PR? If so:
  • Did you read the Coding with AI agents guide?
  • Did you run the self-review skill on the diff?
  • Did you share the final self-review notes in the PR description or a comment? -- I posted as a comment below
• Did you read the contributor guideline?
• Did you read our philosophy doc?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case. -- Add IP-Adapter support to StableDiffusionXLInstructPix2PixPipeline #14761
• Did you make sure to update the documentation with your changes? -- __call__ docstrings for the new arguments
• Did you write any new necessary tests?
• Are you the author (or part of the team) of the model/pipeline (only applicable for model/pipeline related PRs)?

Who can review?

@yiyixuxu @asomoza

[vivi27x]

Self-review notes

Ran the self-review skill over the branch diff (git diff upstream/main...HEAD), against .ai/references/review-rules.md plus code_style.md, pipelines.md and testing.md.

Verdict: READY — no blocking issues in the submitted diff.

Found and fixed during review

1. prepare_ip_adapter_image_embeds was copied from StableDiffusionPipeline (two-way CFG) and then reshaped inside __call__ into [cond, cond, uncond]. It is now copied from StableDiffusionInstructPix2PixPipeline, which is already three-way aware, and the reshape is gone. This also fixed the # Copied from marker — check_copies.py was failing because the copied body had been edited.
2. encode_image had a try/except ImportError around ImageProjection and a CLIPImageProcessor() fallback for an unset self.feature_extractor — defensive paths that code_style.md rules out. ImageProjection is now a module-level import like every other UNet pipeline, and encode_image is a plain # Copied from.
3. check_inputs did not validate ip_adapter_image / ip_adapter_image_embeds.
4. The two new __call__ arguments were undocumented (check_forward_call_docstrings.py covers this).
5. Typing used Optional / Union / List against the repo's PEP 604 style, with Dict and Tuple unused, and image_encoder / feature_extractor were unannotated in __init__.
6. test_components_function was failing because get_dummy_components did not supply the two new components.

Not fixed — deliberate

The four items in the Notes section of the description: the __init__ parameter placement, the added_cond_kwargs guard form, the test_save_load_optional_components duplication, and the absent ip_adapter.md example. Each is a judgement call I would rather leave to the reviewer than guess at.

Dead-code check

encode_image is reachable only via the ip_adapter_image path, whicTesterMixin feeds pre-computed embeddings and loads adapters withimage_encoder_folder=None. That is true of every UNet pipeline in the repo, so I verified the path manually instead (results in the description). The dummy image_encodercannot serve it either:create_ip_adapter_state_dict_embed_dim=unet.cross_attention_dim=64) against a projection_dim=32encoder. The identical mismatch exists in `test_stable_diffusion_xl_imde fixture property rather than something this PR introduces.