Feature Description
StableDiffusionXLInstructPix2PixPipeline is the only InstructPix2Pix pipeline without IP-Adapter support. The SD version got it in #7820, but the SDXL one has no IPAdapterMixin, no image_encoder / feature_extractor components, and no ip_adapter_image argument, so there's no way to combine an instruction edit with an image prompt at SDXL resolution.
Solution I'd like to introduce
Mirror the SD InstructPix2Pix pipeline :
- add IPAdapterMixin and the optional image_encoder / feature_extractor components
- add ip_adapter_image / ip_adapter_image_embeds to call and check_inputs
- Copied from StableDiffusionInstructPix2PixPipeline.prepare_ip_adapter_image_embeds, which is already three-way-CFG aware
The one design question I'd like to confirm before opening a PR is the CFG ordering. InstructPix2Pix expands the batch as [text, image, uncond] and lays out prompt_embeds as [prompt, negative, negative]. Since IP-Adapter conditioning is cross-attention conditioning like the prompt, I ordered the image embeddings the same way, so IP-Adapter strength is scaled by guidance_scale rather than image_guidance_scale. That matches what #7820 does for the SD pipeline.
Alternatives considered
I first ordered the embeddings [cond, cond, uncond], putting the IP-Adapter conditioning in the image_guidance_scale term instead. It runs, but it disagrees with the SD pipeline and makes the adapter much weaker at default settings, so I dropped it. Happy to revisit if you'd prefer that behaviour for the edit pipelines.
Additional context.
I have a working branch with tests (21 passed, 18 skipped on tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_instruction_pix2pix.py, with check_copies / check_dummies / make quality clean). Would you take a PR for this?
Feature Description
StableDiffusionXLInstructPix2PixPipeline is the only InstructPix2Pix pipeline without IP-Adapter support. The SD version got it in #7820, but the SDXL one has no IPAdapterMixin, no image_encoder / feature_extractor components, and no ip_adapter_image argument, so there's no way to combine an instruction edit with an image prompt at SDXL resolution.
Solution I'd like to introduce
Mirror the SD InstructPix2Pix pipeline :
The one design question I'd like to confirm before opening a PR is the CFG ordering. InstructPix2Pix expands the batch as [text, image, uncond] and lays out prompt_embeds as [prompt, negative, negative]. Since IP-Adapter conditioning is cross-attention conditioning like the prompt, I ordered the image embeddings the same way, so IP-Adapter strength is scaled by guidance_scale rather than image_guidance_scale. That matches what #7820 does for the SD pipeline.
Alternatives considered
I first ordered the embeddings [cond, cond, uncond], putting the IP-Adapter conditioning in the image_guidance_scale term instead. It runs, but it disagrees with the SD pipeline and makes the adapter much weaker at default settings, so I dropped it. Happy to revisit if you'd prefer that behaviour for the edit pipelines.
Additional context.
I have a working branch with tests (21 passed, 18 skipped on tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_instruction_pix2pix.py, with check_copies / check_dummies / make quality clean). Would you take a PR for this?