support the inference of FLUX Fill, FLUX Redux, Insert Anything

[huan-yin]

Add Inference Support for Flux Fill, Flux Redux, and Insert Anything

Commit Introduction

Inference capabilities for three FLUX-series models:

• FLUX.1-Fill-dev: Mask-guided image inpainting / outpainting.
• FLUX.1-Redux-dev: Image semantic / style transfer and variation generation.
• Insert-Anything (https://github.com/song-wensong/insert-anything): Object insertion editing based on FLUX.

All three capabilities are exposed through the unified FluxImagePipeline interface.

---

1. FLUX.1-Fill-dev: Masked Image Inpainting

Overview

FLUX.1-Fill-dev is a dedicated inpainting model. Given an image and a binary mask, it fills the masked region according to a text prompt while preserving the unmasked parts of the original image. It is suitable for object removal, content completion, and outpainting tasks.

Example Script

examples/flux/model_inference_low_vram/FLUX.1-Fill-dev.py

import torch
from diffsynth.pipelines.flux_image import FluxImagePipeline, ModelConfig
from PIL import Image
from modelscope import dataset_snapshot_download

vram_config = {
    "offload_dtype": torch.float8_e4m3fn,
    "offload_device": "cpu",
    "onload_dtype": torch.float8_e4m3fn,
    "onload_device": "cpu",
    "preparing_dtype": torch.float8_e4m3fn,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}
pipe = FluxImagePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="black-forest-labs/FLUX.1-Fill-dev", origin_file_pattern="flux1-fill-dev.safetensors", **vram_config),
        ModelConfig(model_id="black-forest-labs/FLUX.1-Fill-dev", origin_file_pattern="text_encoder/model.safetensors", **vram_config),
        ModelConfig(model_id="black-forest-labs/FLUX.1-Fill-dev", origin_file_pattern="text_encoder_2/*.safetensors", **vram_config),
        ModelConfig(model_id="black-forest-labs/FLUX.1-Fill-dev", origin_file_pattern="ae.safetensors", **vram_config),
    ],
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5,
)

dataset_snapshot_download(
    dataset_id="HuanJue/example_dataset",
    local_dir="./",
    allow_file_pattern=f"FLUX.1-Fill-dev/*",
)

flux_fill_image = Image.open("FLUX.1-Fill-dev/cup.png").convert("RGB")
flux_fill_mask = Image.open("FLUX.1-Fill-dev/cup_mask.png").convert("L")
prompt = "a white paper cup"
image = pipe(prompt=prompt, flux_fill_image=flux_fill_image, flux_fill_mask=flux_fill_mask, height=1632, width=1232, seed=0, embedded_guidance=30.0, num_inference_steps=50)

image.save("image_FLUX.1-Fill-dev.jpg")

Inputs and Output

• Inputs:
  • cup.png: Source image of a paper cup placed on sand.
  • cup_mask.png: Mask marking the region to regenerate.
  • Prompt: a white paper cup
• Output:

The model regenerates a white paper cup with a sun-like pattern inside the masked area, keeping the surrounding sand and lighting consistent.

---

2. FLUX.1-Redux-dev: Image Semantic Transfer

Overview

FLUX.1-Redux-dev is designed for image semantic transfer (image variation / style and structure transfer). Given a reference image, the model extracts high-level semantic information and generates a new image that preserves the subject, composition, or style—without requiring a text prompt.

Example Script

examples/flux/model_inference_low_vram/FLUX.1-Redux-dev.py

import torch
from diffsynth.pipelines.flux_image import FluxImagePipeline, ModelConfig
from PIL import Image
from modelscope import dataset_snapshot_download

vram_config = {
    "offload_dtype": torch.float8_e4m3fn,
    "offload_device": "cpu",
    "onload_dtype": torch.float8_e4m3fn,
    "onload_device": "cpu",
    "preparing_dtype": torch.float8_e4m3fn,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}

pipe = FluxImagePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="flux1-dev.safetensors", **vram_config),
        ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="text_encoder/model.safetensors", **vram_config),
        ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="text_encoder_2/*.safetensors", **vram_config),
        ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="ae.safetensors", **vram_config),
        ModelConfig(model_id="black-forest-labs/FLUX.1-Redux-dev", origin_file_pattern="image_encoder/model.safetensors", **vram_config),
        ModelConfig(model_id="black-forest-labs/FLUX.1-Redux-dev", origin_file_pattern="image_embedder/diffusion_pytorch_model.safetensors", **vram_config),
    ],
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5,
)
dataset_snapshot_download(
    dataset_id="HuanJue/example_dataset",
    local_dir="./",
    allow_file_pattern=f"FLUX.1-Redux-dev/*",
)

flux_redux_image = Image.open("FLUX.1-Redux-dev/robot.png").convert("RGB")

image = pipe(flux_redux_image=flux_redux_image, embedded_guidance=2.5, num_inference_steps=50)

image.save("image_FLUX.1-Redux-dev.jpg")

Inputs and Output

• Input:
  • robot.png: Reference image of a red cartoon robot standing next to a podium.
• Output:

The generated result preserves the red robot, podium, and antenna as core semantic elements, while producing a reasonable re-interpretation of pose and detail.

---

3. Insert-Anything: Arbitrary Object Insertion

Overview

Insert-Anything is a community LoRA built on FLUX for "insert any object into any scene". It takes four inputs:

• Source image: The image to be edited.
• Source mask: The region in the source image to be replaced/filled.
• Reference image: The image containing the object to insert.
• Reference mask: The precise mask of the object in the reference image.

The model blends the reference object into the masked region of the source image while matching lighting, perspective, and style.

Example Script

examples/flux/model_inference_low_vram/Insert-Anything.py

import torch
from diffsynth.pipelines.flux_image import FluxImagePipeline, ModelConfig
from PIL import Image
from modelscope import dataset_snapshot_download

vram_config = {
    "offload_dtype": torch.float8_e4m3fn,
    "offload_device": "cpu",
    "onload_dtype": torch.float8_e4m3fn,
    "onload_device": "cpu",
    "preparing_dtype": torch.float8_e4m3fn,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}

pipe = FluxImagePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="black-forest-labs/FLUX.1-Fill-dev", origin_file_pattern="flux1-fill-dev.safetensors", **vram_config),
        ModelConfig(model_id="black-forest-labs/FLUX.1-Fill-dev", origin_file_pattern="text_encoder/model.safetensors", **vram_config),
        ModelConfig(model_id="black-forest-labs/FLUX.1-Fill-dev", origin_file_pattern="text_encoder_2/*.safetensors", **vram_config),
        ModelConfig(model_id="black-forest-labs/FLUX.1-Fill-dev", origin_file_pattern="ae.safetensors", **vram_config),
        ModelConfig(model_id="black-forest-labs/FLUX.1-Redux-dev", origin_file_pattern="image_encoder/model.safetensors", **vram_config),
        ModelConfig(model_id="black-forest-labs/FLUX.1-Redux-dev", origin_file_pattern="image_embedder/diffusion_pytorch_model.safetensors", **vram_config),
    ],
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5,
)

pipe.load_lora(pipe.dit, ModelConfig(model_id="HuanJue/Insert-Anything", origin_file_pattern="20250321_steps5000_pytorch_lora_weights.safetensors"))

dataset_snapshot_download(
    dataset_id="HuanJue/example_dataset",
    local_dir="./",
    allow_file_pattern=f"Insert-Anything/*",
)

source_image = Image.open("Insert-Anything/source_image.png").convert("RGB")
source_mask = Image.open("Insert-Anything/source_mask.png").convert("L")
ref_image = Image.open("Insert-Anything/ref_image.png").convert("RGB")
ref_mask = Image.open("Insert-Anything/ref_mask.png").convert("L")

seed = 666

image = pipe(
    insert_anything_source_image=source_image,
    insert_anything_source_mask=source_mask,
    insert_anything_ref_image=ref_image,
    insert_anything_ref_mask=ref_mask,
    seed=seed,
    embedded_guidance=30.0,
    num_inference_steps=50,
)

image.save("image_Insert-Anything.jpg")

Inputs and Output

• Inputs:
  • source_image.png: Source image (e.g., a person photo).
  • source_mask.png: Mask of the region to replace in the source image (e.g., the jersey area).
  • ref_image.png: Reference image (e.g., another jersey).
  • ref_mask.png: Mask of the object to insert from the reference image.
• Output:

The output naturally fuses the reference object onto the person in the source image, preserving pose, skin tone, and background while performing the garment replacement insertion.

---