Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
7969840
[modular] derive required inputs and defaults from the ordered block …
yiyixuxu Oct 27, 2026
7ab5f5f
[modular] a sequence keeps the first default and warns on a later dif…
yiyixuxu Nov 6, 2026
8ca4741
[modular] sync class docstrings with the per-branch default wording
yiyixuxu Nov 6, 2026
3bedb58
[modular] docstring says "depends on the workflow" for a per-branch i…
yiyixuxu Nov 8, 2026
3638fd1
[modular] P-H3 Image: declare height and width on the pipeline
yiyixuxu Nov 8, 2026
dc5c91b
[modular] regenerate auto docstrings for the workflow-dependent wording
yiyixuxu Nov 8, 2026
c266754
[docs] modular: how a composite derives inputs and when to use `*_ove…
yiyixuxu Nov 8, 2026
b678487
[modular] HunyuanVideo 1.5: expand text embeddings in the input step,…
yiyixuxu Nov 13, 2026
b7e3831
[docs] modular: tighten the composite inputs wording
yiyixuxu Nov 13, 2026
3a9f1ca
[modular] remove `intermediate_output_overrides`
yiyixuxu Nov 13, 2026
649bc13
[modular] a skippable block nested in a sub-block still only maybe pr…
yiyixuxu Nov 14, 2026
d5e6150
[modular] `*_overrides` are lists of declarations
yiyixuxu Nov 14, 2026
d3408bf
[modular] replace composite `*_overrides` with `ModularPipelineBlocks…
yiyixuxu Nov 15, 2026
cecf77c
[modular] `get_execution_blocks` keeps the outputs of the full blocks
yiyixuxu Nov 15, 2026
71336b5
add tests for checking optional and required inputs for modular
sayakpaul Oct 26, 2026
5d9d639
[modular tests] require the canonical required inputs and that requir…
yiyixuxu Nov 16, 2026
f7b4b80
[modular] rewrap a docstring line in `new()` for doc-builder
yiyixuxu Nov 16, 2026
85ae9db
Merge remote-tracking branch 'origin/main' into sequential-required-i…
yiyixuxu Nov 16, 2026
3597380
[modular] regenerate PAN 2 and MiniMax H3 Image docstrings after merg…
yiyixuxu Nov 16, 2026
4aeab7b
[modular] the Flux, SD3, SDXL, Wan and Z-Image text encoders require …
yiyixuxu Nov 17, 2026
0734863
[modular tests] Flux2, Ernie-Image and HunyuanVideo 1.5 require no input
yiyixuxu Nov 17, 2026
3e1bf4a
[modular tests] `batch_params` defaults to `prompt`; the mixin skips …
yiyixuxu Nov 17, 2026
a905d88
[modular tests] drop `params`; `test_pipeline_not_parameters` keeps t…
yiyixuxu Nov 17, 2026
ed87967
[modular tests] `optional_params` overrides compose from `OPTIONAL_PA…
yiyixuxu Nov 17, 2026
fcb564b
[modular] note that QwenImage's auto text encoder is kept for backwar…
yiyixuxu Nov 17, 2026
6dec230
[modular] a skippable block that rewrites its own input does not supp…
yiyixuxu Nov 17, 2026
0556f3d
[docs] modular guide: `new()` gets its own section, composite declara…
yiyixuxu Nov 17, 2026
ddbc216
[docs] modular: simplify the composite declarations and conflicting d…
yiyixuxu Nov 17, 2026
588c282
[modular tests] pipeline-specific inputs go into `optional_params` or…
yiyixuxu Nov 18, 2026
2a04bc6
[docs] modular: where to change a default on a composite, `new()` rep…
yiyixuxu Nov 20, 2026
ba48723
[modular tests] `conditional_blocks` takes the sub-blocks directly
yiyixuxu Nov 20, 2026
1f73330
[modular tests] inline the `per_branch` test helper
yiyixuxu Nov 20, 2026
6c3f206
[modular tests] build real nested blocks in `test_nested_requirements…
yiyixuxu Nov 20, 2026
3dfd648
[modular tests] plain block classes for the optional-output tests; `f…
yiyixuxu Nov 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 51 additions & 12 deletions .ai/references/modular.md
Original file line number Diff line number Diff line change
Expand Up @@ -269,7 +269,7 @@ OutputParam(
)
```

If a template's predefined description doesn't fit (e.g. the `"latents"` output template means "Denoised latents", which is wrong for the noisy latents out of a prepare-latents step) — drop the template and declare the field directly with an accurate description. See gotcha #5.
If a template's predefined description doesn't fit (e.g. the `"latents"` output template means "Denoised latents", which is wrong for the noisy latents out of a prepare-latents step) — keep the template for the name and type hint and pass your own `description=`, or declare the field directly. See gotcha #5.

**Declare defaults in the `InputParam`, not inside `__call__`.**

Expand All @@ -282,20 +282,23 @@ if block_state.num_frames is None:
block_state.num_frames = 189
```

A declared default is part of the block's contract, so the assembled pipeline is aware of it: the generated docstring shows it and `default_call_parameters` reports it. Resolved inside the body instead, the input renders as `*optional*` with no default, and nothing at the pipeline level can report what the block will actually do. Don't worry about branches of a conditional blockset declaring different defaults for the same input — each branch resolves its own at runtime. Resolve inside `__call__` only when the default is *computed* — derived from other inputs or component config (`height = components.default_sample_size * components.vae_scale_factor`). And when several blocks in a sequence share an input, declare the same default on each (or only on the first block that reads it): in a sequence the input is one shared value, so disagreeing declarations are silently resolved first-block-wins.

**A composed `SequentialPipelineBlocks` can override `inputs` / `outputs`.** Two uses: narrow `outputs` to what downstream actually consumes, so the docstring shows the step's product instead of every internal intermediate (Wan-Animate-2's core denoise exposes only `segment_frames`); and change one input default per preset by mapping over `super().inputs` (the distilled core denoise turns `num_inference_steps` into `default=10`):
A declared default is part of the block's contract: it shows in the generated docstring and in `default_call_parameters`; one resolved in the body does not. Resolve inside `__call__` only when the default is *computed*, derived from other inputs or component config:

```python
@property
def inputs(self):
# The distilled checkpoint samples in few steps.
return [
InputParam.template("num_inference_steps", default=10) if param.name == "num_inference_steps" else param
for param in super().inputs
]
if block_state.height is None:
block_state.height = components.default_sample_size * components.vae_scale_factor
```

The other two ways to declare an input:
(1) `required=True` when the block cannot work without the value, so the caller has to pass it; the block raises `ValueError("Required input 'x' is missing")` when it is not passed or `None`.
(2) Optional with no default when the block works with `None`. Most often that is the computed case above, and the block should then also declare the value as an output.

A composite (`SequentialPipelineBlocks`, `ConditionalPipelineBlocks`, `LoopSequentialPipelineBlocks`) derives its inputs and intermediate outputs from its sub-blocks. They are part of each sub-block's logic, the values its `__call__` reads and writes, so they can't be added or removed on the composite. Don't override `inputs` or `intermediate_outputs` there: to change the default of an input, create the sub-block with the default you need using `new()`. See "Reusing a block with `new()`" below.

`outputs` is different: it is what the composite presents as its product. By default it is every intermediate output. Override the property to list only what the step is for, so the docstring shows the product instead of every internal intermediate (most core denoise steps list only `latents`). It changes the docstring and `output_names` only: everything stays in the pipeline state, and `pipe(..., output=<name>)` still fetches any intermediate, listed or not.

When two sub-blocks of a sequence declare different defaults for the same input, the sequence uses the first one. The best practice is to make sure every sub-block you use has the intended default (use `new()` where it does not). An input a sub-block requires can show as optional on the composite, when an earlier sub-block gives it a default or may produce it; that is expected.

## ComponentSpec patterns

```python
Expand All @@ -312,6 +315,40 @@ ComponentSpec(
)
```

A composite inherits its sub-blocks' `ComponentSpec`s and `ConfigSpec`s. To point one at another repo, loading method or config value for the assembled pipeline, see "Reusing a block with `new()`" below.

## Reusing a block with `new()`

A composite's inputs, intermediate outputs, components and configs all derive from its sub-blocks, so you should change the declaration on the sub-block, not on the composite.

A common case: a block does what you need, but one of its declarations does not fit, e.g. the default or description of an input, the repo and loading info of a component, the value of a config. Don't copy or subclass the block for that, create it with `new()`:

```python
class DistilledBlocks(SequentialPipelineBlocks):
block_classes = [
TextEncoderStep,
# the distilled checkpoint samples in few steps
SetTimestepsStep().new(inputs=[InputParam.template("num_inference_steps", default=10)]),
# it has its own transformer weights and skips guidance
DenoiseStep().new(
components=[
ComponentSpec("transformer", YourTransformerModel, repo="org/model-distilled", subfolder="transformer")
],
configs=[ConfigSpec("is_distilled", default=True)],
),
DecodeStep,
]
block_names = ["text_encoder", "set_timesteps", "denoise", "decode"]
```

`block.new(description=..., inputs=[...], intermediate_outputs=[...], outputs=[...], components=[...], configs=[...])` returns a copy of the block where the entries you pass override the block's own, matched by name. Entries you don't list are unchanged; an entry you do list is replaced whole (gotcha #13). Typical uses: set a preset's default or checkpoint as above, settle a default two sub-blocks disagree on, or reword a description.

What `new()` does not do:

- It cannot add or remove an entry, only replace one the block already declares.
- It only works on a block without sub-blocks.
- It does not change the block's code. A block that reads other inputs or runs other code is a block of its own, not a `new()` of another (gotcha #13).

## Gotchas

1. **Importing from standard pipelines.** The modular and standard pipeline systems are parallel — modular blocks must not import from `diffusers.pipelines.*`. For shared utility methods (e.g. `_pack_latents`, `retrieve_timesteps`), either redefine as standalone functions or use `# Copied from diffusers.pipelines.<model>...` headers. See `wan/before_denoise.py` and `helios/before_denoise.py` for examples.
Expand All @@ -322,7 +359,7 @@ ComponentSpec(

4. **Instantiating components inline.** If a class like `VideoProcessor` is needed, register it as a `ComponentSpec` and access via `components.video_processor`. Don't create new instances inside block `__call__`.

5. **Using `InputParam.template()` / `OutputParam.template()` when semantics don't match.** Templates carry predefined descriptions — e.g. the `"latents"` output template means "Denoised latents". Don't use it for initial noisy latents from a prepare-latents step. Use a plain `InputParam(...)` / `OutputParam(...)` with an accurate description instead.
5. **Using `InputParam.template()` / `OutputParam.template()` when semantics don't match.** Templates carry predefined descriptions — e.g. the `"latents"` output template means "Denoised latents". Don't use it as is for initial noisy latents from a prepare-latents step: keep the template for the name and type hint and pass `description=` (`OutputParam.template("latents", description="Initial noisy latents")`), or use a plain `InputParam(...)` / `OutputParam(...)`.

6. **`required=True` on an input that carries a default.** An input cannot be both required and have a default. When that happens, `InputParam` keeps the default, treats the input as optional and logs a warning. This is likely to happen when you reuse a template for its type hint and description but make it required: `InputParam.template("num_inference_steps", required=True)` stays an optional input that defaults to 50, plus a warning. Decide which one you mean: drop `required=True` to keep the template default without the warning, or pass `default=None` explicitly to make the input required.

Expand All @@ -341,6 +378,8 @@ ComponentSpec(

12. **Helpers taking `components`.** Prefer explicit args, for example, `get_xx_prompt_embeds(text_encoder, tokenizer, prompt)` instead of `get_xx_prompt_embeds(components, prompt)`. Helpers and block methods should take only what they use so dependencies are visible and they can run without an assembled pipeline. Leave the full `components` object to `__call__`, which unpacks what its helpers need. See `get_qwen_prompt_embeds` in `qwenimage/encoders.py`.

13. **`new()` replaces a whole entry, not a field.** List only the entries you want to change; the others stay as the block declared them. But each entry you list replaces the block's entry completely: `block.new(inputs=[InputParam("num_inference_steps", default=10)])` loses the type hint and description the block had. Build the entry from the template, `InputParam.template("num_inference_steps", default=10)`, or write out every field.

## Conversion checklist

- [ ] Read original pipeline's `__call__` end-to-end, map stages
Expand Down
37 changes: 24 additions & 13 deletions docs/source/en/modular_diffusers/sequential_pipeline_blocks.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,23 +110,34 @@ class ImageProcessingStep(SequentialPipelineBlocks):
" - Encodes the image into latent space."
)
```
When you create a [`~modular_pipelines.SequentialPipelineBlocks`], `inputs`, `intermediate_outputs`, `expected_components` and `expected_configs` are aggregated from the sub-blocks, so there is no need to define them again. The sequence cannot add, remove or redefine any of them: they belong to the sub-blocks. To reuse a block with another default for an input, another description, or other repo info for a component, create the sub-block with `new()`. It returns a copy of the block with the listed declarations replaced, matched by name; the block's code is unchanged.

When you create a [`~modular_pipelines.SequentialPipelineBlocks`], properties like `inputs`, `intermediate_outputs`, and `expected_components` are automatically aggregated from the sub-blocks, so there is no need to define them again.
```py
class ImageProcessingStep(SequentialPipelineBlocks):
"""
# auto_docstring
"""
model_name = "my_model"
block_classes = [
# `InputBlock` declares `num_images_per_prompt` without a default; this step defaults it to 1
InputBlock().new(
inputs=[
InputParam(
name="num_images_per_prompt", type_hint=int, default=1, description="number of images per prompt"
)
]
),
ImageEncoderBlock(),
]
block_names = ["input", "image_encoder"]
```

When two sub-blocks declare different defaults for the same input, the sequence uses the first one. To remove the ambiguity, make sure every sub-block that declares the input has the intended default, creating it with `new()` where it does not.

There are a few properties you should set:
A few properties you set yourself:

- `description`: We recommend adding a description for the assembled block to explain what the combined step does.
- `model_name`: This is automatically derived from the sub-blocks but isn't always correct, so you may need to override it.
- `outputs`: By default this is the same as `intermediate_outputs`, but you can manually set it to control which values appear in the doc. This is useful for showing only the final outputs instead of all intermediate values.
- `outputs`: By default this is the same as `intermediate_outputs`. Set it to list only the values the step is for, so the doc shows the final outputs instead of every intermediate value. Everything stays in the pipeline state either way. The blocks `get_execution_blocks` and `get_workflow` return keep this list too, minus any output their selected blocks don't produce.

These properties, together with the aggregated `inputs`, `intermediate_outputs`, and `expected_components`, are used to automatically generate the `doc` property.


Print the `ImageProcessingStep` block to inspect its sub-blocks, and use `doc` for a full summary of the block's inputs, outputs, and components.


```py
blocks = ImageProcessingStep()
print(blocks)
print(blocks.doc)
```
16 changes: 8 additions & 8 deletions src/diffusers/modular_pipelines/anima/modular_blocks_anima.py
Original file line number Diff line number Diff line change
Expand Up @@ -230,21 +230,21 @@ class AnimaAutoCoreDenoiseStep(AutoPipelineBlocks):
Negative T5 prompt attention mask generated by the text encoder step.
num_images_per_prompt (`int`, *optional*, defaults to 1):
The number of images to generate per prompt.
image_latents (`Tensor`, *optional*):
image_latents (`Tensor`, *optional*, depends on the workflow):
image latents used to guide the image generation. Can be generated from vae_encoder step.
height (`int`, *optional*):
The height in pixels of the generated image.
width (`int`, *optional*):
The width in pixels of the generated image.
num_inference_steps (`int`):
num_inference_steps (`int`, *optional*, defaults to 50):
The number of denoising steps.
sigmas (`list`, *optional*):
Custom sigmas for the denoising process.
strength (`float`, *optional*, defaults to 0.9):
Strength for img2img/inpainting.
generator (`Generator`, *optional*):
Torch generator for deterministic generation.
latents (`Tensor`):
latents (`Tensor`, *optional*):
Pre-generated noisy latents for image generation.
**denoiser_input_fields (`None`, *optional*):
The conditional model inputs for the Anima denoiser.
Expand Down Expand Up @@ -278,7 +278,7 @@ class AnimaAutoVaeImageEncoderStep(AutoPipelineBlocks):
vae (`AutoencoderKLQwenImage`) image_processor (`VaeImageProcessor`)

Inputs:
image (`Image | list`, *optional*):
image (`Image | list`, *optional*, depends on the workflow):
Reference image(s) for denoising. Can be a single image or list of images.
height (`int`, *optional*):
The height in pixels of the generated image.
Expand Down Expand Up @@ -330,7 +330,7 @@ class AnimaAutoBlocks(SequentialPipelineBlocks):
The prompt or prompts not to guide the image generation.
max_sequence_length (`int`, *optional*, defaults to 512):
Maximum sequence length for prompt encoding.
image (`Image | list`, *optional*):
image (`Image | list`, *optional*, depends on the workflow):
Reference image(s) for denoising. Can be a single image or list of images.
height (`int`, *optional*):
The height in pixels of the generated image.
Expand All @@ -340,15 +340,15 @@ class AnimaAutoBlocks(SequentialPipelineBlocks):
Torch generator for deterministic generation.
num_images_per_prompt (`int`, *optional*, defaults to 1):
The number of images to generate per prompt.
image_latents (`Tensor`, *optional*):
image_latents (`Tensor`, *optional*, depends on the workflow):
image latents used to guide the image generation. Can be generated from vae_encoder step.
num_inference_steps (`int`):
num_inference_steps (`int`, *optional*, defaults to 50):
The number of denoising steps.
sigmas (`list`, *optional*):
Custom sigmas for the denoising process.
strength (`float`, *optional*, defaults to 0.9):
Strength for img2img/inpainting.
latents (`Tensor`):
latents (`Tensor`, *optional*):
Pre-generated noisy latents for image generation.
**denoiser_input_fields (`None`, *optional*):
The conditional model inputs for the Anima denoiser.
Expand Down
2 changes: 0 additions & 2 deletions src/diffusers/modular_pipelines/cosmos/denoise.py
Original file line number Diff line number Diff line change
Expand Up @@ -993,8 +993,6 @@ class Cosmos3TransferDenoiseStep(Cosmos3DenoiseLoopWrapper):
Timestep interval [lo, hi] over which text guidance is active (None = always).
control_guidance_interval (`tuple`, *optional*):
Timestep interval [lo, hi] over which control guidance is active (None = always).
latents (`Tensor`):
Noisy target latents to update.
condition_latents (`Tensor`):
Clean target latents on the conditioned frames (the autoregressive seed).

Expand Down
Loading