Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 25 additions & 12 deletions src/diffusers/modular_pipelines/anima/modular_blocks_anima.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,10 @@ class AnimaCoreDenoiseStep(SequentialPipelineBlocks):
Denoise block that takes encoded Anima text inputs and runs the denoising process.

Components:
text_conditioner (`AnimaTextConditioner`) transformer (`CosmosTransformer3DModel`) scheduler
(`FlowMatchEulerDiscreteScheduler`) guider (`ClassifierFreeGuidance`)
- text_conditioner (`AnimaTextConditioner`)
- transformer (`CosmosTransformer3DModel`)
- scheduler (`FlowMatchEulerDiscreteScheduler`)
- guider (`ClassifierFreeGuidance`)

Inputs:
qwen_prompt_embeds (`Tensor`):
Expand Down Expand Up @@ -100,7 +102,8 @@ class AnimaDecodeStep(SequentialPipelineBlocks):
Decode Anima latents into generated images.

Components:
vae (`AutoencoderKLQwenImage`) image_processor (`VaeImageProcessor`)
- vae (`AutoencoderKLQwenImage`)
- image_processor (`VaeImageProcessor`)

Inputs:
latents (`Tensor`):
Expand Down Expand Up @@ -132,8 +135,10 @@ class AnimaImg2ImgCoreDenoiseStep(SequentialPipelineBlocks):
AnimaImg2ImgVaeEncoderStep.

Components:
text_conditioner (`AnimaTextConditioner`) transformer (`CosmosTransformer3DModel`) scheduler
(`FlowMatchEulerDiscreteScheduler`) guider (`ClassifierFreeGuidance`)
- text_conditioner (`AnimaTextConditioner`)
- transformer (`CosmosTransformer3DModel`)
- scheduler (`FlowMatchEulerDiscreteScheduler`)
- guider (`ClassifierFreeGuidance`)

Inputs:
qwen_prompt_embeds (`Tensor`):
Expand Down Expand Up @@ -208,8 +213,10 @@ class AnimaAutoCoreDenoiseStep(AutoPipelineBlocks):
`AnimaImg2ImgCoreDenoiseStep` (img2img) is used when image_latents are present.

Components:
text_conditioner (`AnimaTextConditioner`) transformer (`CosmosTransformer3DModel`) scheduler
(`FlowMatchEulerDiscreteScheduler`) guider (`ClassifierFreeGuidance`)
- text_conditioner (`AnimaTextConditioner`)
- transformer (`CosmosTransformer3DModel`)
- scheduler (`FlowMatchEulerDiscreteScheduler`)
- guider (`ClassifierFreeGuidance`)

Inputs:
qwen_prompt_embeds (`Tensor`):
Expand Down Expand Up @@ -275,7 +282,8 @@ class AnimaAutoVaeImageEncoderStep(AutoPipelineBlocks):
(text-to-image workflow).

Components:
vae (`AutoencoderKLQwenImage`) image_processor (`VaeImageProcessor`)
- vae (`AutoencoderKLQwenImage`)
- image_processor (`VaeImageProcessor`)

Inputs:
image (`Image | list`, *optional*):
Expand Down Expand Up @@ -318,10 +326,15 @@ class AnimaAutoBlocks(SequentialPipelineBlocks):
- `img2img`: requires `image`, `prompt`

Components:
text_encoder (`Qwen3Model`) tokenizer (`Qwen2Tokenizer`) t5_tokenizer (`T5Tokenizer`) guider
(`ClassifierFreeGuidance`) vae (`AutoencoderKLQwenImage`) image_processor (`VaeImageProcessor`)
text_conditioner (`AnimaTextConditioner`) transformer (`CosmosTransformer3DModel`) scheduler
(`FlowMatchEulerDiscreteScheduler`)
- text_encoder (`Qwen3Model`)
- tokenizer (`Qwen2Tokenizer`)
- t5_tokenizer (`T5Tokenizer`)
- guider (`ClassifierFreeGuidance`)
- vae (`AutoencoderKLQwenImage`)
- image_processor (`VaeImageProcessor`)
- text_conditioner (`AnimaTextConditioner`)
- transformer (`CosmosTransformer3DModel`)
- scheduler (`FlowMatchEulerDiscreteScheduler`)

Inputs:
prompt (`str`):
Expand Down
3 changes: 2 additions & 1 deletion src/diffusers/modular_pipelines/cosmos/denoise.py
Original file line number Diff line number Diff line change
Expand Up @@ -963,7 +963,8 @@ class Cosmos3TransferDenoiseStep(Cosmos3DenoiseLoopWrapper):
Runs the per-chunk transfer denoising loop over scheduler timesteps.

Components:
transformer (`Cosmos3OmniTransformer`) scheduler (`UniPCMultistepScheduler`)
- transformer (`Cosmos3OmniTransformer`)
- scheduler (`UniPCMultistepScheduler`)

Inputs:
timesteps (`Tensor`):
Expand Down
54 changes: 36 additions & 18 deletions src/diffusers/modular_pipelines/cosmos/modular_blocks_cosmos3.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ class Cosmos3TransferTextBlocks(SequentialPipelineBlocks):
transfer mode using the per-chunk frame count.

Components:
video_processor (`VideoProcessor`) text_tokenizer (`AutoTokenizer`)
- video_processor (`VideoProcessor`)
- text_tokenizer (`AutoTokenizer`)

Inputs:
control_videos (`dict`):
Expand Down Expand Up @@ -121,7 +122,8 @@ class Cosmos3AutoTextEncoderStep(AutoPipelineBlocks):
- Cosmos3TextEncoderStep runs otherwise.

Components:
video_processor (`VideoProcessor`) text_tokenizer (`AutoTokenizer`)
- video_processor (`VideoProcessor`)
- text_tokenizer (`AutoTokenizer`)

Configs:
default_use_system_prompt (default: True) enable_safety_checker (default: True)
Expand Down Expand Up @@ -204,7 +206,8 @@ class Cosmos3AutoVaeEncoderStep(ConditionalPipelineBlocks):
- when no action, image, or video conditioning is provided, this block is skipped.

Components:
vae (`AutoencoderKLWan`) video_processor (`VideoProcessor`)
- vae (`AutoencoderKLWan`)
- video_processor (`VideoProcessor`)

Inputs:
action (`CosmosActionCondition`, *optional*):
Expand Down Expand Up @@ -280,7 +283,7 @@ class Cosmos3AutoSoundDecodeStep(AutoPipelineBlocks):
- if sound_latents are not provided, this block is skipped.

Components:
sound_tokenizer (`Cosmos3AVAEAudioTokenizer`)
- sound_tokenizer (`Cosmos3AVAEAudioTokenizer`)

Inputs:
sound_latents (`Tensor`, *optional*):
Expand Down Expand Up @@ -313,7 +316,9 @@ class Cosmos3DecodeStep(SequentialPipelineBlocks):
Decodes denoised latents into modality outputs.

Components:
vae (`AutoencoderKLWan`) video_processor (`VideoProcessor`) sound_tokenizer (`Cosmos3AVAEAudioTokenizer`)
- vae (`AutoencoderKLWan`)
- video_processor (`VideoProcessor`)
- sound_tokenizer (`Cosmos3AVAEAudioTokenizer`)

Inputs:
latents (`Tensor`):
Expand Down Expand Up @@ -368,7 +373,8 @@ class Cosmos3VisionCoreDenoiseStep(SequentialPipelineBlocks):
Runs the text-and-vision Cosmos3 denoising workflow.

Components:
transformer (`Cosmos3OmniTransformer`) scheduler (`UniPCMultistepScheduler`)
- transformer (`Cosmos3OmniTransformer`)
- scheduler (`UniPCMultistepScheduler`)

Configs:
use_native_flow_schedule (default: False)
Expand Down Expand Up @@ -448,7 +454,8 @@ class Cosmos3VisionSoundCoreDenoiseStep(SequentialPipelineBlocks):
Runs the text, vision, and sound Cosmos3 denoising workflow.

Components:
transformer (`Cosmos3OmniTransformer`) scheduler (`UniPCMultistepScheduler`)
- transformer (`Cosmos3OmniTransformer`)
- scheduler (`UniPCMultistepScheduler`)

Configs:
use_native_flow_schedule (default: False)
Expand Down Expand Up @@ -541,7 +548,8 @@ class Cosmos3VisionActionCoreDenoiseStep(SequentialPipelineBlocks):
Runs the text, vision, and action Cosmos3 denoising workflow.

Components:
transformer (`Cosmos3OmniTransformer`) scheduler (`UniPCMultistepScheduler`)
- transformer (`Cosmos3OmniTransformer`)
- scheduler (`UniPCMultistepScheduler`)

Configs:
use_native_flow_schedule (default: False)
Expand Down Expand Up @@ -638,7 +646,8 @@ class Cosmos3VisionSoundActionCoreDenoiseStep(SequentialPipelineBlocks):
Runs the text, vision, sound, and action Cosmos3 denoising workflow.

Components:
transformer (`Cosmos3OmniTransformer`) scheduler (`UniPCMultistepScheduler`)
- transformer (`Cosmos3OmniTransformer`)
- scheduler (`UniPCMultistepScheduler`)

Configs:
use_native_flow_schedule (default: False)
Expand Down Expand Up @@ -748,8 +757,10 @@ class Cosmos3TransferChunkDenoiseStep(SequentialPipelineBlocks):
cross-carry (previous_output, output_chunks) lives on PipelineState.

Components:
vae (`AutoencoderKLWan`) video_processor (`VideoProcessor`) transformer (`Cosmos3OmniTransformer`) scheduler
(`UniPCMultistepScheduler`)
- vae (`AutoencoderKLWan`)
- video_processor (`VideoProcessor`)
- transformer (`Cosmos3OmniTransformer`)
- scheduler (`UniPCMultistepScheduler`)

Inputs:
chunk_id (`int`, *optional*, defaults to 0):
Expand Down Expand Up @@ -903,8 +914,10 @@ class Cosmos3TransferCoreDenoiseStep(SequentialPipelineBlocks):
Transfer denoise stage: prepare shared text segments once, then run the autoregressive chunk loop.

Components:
transformer (`Cosmos3OmniTransformer`) vae (`AutoencoderKLWan`) video_processor (`VideoProcessor`) scheduler
(`UniPCMultistepScheduler`)
- transformer (`Cosmos3OmniTransformer`)
- vae (`AutoencoderKLWan`)
- video_processor (`VideoProcessor`)
- scheduler (`UniPCMultistepScheduler`)

Inputs:
cond_input_ids (`None`):
Expand Down Expand Up @@ -1031,8 +1044,10 @@ class Cosmos3AutoCoreDenoiseStep(ConditionalPipelineBlocks):
- vision runs otherwise.

Components:
transformer (`Cosmos3OmniTransformer`) vae (`AutoencoderKLWan`) video_processor (`VideoProcessor`) scheduler
(`UniPCMultistepScheduler`)
- transformer (`Cosmos3OmniTransformer`)
- vae (`AutoencoderKLWan`)
- video_processor (`VideoProcessor`)
- scheduler (`UniPCMultistepScheduler`)

Configs:
use_native_flow_schedule (default: False)
Expand Down Expand Up @@ -1229,9 +1244,12 @@ class Cosmos3OmniBlocks(SequentialPipelineBlocks):
- `action_inverse_dynamics`: requires `prompt`, `action`

Components:
video_processor (`VideoProcessor`) text_tokenizer (`AutoTokenizer`) vae (`AutoencoderKLWan`) transformer
(`Cosmos3OmniTransformer`) scheduler (`UniPCMultistepScheduler`) sound_tokenizer
(`Cosmos3AVAEAudioTokenizer`)
- video_processor (`VideoProcessor`)
- text_tokenizer (`AutoTokenizer`)
- vae (`AutoencoderKLWan`)
- transformer (`Cosmos3OmniTransformer`)
- scheduler (`UniPCMultistepScheduler`)
- sound_tokenizer (`Cosmos3AVAEAudioTokenizer`)

Configs:
default_use_system_prompt (default: True) enable_safety_checker (default: True) use_native_flow_schedule
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@ class Cosmos3DistilledAutoVaeEncoderStep(ConditionalPipelineBlocks):
- when no image or video conditioning is provided, this block is skipped.

Components:
vae (`AutoencoderKLWan`) video_processor (`VideoProcessor`)
- vae (`AutoencoderKLWan`)
- video_processor (`VideoProcessor`)

Inputs:
video (`None`, *optional*):
Expand Down Expand Up @@ -83,7 +84,8 @@ class Cosmos3DistilledVisionCoreDenoiseStep(SequentialPipelineBlocks):
Runs the text-and-vision distilled Cosmos3 denoising workflow.

Components:
transformer (`Cosmos3OmniTransformer`) scheduler (`FlowMatchEulerDiscreteScheduler`)
- transformer (`Cosmos3OmniTransformer`)
- scheduler (`FlowMatchEulerDiscreteScheduler`)

Configs:
is_distilled (default: True) distilled_sigmas (default: None)
Expand Down Expand Up @@ -170,8 +172,11 @@ class Cosmos3DistilledBlocks(SequentialPipelineBlocks):
- `video2video`: requires `prompt`, `video`

Components:
text_tokenizer (`AutoTokenizer`) vae (`AutoencoderKLWan`) video_processor (`VideoProcessor`) transformer
(`Cosmos3OmniTransformer`) scheduler (`FlowMatchEulerDiscreteScheduler`)
- text_tokenizer (`AutoTokenizer`)
- vae (`AutoencoderKLWan`)
- video_processor (`VideoProcessor`)
- transformer (`Cosmos3OmniTransformer`)
- scheduler (`FlowMatchEulerDiscreteScheduler`)

Configs:
default_use_system_prompt (default: True) enable_safety_checker (default: True) is_distilled (default: True)
Expand Down
2 changes: 1 addition & 1 deletion src/diffusers/modular_pipelines/echo/denoise.py
Original file line number Diff line number Diff line change
Expand Up @@ -359,7 +359,7 @@ class EchoDenoiseLoopStep(LoopSequentialPipelineBlocks):
sigma schedule.

Components:
transformer (`LTX2VideoTransformer3DModel`)
- transformer (`LTX2VideoTransformer3DModel`)

Inputs:
sigmas (`list | tuple`):
Expand Down
23 changes: 16 additions & 7 deletions src/diffusers/modular_pipelines/echo/modular_blocks_echo.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,9 @@ class EchoTextConditioningStep(SequentialPipelineBlocks):
Positive-only text conditioning for the guidance-free Echo DMD checkpoint.

Components:
text_encoder (`PreTrainedModel`) tokenizer (`PreTrainedTokenizerBase`) connectors (`LTX2TextConnectors`)
- text_encoder (`PreTrainedModel`)
- tokenizer (`PreTrainedTokenizerBase`)
- connectors (`LTX2TextConnectors`)

Inputs:
prompt (`str`):
Expand Down Expand Up @@ -83,7 +85,7 @@ class EchoDenoiseStep(SequentialPipelineBlocks):
Prepare Echo conditioning and target latents, then run stochastic DMD denoising.

Components:
transformer (`LTX2VideoTransformer3DModel`)
- transformer (`LTX2VideoTransformer3DModel`)

Inputs:
first_frame_latents (`Tensor`, *optional*):
Expand Down Expand Up @@ -166,8 +168,10 @@ class EchoDecoderStep(SequentialPipelineBlocks):
Decode Echo video and audio outputs with mixed-precision-safe audio vocoding.

Components:
vae (`AutoencoderKLLTX2Video`) video_processor (`VideoProcessor`) audio_vae (`AutoencoderKLLTX2Audio`)
vocoder (`LTX2Vocoder`)
- vae (`AutoencoderKLLTX2Video`)
- video_processor (`VideoProcessor`)
- audio_vae (`AutoencoderKLLTX2Audio`)
- vocoder (`LTX2Vocoder`)

Inputs:
latents (`Tensor`):
Expand Down Expand Up @@ -223,9 +227,14 @@ class EchoBlocks(SequentialPipelineBlocks):
stochastic DMD denoising.

Components:
text_encoder (`PreTrainedModel`) tokenizer (`PreTrainedTokenizerBase`) connectors (`LTX2TextConnectors`) vae
(`AutoencoderKLLTX2Video`) audio_vae (`AutoencoderKLLTX2Audio`) transformer (`LTX2VideoTransformer3DModel`)
video_processor (`VideoProcessor`) vocoder (`LTX2Vocoder`)
- text_encoder (`PreTrainedModel`)
- tokenizer (`PreTrainedTokenizerBase`)
- connectors (`LTX2TextConnectors`)
- vae (`AutoencoderKLLTX2Video`)
- audio_vae (`AutoencoderKLLTX2Audio`)
- transformer (`LTX2VideoTransformer3DModel`)
- video_processor (`VideoProcessor`)
- vocoder (`LTX2Vocoder`)

Inputs:
prompt (`str`):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,8 @@ class ErnieImageAutoPromptEnhancerStep(ConditionalPipelineBlocks):
- If `use_pe` is `None` or `False`, the step is skipped.

Components:
pe (`Ministral3ForCausalLM`) pe_tokenizer (`AutoTokenizer`)
- pe (`Ministral3ForCausalLM`)
- pe_tokenizer (`AutoTokenizer`)

Inputs:
prompt (`str`, *optional*):
Expand Down Expand Up @@ -86,8 +87,9 @@ class ErnieImageCoreDenoiseStep(SequentialPipelineBlocks):
Denoise block that takes encoded conditions and runs the denoising process for ErnieImage.

Components:
transformer (`ErnieImageTransformer2DModel`) scheduler (`FlowMatchEulerDiscreteScheduler`) guider
(`ClassifierFreeGuidance`)
- transformer (`ErnieImageTransformer2DModel`)
- scheduler (`FlowMatchEulerDiscreteScheduler`)
- guider (`ClassifierFreeGuidance`)

Inputs:
prompt_embeds (`list`):
Expand Down Expand Up @@ -140,10 +142,16 @@ class ErnieImageAutoBlocks(SequentialPipelineBlocks):
- `text2image`: requires `prompt`

Components:
pe (`Ministral3ForCausalLM`) pe_tokenizer (`AutoTokenizer`) text_encoder (`Mistral3Model`) tokenizer
(`AutoTokenizer`) guider (`ClassifierFreeGuidance`) transformer (`ErnieImageTransformer2DModel`) scheduler
(`FlowMatchEulerDiscreteScheduler`) vae (`AutoencoderKLFlux2`) pachifier (`ErnieImagePachifier`)
image_processor (`VaeImageProcessor`)
- pe (`Ministral3ForCausalLM`)
- pe_tokenizer (`AutoTokenizer`)
- text_encoder (`Mistral3Model`)
- tokenizer (`AutoTokenizer`)
- guider (`ClassifierFreeGuidance`)
- transformer (`ErnieImageTransformer2DModel`)
- scheduler (`FlowMatchEulerDiscreteScheduler`)
- vae (`AutoencoderKLFlux2`)
- pachifier (`ErnieImagePachifier`)
- image_processor (`VaeImageProcessor`)

Inputs:
prompt (`str`, *optional*):
Expand Down
Loading
Loading