Request support for MAGI-1

[lavinal712]

Model/Pipeline/Scheduler description

MAGI-1 is a video generation model that has achieved stunning visual effects.

Open source status

• The model implementation is available.
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

https://github.com/SandAI-org/MAGI-1
https://github.com/SandAI-org/MagiAttention

[kuantuna]

Hi, just checking in to see if there are any plans to integrate MAGI-1 into diffusers. I’d love to use it within the library. Any updates or thoughts on this would be great to hear!

[yiyixuxu]

it is not diffusion model so we won't be supporting it for now

[lavinal712]

I think it use diffusion method (or flow matching) in sampling. Am I wrong?

[a-r-r-o-w]

@yiyixuxu It is essentially a flow matching model but different chunks of frames are generated autoregressively once the previous chunk reaches a certain noise threshold. If we wanted to compare against a model that was similar to MAGI, I would say it's like Framepack.

In framepack, we generate the video in chunks as well, but the denoising of each chunk starts after previous one fully finishes (the relation between past chunks and current chunk is established via the history latents).

In magi, it's somewhat similar, but the relation between current chunk with previous chunks is established with a causal mask, after previous chunks reach certain noise level during the denoising process

[yiyixuxu]

ohh thanks for explaining! so it is still fundamentally a denoising process. We can look into support then if it can fit into our existing model/pipeline design

[nitinmukesh]

MAGI-1 4.5B distill and distill+quant models are now released
https://huggingface.co/sand-ai/MAGI-1/tree/main/ckpt/magi

[tolgacangoz]

Looks like there seem to be some similarities with SkyReels-V2, too. SkyReels-V2 utilizes Diffusion Forcing to stay autoregressive, keeping things rolling with a delay between chunks (like ar_step=5, causal_block_size=5). It ties past and future chunks with partially denoised latents + causal mask.

If MAGI-1 is proper for diffusers and contributions-welcome, I'd love to work on porting MAGI-1 to diffusers.

[yiyixuxu]

hi @tolgacangoz sounds good!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[lavinal712]

#14760