Skip to content

[docs] Context parallelism guide only parallelizes transformer; Wan 2.2 A14B's transformer_2 stays unsplit #15317

Closed

Description

@menglcai

Is your feature request related to a problem? Please describe.
Wan 2.2 A14B pipelines (WanPipeline, WanImageToVideoPipeline) have two transformers. transformer runs the high-noise steps and transformer_2 runs the low-noise steps. Every example in the context parallelism guide calls pipeline.transformer.enable_parallelism(...), and the guide doesn't say that enable_parallelism only applies to the model it's called on. If you follow it, the transformer_2 steps are not split, and every GPU runs them in full. With the default 40 steps and boundary_ratio=0.9, transformer_2 runs 29 of the 40 steps. The output is still correct, so nothing tells the user that most of the speedup is missing.

Full pipeline call, Wan-AI/Wan2.2-I2V-A14B-Diffusers, bf16, 480x832, 33 frames, 40 steps, Ulysses degree 2, 2 GCDs of one AMD MI250 (ROCm, perf level auto). The two configs were interleaved (A B A B).

CP enabled on run 1 run 2
transformer only (current docs) 365.1 s 363.8 s
transformer and transformer_2 229.4 s 237.0 s

Describe the solution you'd like.
Add a short note to the context parallelism guide, docs/source/en/optimization/context_parallelism.md, right after the Ring Attention example, which every other variant section points back to. It explains that enable_parallelism only applies to the model it's called on, and shows how to enable it on both Wan 2.2 A14B transformers:

for transformer in (pipeline.transformer, pipeline.transformer_2):
    transformer.enable_parallelism(config=ContextParallelConfig(ring_degree=world_size))

This is a docs-only change. I have it ready and can open a PR if the maintainers agree with the approach.

Describe alternatives you've considered.
A pipeline-level enable_parallelism() that applies the config to every model with a _cp_plan, similar to enable_group_offload. That adds a new public method to DiffusionPipeline, so I'd leave that decision to the maintainers and start with the docs.

Additional context.
I was testing context parallelism on popular multi-GPU workloads on AMD MI250 when I found this. LoRA (#12074, load_into_transformer_2) and MagCache (#14025) also had to handle Wan 2.2's second transformer.

cc @stevhliu @sayakpaul

Activity

  1. sayakpaul commented on Nov 17, 2026

    @sayakpaul
    Member

    I think context parallelism is an advanced topic anyway and it's not unreasonable to expect the users to have some background of it + how it's integrated in the library. So, I don't think this would add much value but I will defer this @stevhliu.

  2. stevhliu commented on Nov 17, 2026

    @stevhliu
    Member

    I think it's worth a short note, since the issue is more about the API scope than CP background, and its relatively cheap to do so, but I'd make it a bit more general rather than about Wan.

  3. menglcai commented on Nov 18, 2026

    @menglcai
    ContributorAuthor

    Thanks @stevhliu and @sayakpaul! Based on @stevhliu's suggestion, I opened #15334. It adds a short, general note after the Ring Attention example: enable_parallelism only applies to the model it's called on, so call it on each transformer when a pipeline has more than one. Could you please take a look when you get a chance? Happy to change the wording or placement if needed. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions