From 660dd509176300677ef086f10d85e16eafe635f1 Mon Sep 17 00:00:00 2001 From: menglcai Date: Wed, 18 Nov 2026 15:02:33 +0800 Subject: [PATCH 1/2] [docs] Note that enable_parallelism only applies to the model it's called on The context parallelism guide only calls `pipeline.transformer.enable_parallelism(...)`. A pipeline with more than one transformer, such as Wan 2.2 A14B with `transformer` and `transformer_2`, needs it on each one, or the steps the other transformers run aren't split across GPUs. Add a short note after the Ring Attention example. Signed-off-by: menglcai --- docs/source/en/optimization/context_parallelism.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/source/en/optimization/context_parallelism.md b/docs/source/en/optimization/context_parallelism.md index 429e64b9e9f2..473d2be589e5 100644 --- a/docs/source/en/optimization/context_parallelism.md +++ b/docs/source/en/optimization/context_parallelism.md @@ -92,6 +92,9 @@ torchrun --nproc-per-node 2 ring_attention.py To try the other variants, replace the `enable_parallelism` call in `ring_attention.py` with the one from their section. +> [!NOTE] +> [`~ModelMixin.enable_parallelism`] only applies to the model you call it on. If a pipeline has more than one transformer, such as `transformer` and `transformer_2`, call it on each one. Otherwise, the steps the other transformers run aren't split across GPUs. + ## Ulysses Attention [Ulysses Attention](https://huggingface.co/papers/2309.14509) uses *all-to-all* communication, where every GPU exchanges data with every other GPU. The first all-to-all regroups the data so each GPU holds every token for a subset of attention heads. Each GPU computes attention for its heads, and a second all-to-all regroups the results by token for the next layer. From 1c4398292577e1a9fa0cef0ef50b85e1033f1daf Mon Sep 17 00:00:00 2001 From: menglcai Date: Thu, 19 Nov 2026 00:42:21 +0800 Subject: [PATCH 2/2] Move the enable_parallelism note to the intro and apply review wording Signed-off-by: menglcai --- docs/source/en/optimization/context_parallelism.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/en/optimization/context_parallelism.md b/docs/source/en/optimization/context_parallelism.md index 473d2be589e5..5c285b2a870a 100644 --- a/docs/source/en/optimization/context_parallelism.md +++ b/docs/source/en/optimization/context_parallelism.md @@ -16,6 +16,9 @@ specific language governing permissions and limitations under the License. Context parallelism runs inside the attention backend, and only some backends support it. If yours doesn't, `enable_parallelism` raises an error that lists the compatible backends. Switch backends with [`~ModelMixin.set_attention_backend`], and see the [available backends](./attention_backends#available-backends) for the full list. +> [!NOTE] +> [`~ModelMixin.enable_parallelism`] only applies to the model you call it on. If a pipeline has more than one transformer, such as `transformer` and `transformer_2`, call it on each one. Otherwise, the denoising steps that use the other transformers aren't split across GPUs. + Configure every variant below with [`ContextParallelConfig`]. | Variant | Config | Requirements | Use when | @@ -92,9 +95,6 @@ torchrun --nproc-per-node 2 ring_attention.py To try the other variants, replace the `enable_parallelism` call in `ring_attention.py` with the one from their section. -> [!NOTE] -> [`~ModelMixin.enable_parallelism`] only applies to the model you call it on. If a pipeline has more than one transformer, such as `transformer` and `transformer_2`, call it on each one. Otherwise, the steps the other transformers run aren't split across GPUs. - ## Ulysses Attention [Ulysses Attention](https://huggingface.co/papers/2309.14509) uses *all-to-all* communication, where every GPU exchanges data with every other GPU. The first all-to-all regroups the data so each GPU holds every token for a subset of attention heads. Each GPU computes attention for its heads, and a second all-to-all regroups the results by token for the next layer.