Skip to content

Ulysses Anything Attention misaligns the attention mask when two or more ranks hold a shorter shard #15310

Closed as not planned

Description

@klin2024

Describe the bug

With Ulysses Anything Attention (ulysses_degree >= 3, ulysses_anything=True) and an attention mask, the output differs from the single-device result whenever two or more ranks hold a shorter sequence shard. No error is raised.

TemplatedUlyssesAnythingAttention.forward pads each rank's local mask to the longest local length, all-gathers, concatenates, and keeps the first sum(mask_local_sizes) entries. Shards come from tensor_split, so the shorter ranks are the last ones. With one shorter rank the padding sits at the end and is trimmed correctly. With two or more, the padding of the earlier short ranks lands in the middle: for 10 tokens on 3 ranks ([4, 3, 3]) the concatenated mask is [r0 x4][r1 x3, pad][r2 x3, pad], and taking the first 10 entries keeps rank 1's padding and drops rank 2's last entry, so the mask is shifted against the keys. Two ranks never hit this, which is the setup the existing context-parallel tests use.

I plan to open a PR that trims each gathered chunk to its own size before concatenating.

Reproduction

A tiny QwenImageTransformer2DModel (2 layers, 12 heads, head dim 16, random weights, 36 image tokens) with a text padding mask (encoder_hidden_states_mask[:, 7:] = 0), run with ContextParallelConfig(ulysses_degree=world_size, ulysses_anything=True) and compared with the same model without parallelism, fp32, torchrun --nproc-per-node N:

GPUs text tokens -> shards max abs error
2 10 -> [5, 5], 11 -> [6, 5] 0, 0
3 10 -> [4, 3, 3] 0.0044
3 11 -> [4, 4, 3], 12 -> [4, 4, 4] 0, 0
4 13 -> [4, 3, 3, 3] 0.0073
4 14 -> [4, 4, 3, 3] 0.0035
4 12 -> [3, 3, 3, 3], 15 -> [4, 4, 4, 3] 0, 0

(output max is about 2.2)

The same happens at the op level: dispatch_attention_fn on 3 CPU ranks (cpu:gloo) with 10 tokens and a boolean mask gives a max abs error of 1.14 against full-sequence scaled_dot_product_attention.

Logs

No error; the output is wrong.

System Info

  • Diffusers version: 0.44.0.dev0 (main, 1dabc29)
  • Platform: Linux-6.8.0-138-generic-x86_64-with-glibc2.35
  • Python version: 3.12.4
  • PyTorch version (GPU?): 2.14.1+cu130 (True)
  • Accelerator: NVIDIA H100 80GB HBM3
  • Using GPU in script?: yes
  • Using distributed or parallel set-up in script?: yes, context parallelism (Ulysses Anything) on 2 to 4 GPUs via torchrun

Who can help?

@sayakpaul @zhtmike

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions