You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Ulysses Anything Attention misaligns the attention mask when two or more ranks hold a shorter shard #15310
With Ulysses Anything Attention (ulysses_degree >= 3, ulysses_anything=True) and an attention mask, the output differs from the single-device result whenever two or more ranks hold a shorter sequence shard. No error is raised.
TemplatedUlyssesAnythingAttention.forward pads each rank's local mask to the longest local length, all-gathers, concatenates, and keeps the first sum(mask_local_sizes) entries. Shards come from tensor_split, so the shorter ranks are the last ones. With one shorter rank the padding sits at the end and is trimmed correctly. With two or more, the padding of the earlier short ranks lands in the middle: for 10 tokens on 3 ranks ([4, 3, 3]) the concatenated mask is [r0 x4][r1 x3, pad][r2 x3, pad], and taking the first 10 entries keeps rank 1's padding and drops rank 2's last entry, so the mask is shifted against the keys. Two ranks never hit this, which is the setup the existing context-parallel tests use.
I plan to open a PR that trims each gathered chunk to its own size before concatenating.
Reproduction
A tiny QwenImageTransformer2DModel (2 layers, 12 heads, head dim 16, random weights, 36 image tokens) with a text padding mask (encoder_hidden_states_mask[:, 7:] = 0), run with ContextParallelConfig(ulysses_degree=world_size, ulysses_anything=True) and compared with the same model without parallelism, fp32, torchrun --nproc-per-node N:
GPUs
text tokens -> shards
max abs error
2
10 -> [5, 5], 11 -> [6, 5]
0, 0
3
10 -> [4, 3, 3]
0.0044
3
11 -> [4, 4, 3], 12 -> [4, 4, 4]
0, 0
4
13 -> [4, 3, 3, 3]
0.0073
4
14 -> [4, 4, 3, 3]
0.0035
4
12 -> [3, 3, 3, 3], 15 -> [4, 4, 4, 3]
0, 0
(output max is about 2.2)
The same happens at the op level: dispatch_attention_fn on 3 CPU ranks (cpu:gloo) with 10 tokens and a boolean mask gives a max abs error of 1.14 against full-sequence scaled_dot_product_attention.
Activity