Repository navigation
Description
Is your feature request related to a problem? Please describe.
JointAttnProcessor2_0 (SD3 / SD3.5, SD3 ControlNet) transposes q/k to [B, H, T, D] and then applies norm_q / norm_k / norm_added_q / norm_added_k. So the QK norm reads a strided, non-contiguous tensor:
diffusers/src/diffusers/models/attention_processor.py
Lines 1450 to 1478 in 43e3002
| query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2) | |
| key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2) | |
| value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2) | |
| if attn.norm_q is not None: | |
| query = attn.norm_q(query) | |
| if attn.norm_k is not None: | |
| key = attn.norm_k(key) | |
| # `context` projections. | |
| if encoder_hidden_states is not None: | |
| encoder_hidden_states_query_proj = attn.add_q_proj(encoder_hidden_states) | |
| encoder_hidden_states_key_proj = attn.add_k_proj(encoder_hidden_states) | |
| encoder_hidden_states_value_proj = attn.add_v_proj(encoder_hidden_states) | |
| encoder_hidden_states_query_proj = encoder_hidden_states_query_proj.view( | |
| batch_size, -1, attn.heads, head_dim | |
| ).transpose(1, 2) | |
| encoder_hidden_states_key_proj = encoder_hidden_states_key_proj.view( | |
| batch_size, -1, attn.heads, head_dim | |
| ).transpose(1, 2) | |
| encoder_hidden_states_value_proj = encoder_hidden_states_value_proj.view( | |
| batch_size, -1, attn.heads, head_dim | |
| ).transpose(1, 2) | |
| if attn.norm_added_q is not None: | |
| encoder_hidden_states_query_proj = attn.norm_added_q(encoder_hidden_states_query_proj) | |
| if attn.norm_added_k is not None: | |
| encoder_hidden_states_key_proj = attn.norm_added_k(encoder_hidden_states_key_proj) |
The "Tensor contiguity" gotcha in .ai/references/models.md describes this type of slowdown.
Other joint-attention processors with the same QK norms apply the norms on the contiguous [B, T, H, D] view and call dispatch_attention_fn without a transpose:
FluxAttnProcessor: transformer_flux.py L98-L121QwenDoubleStreamAttnProcessor2_0: transformer_qwenimage.py L528-L560HunyuanVideoAttnProcessor2_0: transformer_hunyuan_video.py L71-L132
Describe the solution you'd like.
Apply the norm on the [B, T, H, D] view, then transpose (one function, +14/-8):
query = query.view(batch_size, -1, attn.heads, head_dim)
key = key.view(batch_size, -1, attn.heads, head_dim)
value = value.view(batch_size, -1, attn.heads, head_dim)
if attn.norm_q is not None:
query = attn.norm_q(query)
if attn.norm_k is not None:
key = attn.norm_k(key)
query, key, value = query.transpose(1, 2), key.transpose(1, 2), value.transpose(1, 2)The context projections get the same change. The QK norms reduce over the last axis, and transpose(1, 2) does not move it, so the result does not change. The change adds no .contiguous() and no op: in torch.profiler, one transformer forward runs the same aten ops before and after (10105 on R9700, 9734 on RTX 5090).
End-to-end, StableDiffusion3Pipeline, SD3.5-medium, bf16, 40 steps, text_encoder_3=None:
| GPU | 512x512 | 1024x1024 |
|---|---|---|
| AMD Radeon AI PRO R9700 (gfx1201) | +2.80% | +2.19% |
| NVIDIA GeForce RTX 5090 | +0.74% | +0.46% |
- Output:
torch.equalon the decoded image isTrueon both GPUs, at both sizes, with the real weights. - Peak memory: no change.
- Tests: the fast tests for SD3, SD3 ControlNet, PAG SD3 and SD3 LoRA give the same per-test results before and after (361 passed, 100 skipped, 1 xfailed).
Describe alternatives you've considered.
- Change the order as above (my suggestion). Small, bit-identical, no change to the processor structure.
- Move
JointAttnProcessor2_0todispatch_attention_fn, which accepts[B, T, H, D]and removes the transpose. More complete, but it changes the attention backend path for SD3, so bit-identical output is not guaranteed. I can do this as a separate PR if you prefer it. - Keep the current code.
I have implemented and measured option 1, but I am not sure which direction the maintainers prefer. I would be grateful for your advice before I open a PR.
Additional context.
Setup: Linux, diffusers 43e30028a, torch 2.15 nightly (RTX 5090) / 2.14 ROCm (R9700). Median of 8 interleaved runs in one process.
cc @sayakpaul
Activity