Skip to content

JointAttnProcessor2_0: apply the QK norm before the head transpose #15319

Open

Description

@menglcai

Is your feature request related to a problem? Please describe.

JointAttnProcessor2_0 (SD3 / SD3.5, SD3 ControlNet) transposes q/k to [B, H, T, D] and then applies norm_q / norm_k / norm_added_q / norm_added_k. So the QK norm reads a strided, non-contiguous tensor:

query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
if attn.norm_q is not None:
query = attn.norm_q(query)
if attn.norm_k is not None:
key = attn.norm_k(key)
# `context` projections.
if encoder_hidden_states is not None:
encoder_hidden_states_query_proj = attn.add_q_proj(encoder_hidden_states)
encoder_hidden_states_key_proj = attn.add_k_proj(encoder_hidden_states)
encoder_hidden_states_value_proj = attn.add_v_proj(encoder_hidden_states)
encoder_hidden_states_query_proj = encoder_hidden_states_query_proj.view(
batch_size, -1, attn.heads, head_dim
).transpose(1, 2)
encoder_hidden_states_key_proj = encoder_hidden_states_key_proj.view(
batch_size, -1, attn.heads, head_dim
).transpose(1, 2)
encoder_hidden_states_value_proj = encoder_hidden_states_value_proj.view(
batch_size, -1, attn.heads, head_dim
).transpose(1, 2)
if attn.norm_added_q is not None:
encoder_hidden_states_query_proj = attn.norm_added_q(encoder_hidden_states_query_proj)
if attn.norm_added_k is not None:
encoder_hidden_states_key_proj = attn.norm_added_k(encoder_hidden_states_key_proj)

The "Tensor contiguity" gotcha in .ai/references/models.md describes this type of slowdown.

Other joint-attention processors with the same QK norms apply the norms on the contiguous [B, T, H, D] view and call dispatch_attention_fn without a transpose:

Describe the solution you'd like.

Apply the norm on the [B, T, H, D] view, then transpose (one function, +14/-8):

query = query.view(batch_size, -1, attn.heads, head_dim)
key = key.view(batch_size, -1, attn.heads, head_dim)
value = value.view(batch_size, -1, attn.heads, head_dim)
if attn.norm_q is not None:
    query = attn.norm_q(query)
if attn.norm_k is not None:
    key = attn.norm_k(key)
query, key, value = query.transpose(1, 2), key.transpose(1, 2), value.transpose(1, 2)

The context projections get the same change. The QK norms reduce over the last axis, and transpose(1, 2) does not move it, so the result does not change. The change adds no .contiguous() and no op: in torch.profiler, one transformer forward runs the same aten ops before and after (10105 on R9700, 9734 on RTX 5090).

End-to-end, StableDiffusion3Pipeline, SD3.5-medium, bf16, 40 steps, text_encoder_3=None:

GPU 512x512 1024x1024
AMD Radeon AI PRO R9700 (gfx1201) +2.80% +2.19%
NVIDIA GeForce RTX 5090 +0.74% +0.46%
  • Output: torch.equal on the decoded image is True on both GPUs, at both sizes, with the real weights.
  • Peak memory: no change.
  • Tests: the fast tests for SD3, SD3 ControlNet, PAG SD3 and SD3 LoRA give the same per-test results before and after (361 passed, 100 skipped, 1 xfailed).

Describe alternatives you've considered.

  1. Change the order as above (my suggestion). Small, bit-identical, no change to the processor structure.
  2. Move JointAttnProcessor2_0 to dispatch_attention_fn, which accepts [B, T, H, D] and removes the transpose. More complete, but it changes the attention backend path for SD3, so bit-identical output is not guaranteed. I can do this as a separate PR if you prefer it.
  3. Keep the current code.

I have implemented and measured option 1, but I am not sure which direction the maintainers prefer. I would be grateful for your advice before I open a PR.

Additional context.

Setup: Linux, diffusers 43e30028a, torch 2.15 nightly (RTX 5090) / 2.14 ROCm (R9700). Median of 8 interleaved runs in one process.

cc @sayakpaul

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions