Skip to content

Stable Audio 3 attention repeats RoPE cos / sin and q / k norm work on every call #15239

Open

Description

@chsu-amd

Is your feature request related to a problem? Please describe.

While profiling diffusers pipelines on AMD and NVIDIA GPUs, we found that the Stable Audio 3 DiT and the AutoencoderSAME attention repeat the same small ops on every call. Both use differential attention in stable-audio-3-medium (use_differential_attention=True):

  • _apply_rotary_partial computes freqs.cos() / freqs.sin() 4 times per DiT block (q1, q2, k1, k2) and 4 times per AutoencoderSAME attention call, although the angle table does not change within one DiT forward (or within one AutoencoderSAME attention call).
  • StableAudio3RMSNorm casts gamma with .to(compute_dtype) on every call. Every instance uses force_fp32=True, so type promotion already does the multiply in float32.
  • q1 / q2 share q_norm and k1 / k2 share k_norm, but are normalized in 4 separate calls. In the DiT self-attention and the AutoencoderSAME attention, all four also share one angle table but are rotated in 4 separate calls.

Describe the solution you'd like.

Compute (cos, sin) once per DiT forward and once per AutoencoderSAME attention call, drop the gamma cast, and normalize / rotate [q1 | q2] and [k1 | k2] each as one (2 * heads)-head tensor. No new branches or flags.

Full pipeline call, main (8069682) -> this change, bf16, seconds: 0.334 -> 0.233 (1.43x) on an RTX 5090 (torch 2.14.1+cu130), 0.283 -> 0.234 (1.21x) on a Radeon AI PRO R9700 (torch 2.16.0a0+rocm10.3). stabilityai/stable-audio-3-medium converted with scripts/convert_stable_audio_3_to_diffusers.py, prompt "A gentle piano melody with soft strings in a concert hall", duration=10, default 8 steps, batch 1, default attention backend (native), CUDA generator seed 0. Output is bit-identical for seeds 0, 1, 7.

timing script
# TORCHDYNAMO_DISABLE=1 PYTHONPATH=<checkout>/src python e2e_branch.py TAG MODEL_DIR OUT_DIR 10 2
import sys
import time

import torch

from diffusers import StableAudio3Pipeline

TAG, MODEL, OUT, N, COOL = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4]), float(sys.argv[5])
PROMPT = "A gentle piano melody with soft strings in a concert hall"
pipe = StableAudio3Pipeline.from_pretrained(MODEL, torch_dtype=torch.bfloat16).to("cuda")
pipe.set_progress_bar_config(disable=True)


def gen(seed):
    g = torch.Generator("cuda").manual_seed(seed)
    with torch.inference_mode():
        return pipe(PROMPT, duration=10.0, generator=g).audios


for _ in range(2):
    gen(0)
for s in (0, 1, 7):
    torch.save(gen(s).cpu(), f"{OUT}/{TAG}_seed{s}.pt")
times = []
for _ in range(N):
    time.sleep(COOL)
    torch.cuda.synchronize()
    t0 = time.perf_counter()
    gen(0)
    torch.cuda.synchronize()
    times.append(time.perf_counter() - t0)
times.sort()
print(f"{TAG}: median {times[len(times) // 2]:.4f}s")

Describe alternatives you've considered.

A fused Triton / CUDA kernel would remove more launches but would add a kernel dependency; this change keeps the same PyTorch ops and only removes repeated calls.

Additional context.

#15220 notes that the idea in #15215 also applies to "Stable Audio 3 partial RoPE". This change leaves that part alone (the t_rot.float() copy in _apply_rotary_partial stays), so that fix can be applied on top of this one. It is also independent of #14936 (fp32 RoPE angles in StableAudio3RotaryEmbedding); they touch different lines.

cc @yiyixuxu @sayakpaul

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions