Repository navigation
Description
Is your feature request related to a problem? Please describe.
While profiling diffusers pipelines on AMD and NVIDIA GPUs, we found that the Stable Audio 3 DiT and the AutoencoderSAME attention repeat the same small ops on every call. Both use differential attention in stable-audio-3-medium (use_differential_attention=True):
_apply_rotary_partialcomputesfreqs.cos()/freqs.sin()4 times per DiT block (q1,q2,k1,k2) and 4 times perAutoencoderSAMEattention call, although the angle table does not change within one DiT forward (or within oneAutoencoderSAMEattention call).StableAudio3RMSNormcastsgammawith.to(compute_dtype)on every call. Every instance usesforce_fp32=True, so type promotion already does the multiply in float32.q1/q2shareq_normandk1/k2sharek_norm, but are normalized in 4 separate calls. In the DiT self-attention and theAutoencoderSAMEattention, all four also share one angle table but are rotated in 4 separate calls.
Describe the solution you'd like.
Compute (cos, sin) once per DiT forward and once per AutoencoderSAME attention call, drop the gamma cast, and normalize / rotate [q1 | q2] and [k1 | k2] each as one (2 * heads)-head tensor. No new branches or flags.
Full pipeline call, main (8069682) -> this change, bf16, seconds: 0.334 -> 0.233 (1.43x) on an RTX 5090 (torch 2.14.1+cu130), 0.283 -> 0.234 (1.21x) on a Radeon AI PRO R9700 (torch 2.16.0a0+rocm10.3). stabilityai/stable-audio-3-medium converted with scripts/convert_stable_audio_3_to_diffusers.py, prompt "A gentle piano melody with soft strings in a concert hall", duration=10, default 8 steps, batch 1, default attention backend (native), CUDA generator seed 0. Output is bit-identical for seeds 0, 1, 7.
timing script
# TORCHDYNAMO_DISABLE=1 PYTHONPATH=<checkout>/src python e2e_branch.py TAG MODEL_DIR OUT_DIR 10 2
import sys
import time
import torch
from diffusers import StableAudio3Pipeline
TAG, MODEL, OUT, N, COOL = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4]), float(sys.argv[5])
PROMPT = "A gentle piano melody with soft strings in a concert hall"
pipe = StableAudio3Pipeline.from_pretrained(MODEL, torch_dtype=torch.bfloat16).to("cuda")
pipe.set_progress_bar_config(disable=True)
def gen(seed):
g = torch.Generator("cuda").manual_seed(seed)
with torch.inference_mode():
return pipe(PROMPT, duration=10.0, generator=g).audios
for _ in range(2):
gen(0)
for s in (0, 1, 7):
torch.save(gen(s).cpu(), f"{OUT}/{TAG}_seed{s}.pt")
times = []
for _ in range(N):
time.sleep(COOL)
torch.cuda.synchronize()
t0 = time.perf_counter()
gen(0)
torch.cuda.synchronize()
times.append(time.perf_counter() - t0)
times.sort()
print(f"{TAG}: median {times[len(times) // 2]:.4f}s")Describe alternatives you've considered.
A fused Triton / CUDA kernel would remove more launches but would add a kernel dependency; this change keeps the same PyTorch ops and only removes repeated calls.
Additional context.
#15220 notes that the idea in #15215 also applies to "Stable Audio 3 partial RoPE". This change leaves that part alone (the t_rot.float() copy in _apply_rotary_partial stays), so that fix can be applied on top of this one. It is also independent of #14936 (fp32 RoPE angles in StableAudio3RotaryEmbedding); they touch different lines.
Activity