performance regression on Ring SP

[JasonHe-WQ]

Description

In v1 branch, deploying DiffSynth engine with 2 GPUs without NVLink and inferencing model Qwen Image 2512, ring SP2 is much slower than Ulysses SP2.

However, the total communication volume is identical for both strategies when using 2 GPUs while ring CP can overlap same comunication time spend with computation. Which means CP would be faster theoretically.

Reason Explanations

In v1 branch, when use ring sp (or cp) with sdpa, the attention backend would be chose as _scaled_dot_product_efficient_attention and this kernel is far slower than _scaled_dot_product_flash_attention. Other backends would not be influenced.

DiffSynth-Engine/diffsynth_engine/layers/attention/backends/sdpa.py

Lines 99 to 113 in c23ece5

	seq_len = query.shape[2]
	output, lse = _scaled_dot_product_efficient_attention(
	query,
	key,
	value,
	attn_bias=attn_mask,
	compute_log_sumexp=True,
	is_causal=self.causal,
	scale=self.softmax_scale,
	)[:2]
	output = rearrange(output, "b n s d -> b s n d")
	# the returned lse is padded but not restored, so we need to slice it
	lse = lse[:, :, :seq_len]
	return output, lse

Detailed comparison

On 4 RTX Pro 5000 Blackwell, for one 1024x1024 picture with 5 steps, the benchmarks table can be concluded as below.

kernels	Efficient/kernel	Torch Flash/kernel	FA4/kernel	FA4 vs Flash	Torch Flash/step	FA4/step
cp2cfg	733.068 us	287.672 us	279.057 us	-2.99%	276.576 ms	275.910 ms
cp4	197.949 us	78.009 us	75.263 us	-3.52%	442.760 ms	425.265 ms

As we can see, the Torch Flash kernel is much faster than Efficient kernel.