Repository navigation
Conversation
dc3ef96 to
52d4b5b
Compare
|
Hi @klin2024, thanks for the PR! It does not appear to link an issue it fixes.
See the contribution guide for more details. Please note that PRs without a linked issue are likely to be automatically closed 10 days after this notice. Once the PR links an existing issue, the author box is ticked, or the |
`x.float() * cos + x_rotated.float() * sin` materializes float32 copies of x and x_rotated. With float32 cos / sin, type promotion already computes the products in float32, so drop the explicit casts. Bit-identical output. Signed-off-by: Lin, KuanTing <Kuanting.Lin@amd.com>
Same change as in apply_rotary_emb, applied to the helpers that repeat the `x.float() * cos + x_rotated.float() * sin` expression: LTX, LTX-2, QwenImage, LongCat-AudioDiT, JoyImage, JoyImage-Edit-Plus and Allegro (plus the `# Copied from` copies in QwenImage 2.1, NucleusMoE and the SANA-WM refiner, synced with `make fix-copies`). Bit-identical output. Signed-off-by: Lin, KuanTing <Kuanting.Lin@amd.com>
Checks that apply_rotary_emb matches the explicit float32 formula bit for bit for fp32 / fp16 / bf16 inputs, both use_real_unbind_dim values and both sequence_dim values. Signed-off-by: Lin, KuanTing <Kuanting.Lin@amd.com>
|
This PR has been automatically closed because it does not link an issue and the reminder above was not addressed within 10 days. If this PR is still relevant, please link the issue it fixes (e.g. We are experimenting with this process to keep the review queue manageable, and it will sometimes get it wrong. If you think this PR should stay open, please just say so here and we will reopen it — no need to justify it at length. Thanks again for contributing, and sorry for the noise if we closed this by mistake! |
|
This PR links #15220 ( |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
yiyixuxu
left a comment
There was a problem hiding this comment.
thanks, i left some feedbacks
|
|
||
| xq_out = (xq.float() * cos + _rotate_half(xq) * sin).type_as(xq) | ||
| xk_out = (xk.float() * cos + _rotate_half(xk) * sin).type_as(xk) | ||
| # float32 math through type promotion, without float32 copies of the input |
There was a problem hiding this comment.
| # float32 math through type promotion, without float32 copies of the input | |
| # cast `cos`/`sins` to at least float32 so that input is prompted to float32 inside the multiplication |
There was a problem hiding this comment.
can you use this comment through out?
| @@ -108,6 +108,36 @@ def test_sinoid_embeddings_hardcoded(self): | |||
| 1e-3, | |||
| ) | |||
|
|
|||
| def test_apply_rotary_emb_matches_float32_reference(self): | |||
There was a problem hiding this comment.
can you remove the test?
What does this PR do?
Fixes #15220
apply_rotary_emb(use_real=True) and several model-local RoPE helpers computex.float() * cos + x_rotated.float() * sin. With float32cos/sin, type promotion already does the math in float32, so the.float()calls only add full-size float32 copies of q / k per call. This PR drops them (castingcos/sinto at least float32 first, so bf16 / fp16 freqs keep float32 math). Output is bit-identical.Changed:
apply_rotary_embandapply_rotary_emb_allegro(embeddings.py), and the helpers in LTX, LTX-2, QwenImage, LongCat-AudioDiT, JoyImage and JoyImage-Edit-Plus; the# Copied fromcopies (QwenImage 2.1, NucleusMoE, SANA-WM refiner) viamake fix-copies. QwenImage / QwenImage 2.1 / NucleusMoE only call their helper withuse_real=False, so nothing changes at runtime there.Full pipeline call, bf16, before -> after (seconds):
RTX 5090: torch 2.14.1+cu130. R9700: torch 2.15.0a0+rocm10.1. diffusers main 8c54b38. Pipelines built from each repo's configs with random weights (timing does not depend on weight values); median of 2 calls after warmup, 2 interleaved rounds per side.
Bit-identical (every transformer output and the final output): all 13 on the RTX 5090; FLUX.2-klein, OmniGen and Ovis on the R9700. On the R9700, Cosmos, ACE-Step, HunyuanDiT and Allegro outputs (and Cosmos / ACE-Step timings) already vary run to run on main, so those are not compared.
Op alone, every changed helper against main (bf16
[1, 4608, 24, 128]-sized input): eager 1.15-2.02x (5090) / 1.15-1.80x (R9700), forward + backward 1.06-1.71x / 1.05-1.57x; outputs and gradients bit-identical for fp32 / fp16 / bf16 inputs with fp32 / bf16 / fp64 freqs.apply_rotary_embundertorch.compile: unchanged.Tests
New
EmbeddingsTests.test_apply_rotary_emb_matches_float32_referencechecksapply_rotary_embagainst the explicit float32 formula bit for bit (fp32 / fp16 / bf16, bothuse_real_unbind_dim, bothsequence_dim).The transformer test files of every model touched here (37) plus
tests/models/test_layers_utils.py(tests/models/transformers/test_models_transformer_{flux*,chroma,bria*,cogview4,cosmos*,omnigen,ovis*,hunyuan*,cogvideox,stable_audio*,longcat*,glm_image,easyanimate,motif*,krea*,ace_step,lumina*,ltx*,qwenimage*,nucleus*,sana_wm*,joyimage*,allegro}.py), RTX 5090: 63 failed, 1839 passed, 756 skipped, 5 xfailed. Main: 63 failed, 1838 passed (no new test), the same 63 tests: torch.compile AOT / compiled LoRA hot-swap, GGUF / BFL quantization, layerwise-casting memory.Self-review notes (
self-reviewskill)READY, no blocking issues. Left as is:
x: main rounds it to float32 viax.float(), this keeps float64. No pipeline feeds float64 q / k and it is more precise, so no extra branch.use_real=Truebranch is changed for consistency although no caller uses it.Before submitting
self-reviewskill on the diff?cc @sayakpaul @dg845