Skip to content

[megablox] Remove redundant expert-weight transpose copy in unquantized gmm_v2 DLHS - #5123

Open
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_975477390
Open

[megablox] Remove redundant expert-weight transpose copy in unquantized gmm_v2 DLHS#5123
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_975477390

Conversation

@copybara-service

Copy link
Copy Markdown
Contributor

[megablox] Remove redundant expert-weight transpose copy in unquantized gmm_v2 DLHS

The tokamax gmm_v2 kernel has no native transpose_rhs support, so the V2 DLHS
gradient path (_dlhs_run_tokamax_v2) manually materializes rhs.swapaxes(1, 2)
on every backward step. For the MoE expert weight this is a full-tensor HBM copy
of a large [num_experts, k, n] array (e.g. bf16[16, 7168, 2048]), emitted by
XLA as a standalone copy with op_name="rhs".

The V1 tokamax path already computes the identical DLHS with no transpose: it
calls tokamax.ragged_dot_general with DLHS_RAGGED_DOT_DIM_NUMS, contracting
over the n dimension directly. This change routes the unquantized gmm_v2 DLHS
(quantization_rule is None) through that transpose-free V1 kernel. Quantized
paths are unchanged and keep the V2 kernel, because their scale handling
(_bwd_prepare_inputs / _dlhs_scale_grad_by_rhs_scale) is gmm_v2-specific.

Verified on TPU with a DeepSeek-scale shape (lhs[m=16384, k=7168],
weight[g=16, k=7168, n=2048], bf16): the full expert-weight transpose copy is
removed (1 -> 0), gradients are bit-identical (max relative error 0.0), and the
full gmm fwd+bwd is ~3% faster. The output is bit-identical, so this is a pure
efficiency change with no numerical or convergence impact.

@codecov

codecov Bot commented Sep 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…ed gmm_v2 DLHS

The tokamax gmm_v2 kernel has no native transpose_rhs support, so the V2 DLHS
gradient path (`_dlhs_run_tokamax_v2`) manually materializes `rhs.swapaxes(1, 2)`
on every backward step. For the MoE expert weight this is a full-tensor HBM copy
of a large `[num_experts, k, n]` array (e.g. `bf16[16, 7168, 2048]`), emitted by
XLA as a standalone `copy` with `op_name="rhs"`.

The V1 tokamax path already computes the identical DLHS with no transpose: it
calls `tokamax.ragged_dot_general` with `DLHS_RAGGED_DOT_DIM_NUMS`, contracting
over the `n` dimension directly. This change routes the unquantized gmm_v2 DLHS
(quantization_rule is None) through that transpose-free V1 kernel. Quantized
paths are unchanged and keep the V2 kernel, because their scale handling
(`_bwd_prepare_inputs` / `_dlhs_scale_grad_by_rhs_scale`) is gmm_v2-specific.

Verified on TPU with a DeepSeek-scale shape (lhs[m=16384, k=7168],
weight[g=16, k=7168, n=2048], bf16): the full expert-weight transpose copy is
removed (1 -> 0), gradients are bit-identical (max relative error 0.0), and the
full gmm fwd+bwd is ~3% faster. The output is bit-identical, so this is a pure
efficiency change with no numerical or convergence impact.

PiperOrigin-RevId: 975477390
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant