Add dense Qwen3.5/3.8-27B checkpoint-conversion support - #5149
Open
lokic233 wants to merge 1 commit into
Open
Conversation
The qwen3_5 decoder block in MaxText assumes the MoE variant throughout:
`param_mapping` unconditionally registers `routed_experts-*` and
`shared_expert-*` hooks, `hf_shape` derives MLP shapes from the expert
config, and there is no `qwen3.5-27b` model config. The 27B checkpoint is
dense (`num_experts` absent), so conversion produces hooks for parameters
that do not exist and misses the three that do.
Changes:
* `configs/models/qwen3.5-27b.yml` (new) -- dense 27B: 5120 hidden, 64
layers, 24 q / 4 kv heads, head_dim 256, ffn 17408, vocab 248320,
`inhomogeneous_layer_cycle_interval: 4` (one full-attention layer per
four), GDN 16 key / 48 value heads at 128 dim, conv kernel 4,
`rope_max_timescale: 10000000`, `partial_rotary_factor: 0.25`.
* `param_mapping.py` -- gate the expert hooks on `num_experts > 1` and add
the dense `mlp-wi_0 / wi_1 / wo` branch. The MoE `shared_expert` is the
same `MlpBlock` class the dense branch instantiates, so the same plain
`transpose` applies.
* `hf_shape.py`, `hf_model_configs.py` -- dense MLP shapes and the 27B
HF config entry.
* `models/qwen3_5.py`, `configs/types.py`, `utils/globals.py` -- plumb the
dense path and register the model name.
Every dimension above is transcribed from `Qwen/Qwen3.8-27B`'s own
`config.json`, verified byte-identical by md5 (`0ecc077f5c9eb49889aa0022
f4ae9fa7`) against the stored artifact. Note that 3.8 declares
`model_type: qwen3_5` and `architectures: ["Qwen3_5ForConditionalGeneration"]`,
so the existing `qwen3_5` block is the correct home for it.
Verified by execution, not by review. Driving
QWEN3_5_MAXTEXT_TO_HF_PARAM_HOOK_FN with the real Qwen3.8-27B config.json
(which ships with no `num_experts` key at all -- itself the evidence that the
model is dense):
before: 769 hooks, of which 512 are routed_experts-*/shared_expert-* hooks
for parameters that do not exist in this checkpoint, and 0 of the
3 dense MLP hooks that do
after: 449 hooks, 192 dense MLP hooks (64 layers x wi_0/wi_1/wo), 0 expert
hooks
The MoE arm is the control: forcing `num_experts: 8` into the same config
yields the identical 769-hook table before and after the patch, with the same
eight `-mlp-` hook kinds, so the gate does not disturb the path it gates.
Attention hooks are unchanged in both arms.
Generated against the pinned snapshot 58b2a32,
then rebased onto upstream 4cfec5b.
On the rebased tree the conversion was checked numerically against the
transformers reference implementation on CPU at reduced dimensions
(hidden 256, 8 layers, vocab 512, every parameter randomised):
scan_layers=False max|diff| 1.66893e-06 cosine 1.00000000 argmax 100%
scan_layers=True max|diff| 1.57952e-06 cosine 1.00000000 argmax 100%
The sensitivity of that check is demonstrated rather than assumed: swapping
gate_proj and up_proj -- a same-shape error no assertion in the converter can
see, since silu(g)*u != silu(u)*g -- gives cosine 0.72936636 and 18.8% argmax
agreement.
Signed-off-by: Loki Chen <dengcchi@meta.com>
lokic233
requested review from
A9isha,
NuojCheng,
RissyRan,
SurbhiJainUSC,
abhinavclemson,
aireenmei,
bvandermoon,
darisoy,
dipannita08,
gagika,
gobbleturk,
hengtaoguo,
huytransformer,
igorts-git,
jiangjy1982,
khatwanimohit,
parambole,
richjames0,
shralex,
shuningjin,
vipannalla and
xibinliu
as code owners
September 4, 2026 23:08
There was a problem hiding this comment.
Code Review
This pull request adds support for the dense qwen3.5-27b model (Qwen3.8-27B) to the Qwen3.5 model family. It updates the configuration, shape utilities, parameter mapping, and model architecture to support both dense and MoE variants dynamically. Feedback suggests handling cases where 'num_experts' is explicitly set to None in the configuration dictionary to prevent a potential TypeError.
Comment on lines
+707
to
+709
| # Dense members of the Qwen3.5 family (e.g. qwen3.5-27b) omit every MoE key. | ||
| num_experts = config.get("num_experts", 1) | ||
| is_moe = num_experts > 1 |
There was a problem hiding this comment.
To prevent a potential TypeError if "num_experts" is explicitly set to None in the configuration dictionary, handle None values defensively. This aligns with the defensive style used in GEMMA4_HF_WEIGHTS_TO_SHAPE in this same file.
Suggested change
| # Dense members of the Qwen3.5 family (e.g. qwen3.5-27b) omit every MoE key. | |
| num_experts = config.get("num_experts", 1) | |
| is_moe = num_experts > 1 | |
| # Dense members of the Qwen3.5 family (e.g. qwen3.5-27b) omit every MoE key. | |
| num_experts = config.get("num_experts") | |
| num_experts = num_experts if num_experts is not None else 1 | |
| is_moe = num_experts > 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds HF→MaxText checkpoint-conversion support for the dense Qwen3.5/3.8-27B model.
Why
The
qwen3_5decoder block currently assumes the MoE variant throughout:param_mappingunconditionally registersrouted_experts-*andshared_expert-*hookshf_shapederives MLP shapes from the expert configqwen3.5-27bmodel config at allThe 27B checkpoint is dense —
num_expertsis absent from itsconfig.jsonentirely — so conversion currently emits hooks for parameters that do not exist and
misses the three that do.
Worth flagging for reviewers:
Qwen/Qwen3.8-27Bdeclaresmodel_type: qwen3_5andarchitectures: ["Qwen3_5ForConditionalGeneration"], so the existingqwen3_5block isthe correct home for it despite the version number in the name.
Changes
configs/models/qwen3.5-27b.yml(new) — dense 27B: 5120 hidden, 64 layers,24 q / 4 kv heads, head_dim 256, ffn 17408, vocab 248320,
inhomogeneous_layer_cycle_interval: 4(one full-attention layer per four),GDN 16 key / 48 value heads at 128 dim, conv kernel 4,
rope_max_timescale: 10000000,partial_rotary_factor: 0.25.param_mapping.py— gate the expert hooks onnum_experts > 1; add the densemlp-wi_0 / wi_1 / wobranch. The MoEshared_expertis the sameMlpBlockclass thedense branch instantiates, so the same plain
transposeapplies.hf_shape.py,hf_model_configs.py— dense MLP shapes, and the 27B HF config entry.models/qwen3_5.py,configs/types.py,utils/globals.py— plumb the dense pathand register the model name.
Every dimension is transcribed from
Qwen/Qwen3.8-27B's ownconfig.json, verifiedbyte-identical by md5 (
0ecc077f5c9eb49889aa0022f4ae9fa7) against the stored artifact.Testing
Hook-table diff, driving
QWEN3_5_MAXTEXT_TO_HF_PARAM_HOOK_FNwith the realQwen3.8-27B
config.json:The MoE arm is the control. Forcing
num_experts: 8into the same config yields anidentical 769-hook table before and after, with the same eight
-mlp-hook kinds — thegate does not disturb the path it gates. Attention hooks are unchanged in both arms.
Numerical check against the
transformersreference on CPU at reduced dimensions(hidden 256, 8 layers, vocab 512, every parameter randomized — default init leaves
RMSNorm scales at 0, which would hide a wrong norm mapping):
scan_layers=Falsescan_layers=TrueSensitivity is demonstrated rather than assumed: swapping
gate_proj↔up_proj— asame-shape error no assertion in the converter can catch, since
silu(g)*u != silu(u)*g— scores cosine 0.72936636 / 18.8% argmax agreement.
Depends on #5148
At
--simulated_cpu_devices_count=1this feature cannot run without the Orbax v1leaf-handler fix (#5106); conversion dies in the save path before producing anything.
It works at the default of 16 either way. Please land that one first, or this ships a
feature that is dead on the single-device path.
Rebased onto
4cfec5b20c26fa2775e760034b66b7a4645ba877.