Skip to content

Add dense Qwen3.5/3.8-27B checkpoint-conversion support - #5149

Open
lokic233 wants to merge 1 commit into
AI-Hypercomputer:mainfrom
lokic233:feat/qwen3-5-27b-dense-conversion
Open

Add dense Qwen3.5/3.8-27B checkpoint-conversion support#5149
lokic233 wants to merge 1 commit into
AI-Hypercomputer:mainfrom
lokic233:feat/qwen3-5-27b-dense-conversion

Conversation

@lokic233

@lokic233 lokic233 commented Sep 4, 2026

Copy link
Copy Markdown

What

Adds HF→MaxText checkpoint-conversion support for the dense Qwen3.5/3.8-27B model.

Why

The qwen3_5 decoder block currently assumes the MoE variant throughout:

  • param_mapping unconditionally registers routed_experts-* and shared_expert-* hooks
  • hf_shape derives MLP shapes from the expert config
  • there is no qwen3.5-27b model config at all

The 27B checkpoint is densenum_experts is absent from its config.json
entirely — so conversion currently emits hooks for parameters that do not exist and
misses the three that do.

Worth flagging for reviewers: Qwen/Qwen3.8-27B declares model_type: qwen3_5 and
architectures: ["Qwen3_5ForConditionalGeneration"], so the existing qwen3_5 block is
the correct home for it despite the version number in the name.

Changes

  • configs/models/qwen3.5-27b.yml (new) — dense 27B: 5120 hidden, 64 layers,
    24 q / 4 kv heads, head_dim 256, ffn 17408, vocab 248320,
    inhomogeneous_layer_cycle_interval: 4 (one full-attention layer per four),
    GDN 16 key / 48 value heads at 128 dim, conv kernel 4,
    rope_max_timescale: 10000000, partial_rotary_factor: 0.25.
  • param_mapping.py — gate the expert hooks on num_experts > 1; add the dense
    mlp-wi_0 / wi_1 / wo branch. The MoE shared_expert is the same MlpBlock class the
    dense branch instantiates, so the same plain transpose applies.
  • hf_shape.py, hf_model_configs.py — dense MLP shapes, and the 27B HF config entry.
  • models/qwen3_5.py, configs/types.py, utils/globals.py — plumb the dense path
    and register the model name.

Every dimension is transcribed from Qwen/Qwen3.8-27B's own config.json, verified
byte-identical by md5 (0ecc077f5c9eb49889aa0022f4ae9fa7) against the stored artifact.

Testing

Hook-table diff, driving QWEN3_5_MAXTEXT_TO_HF_PARAM_HOOK_FN with the real
Qwen3.8-27B config.json:

hooks expert hooks dense MLP hooks
before 769 512 (for parameters that do not exist) 0 of 3
after 449 0 192 (64 layers × wi_0/wi_1/wo)

The MoE arm is the control. Forcing num_experts: 8 into the same config yields an
identical 769-hook table before and after, with the same eight -mlp- hook kinds — the
gate does not disturb the path it gates. Attention hooks are unchanged in both arms.

Numerical check against the transformers reference on CPU at reduced dimensions
(hidden 256, 8 layers, vocab 512, every parameter randomized — default init leaves
RMSNorm scales at 0, which would hide a wrong norm mapping):

max|diff| cosine argmax agreement
scan_layers=False 1.66893e-06 1.00000000 100%
scan_layers=True 1.57952e-06 1.00000000 100%

Sensitivity is demonstrated rather than assumed: swapping gate_projup_proj — a
same-shape error no assertion in the converter can catch, since silu(g)*u != silu(u)*g
— scores cosine 0.72936636 / 18.8% argmax agreement.

Depends on #5148

At --simulated_cpu_devices_count=1 this feature cannot run without the Orbax v1
leaf-handler fix (#5106); conversion dies in the save path before producing anything.
It works at the default of 16 either way. Please land that one first, or this ships a
feature that is dead on the single-device path.

Rebased onto 4cfec5b20c26fa2775e760034b66b7a4645ba877.

The qwen3_5 decoder block in MaxText assumes the MoE variant throughout:
`param_mapping` unconditionally registers `routed_experts-*` and
`shared_expert-*` hooks, `hf_shape` derives MLP shapes from the expert
config, and there is no `qwen3.5-27b` model config. The 27B checkpoint is
dense (`num_experts` absent), so conversion produces hooks for parameters
that do not exist and misses the three that do.

Changes:

* `configs/models/qwen3.5-27b.yml` (new) -- dense 27B: 5120 hidden, 64
  layers, 24 q / 4 kv heads, head_dim 256, ffn 17408, vocab 248320,
  `inhomogeneous_layer_cycle_interval: 4` (one full-attention layer per
  four), GDN 16 key / 48 value heads at 128 dim, conv kernel 4,
  `rope_max_timescale: 10000000`, `partial_rotary_factor: 0.25`.
* `param_mapping.py` -- gate the expert hooks on `num_experts > 1` and add
  the dense `mlp-wi_0 / wi_1 / wo` branch. The MoE `shared_expert` is the
  same `MlpBlock` class the dense branch instantiates, so the same plain
  `transpose` applies.
* `hf_shape.py`, `hf_model_configs.py` -- dense MLP shapes and the 27B
  HF config entry.
* `models/qwen3_5.py`, `configs/types.py`, `utils/globals.py` -- plumb the
  dense path and register the model name.

Every dimension above is transcribed from `Qwen/Qwen3.8-27B`'s own
`config.json`, verified byte-identical by md5 (`0ecc077f5c9eb49889aa0022
f4ae9fa7`) against the stored artifact. Note that 3.8 declares
`model_type: qwen3_5` and `architectures: ["Qwen3_5ForConditionalGeneration"]`,
so the existing `qwen3_5` block is the correct home for it.

Verified by execution, not by review. Driving
QWEN3_5_MAXTEXT_TO_HF_PARAM_HOOK_FN with the real Qwen3.8-27B config.json
(which ships with no `num_experts` key at all -- itself the evidence that the
model is dense):

  before:  769 hooks, of which 512 are routed_experts-*/shared_expert-* hooks
           for parameters that do not exist in this checkpoint, and 0 of the
           3 dense MLP hooks that do
  after:   449 hooks, 192 dense MLP hooks (64 layers x wi_0/wi_1/wo), 0 expert
           hooks

The MoE arm is the control: forcing `num_experts: 8` into the same config
yields the identical 769-hook table before and after the patch, with the same
eight `-mlp-` hook kinds, so the gate does not disturb the path it gates.
Attention hooks are unchanged in both arms.

Generated against the pinned snapshot 58b2a32,
then rebased onto upstream 4cfec5b.

On the rebased tree the conversion was checked numerically against the
transformers reference implementation on CPU at reduced dimensions
(hidden 256, 8 layers, vocab 512, every parameter randomised):

  scan_layers=False   max|diff| 1.66893e-06   cosine 1.00000000   argmax 100%
  scan_layers=True    max|diff| 1.57952e-06   cosine 1.00000000   argmax 100%

The sensitivity of that check is demonstrated rather than assumed: swapping
gate_proj and up_proj -- a same-shape error no assertion in the converter can
see, since silu(g)*u != silu(u)*g -- gives cosine 0.72936636 and 18.8% argmax
agreement.

Signed-off-by: Loki Chen <dengcchi@meta.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for the dense qwen3.5-27b model (Qwen3.8-27B) to the Qwen3.5 model family. It updates the configuration, shape utilities, parameter mapping, and model architecture to support both dense and MoE variants dynamically. Feedback suggests handling cases where 'num_experts' is explicitly set to None in the configuration dictionary to prevent a potential TypeError.

Comment on lines +707 to +709
# Dense members of the Qwen3.5 family (e.g. qwen3.5-27b) omit every MoE key.
num_experts = config.get("num_experts", 1)
is_moe = num_experts > 1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To prevent a potential TypeError if "num_experts" is explicitly set to None in the configuration dictionary, handle None values defensively. This aligns with the defensive style used in GEMMA4_HF_WEIGHTS_TO_SHAPE in this same file.

Suggested change
# Dense members of the Qwen3.5 family (e.g. qwen3.5-27b) omit every MoE key.
num_experts = config.get("num_experts", 1)
is_moe = num_experts > 1
# Dense members of the Qwen3.5 family (e.g. qwen3.5-27b) omit every MoE key.
num_experts = config.get("num_experts")
num_experts = num_experts if num_experts is not None else 1
is_moe = num_experts > 1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant