Skip to content

Add from scratch pretrain converter#1504

Closed
Cacapice wants to merge 12 commits into
TransformerLensOrg:mainfrom
Cacapice:add-from-scratch-pretrain-converter
Closed

Add from scratch pretrain converter#1504
Cacapice wants to merge 12 commits into
TransformerLensOrg:mainfrom
Cacapice:add-from-scratch-pretrain-converter

Conversation

@Cacapice

Copy link
Copy Markdown

Description

Adds a weight converter and loader that bring checkpoints from an external
from-scratch pretraining framework into HookedTransformer, so a
foundation model trained outside the HuggingFace ecosystem can be analysed with
the standard TransformerLens hook API (run_with_cache, resid_post, etc.).

The source architecture is a decoder-only transformer with RoPE (adjacent-pair
rotation), RMSNorm, SwiGLU MLPs, and optional dropless top-k Mixture-of-Experts
.
Its checkpoints are not loadable through from_pretrained because they are not
a HuggingFace model; this PR follows the same pattern as the existing
nanogpt / mixtral converters to support them.

Motivation: this is the base-model half of an interpretability project — the
model whose residual stream downstream linear probes read. Loading it into
TransformerLens means the probing side can use the library's tooling directly
instead of a bespoke activation-extraction path.

Fixes # (no issue — new-feature contribution)

Type of change

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update (docstrings included; happy
    to add a docs/ loading example if the maintainers would like one)

What's included

  • weight_conversions/maritime_pretrain.pyconvert_maritime_pretrain_weights.
    Splits the fused QKV projection into per-head W_Q/W_K/W_V, maps SwiGLU
    gate/up/down onto TransformerLens's W_gate/W_in/W_out, and (for MoE) emits
    the router plus every expert under the names MoE/MoEGatedMLP expect. No
    Q/K reordering is needed because RoPE is applied inside attention, not baked
    into the weights.
  • pretrained/maritime_pretrain_loader.pybuild_config and
    load_maritime_pretrain. Builds the matching HookedTransformerConfig
    (positional_embedding_type="rotary", rotary_adjacent_pairs=True,
    normalization_type="RMS", final_rms=True, gated_mlp=True, and
    num_experts/experts_per_token/norm_topk_prob for MoE), merges
    tensor-parallel weight shards, and returns a ready HookedTransformer.
  • weight_conversions/__init__.py — export the new converter (alphabetical).
  • Teststests/unit/pretrained_weight_conversions/test_maritime_pretrain.py.

Key correctness detail

The source rotates adjacent RoPE pairs ([x0, x1] -> [-x1, x0]), which maps to
rotary_adjacent_pairs=True (the GPT-NeoX-style d -> (d 2) frequency layout),
not the default half-split convention. Getting this wrong still loads without
error but silently corrupts every activation, so the equivalence tests below are
the real guardrail.

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (module + function docstrings)
  • My changes generate no new warnings
  • I have added tests that prove my feature works
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

Test evidence

test_maritime_pretrain.py asserts bit-exact logit equivalence between the
source model and its converted HookedTransformer, parametrised across the axes
a future change is most likely to break, plus structural, failure-path and
tensor-parallel coverage (13 tests):

  • dense and MoE MLPs
  • tied and untied embeddings
  • four (d_model, n_heads) combinations
  • converter key/shape correctness for the MoE branch
  • a config/checkpoint shape mismatch raises a clear ValueError (not an opaque
    reshape error)
  • tensor-parallel shard merge reconstructs the original weights for
    N ∈ {1, 2, 4} shards
$ python -m pytest tests/unit/pretrained_weight_conversions/test_maritime_pretrain.py -q
13 passed

Observed max|Δlogit| between source and converted model ≈ 1e-5
(floating-point noise) across every parametrisation. Converter validates each
source weight's shape before reshaping, so a mismatched config fails with an
actionable message. Files formatted with the repo's pinned black==23.3.0 /
isort==5.8.0; pylint 10.00/10 and pyright clean on the new modules.

Notes / open questions for reviewers

  1. Naming. I've used the project-specific name maritime_pretrain to mirror
    nanogpt. If you'd prefer a generic name (e.g. swiglu_moe_gpt) since the
    converter isn't domain-specific, I'm glad to rename.
  2. Loader placement. The loader lives at pretrained/maritime_pretrain_loader.py
    rather than being wired into loading_from_pretrained.py, because these
    checkpoints load from a local run directory, not a hub ID. If you'd rather
    register it as an official model source, point me at the preferred hook.
  3. Targeting dev per the contribution guidelines (this is a feature, not a
    fix against the released version).

pablocs116 and others added 12 commits July 1, 2026 21:03
Adds TransformerBridge support for Arcee AI's AFM-4.5B (ArceeForCausalLM), a
Llama-style dense decoder whose only architectural novelty is an ungated
squared-ReLU (ReLU^2) MLP (up_proj -> ReLU^2 -> down_proj) instead of the gated
SiLU/GeLU used by Llama.

- Register the "relu2" activation (relu(x)**2) in SUPPORTED_ACTIVATIONS so the
  config guard and ActivationFunctionFactory accept HF's hidden_act="relu2".
- New ArceeArchitectureAdapter: Llama attention (RoPE, RMSNorm, GQA, no QK-norm,
  no biases) + ungated MLPBridge. Post-activation ReLU^2 neurons are inspectable
  via blocks.{i}.mlp.hook_post.
- Register at the four sites (factory, supported_architectures __init__,
  model_registry HF_SUPPORTED_ARCHITECTURES + CANONICAL_AUTHORS_BY_ARCH,
  generate_report descriptions) and add the two checkpoints to the registry.
- Unit + integration parity tests (fp32 + eager); integration asserts bridge
  logits match HF and hook_post exposes non-negative d_mlp-width activations.

Closes TransformerLensOrg#1467

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…red-relu-mlp

Add Arcee squared-ReLU MLP adapter (ArceeForCausalLM)
…o dev) (TransformerLensOrg#1482)

* Added minimal TransformerBridge support skeleton for Qwen2MoeForCausalLM

* Fix return type annotation in Qwen2MoeRouterBridge.forward method

* Enhance tests for Qwen2-MoE bridge to validate cache capturing of MoE hooks

---------

Co-authored-by: Canonik <alessandro006@icloud.com>
* Normalized SSM mixer across existing hybrids

* GatedDeltaNet and compute_ssm_state improvements

* Prepping Mamba1 mixer

* Unify vocabulary

* SSM2 interp support

* Mamba1 read parity

* Fixed Mamba1 bugs

* Fix GPU device mismatch

* Additional cleanup and testing

* make format

* Model verification + comment cleanup

* documentation updates

* Final SSM pass
…LM) (TransformerLensOrg#1480)

* Add Falcon H1 Hybrid Parallel Mamba TransformerBridge Adapter

* Add Tiny-90M parity tests and attention-branch ablation coverage for Falcon-H1 adapter

* Drop redundant CI skipif from Falcon-H1 integration tests; rely on slow marker

* Fix mypy attr-defined for Falcon-H1 eps_attr assignment

---------

Co-authored-by: jlarson4 <jonahalarson@comcast.net>
* Add BD3LM architecture adapter

* Address review: fix dead hook_attn_out, elif ordering, add integration test and verify_models support

---------

Co-authored-by: Jonah Larson <jonahalarson@comcast.net>
…#1483)

* Add RecurrentGemma (Griffin) architecture adapter

Adds a TransformerBridge adapter for RecurrentGemmaForCausalLM (Griffin):
RG-LRU real-gated linear recurrence interleaved with local sliding-window
attention (config.block_types, default recurrent/recurrent/attention).

Because the temporal_block substructure varies per layer (recurrent vs.
attention), the adapter wraps each decoder layer whole with residual-stream
hooks only (RecurrentGemmaBlockBridge), mirroring Lfm2MoeArchitectureAdapter —
rather than pretending every layer has a homogeneous attention/MLP
substructure. applicable_phases=[4] (generation); finer-grained RG-LRU state
hooks are a follow-up.

Sets the Gemma-family numerics on the bridge config: RMSNorm (1.0 + weight)
offset, gated MLP, final_rms, and the tanh final-logit soft cap
(config.logits_soft_cap=30.0). ln_final maps to model.final_norm.

Registers the adapter at the factory + supported_architectures __init__ sites
and adds an offline structural unit test.

Closes TransformerLensOrg#1472

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Apply black formatting to RecurrentGemma adapter test

Collapse two test method signatures onto single lines per black
(line-length 100). No behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Register RecurrentGemma in the model-registry sync tables

The RegistrySyncedWithFactory unit tests require every factory
architecture to appear in HF_SUPPORTED_ARCHITECTURES and
CANONICAL_AUTHORS_BY_ARCH (or INTENTIONAL_EXCLUDES). Add
RecurrentGemmaForCausalLM to both (canonical author: google).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* ci: re-trigger checks (flaky nbval notebook)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…1486)

* feat: add Zamba2ForCausalLM architecture adapter

Closes TransformerLensOrg#1463

Supports Zyphra/Zamba2-1.2B, Zamba2-7B, Zamba2-7B-Instruct via
TransformerBridge. Zamba2 is a hybrid Mamba-2 / shared-attention model
where layers_block_type alternates between "mamba" (pure SSM) and
"hybrid" (SSM + shared global-attention block, weight-tied across
num_mem_blocks slots).

Key design decisions:
- SSMBlockBridge for all layers: hook_in/hook_out fire regardless of
  layer type. Mamba layers expose mixer submodule hooks (in_proj,
  conv1d, inner_norm, out_proj); hybrid layers have norm and mixer
  marked optional=True so component_setup skips them gracefully.
- is_stateful=False: Zamba2 threads its unified cache via
  past_key_values (standard KV path), not the Mamba cache_params path.
  Setting is_stateful=True collided with Zamba2's own cache call.
- positional_embedding_type="none": RoPE is handled inside the HF
  attention block; no model-level rotary bridge needed.
- Added mamba_expand, mamba_ngroups, num_mem_blocks, layers_block_type,
  use_shared_attention_adapter to _HF_PASSTHROUGH_ATTRS in both
  sources/transformers.py and sources/_bridge_builder.py so these
  config fields reach the adapter reliably.

21 integration tests pass against Zyphra/Zamba2-1.2B (38 layers:
32 mamba + 6 hybrid). Forward parity is exact (max_diff == 0.0);
greedy generation matches HF bit-for-bit across both layer types.

* fix: add Zamba2ForCausalLM to model registry (HF_SUPPORTED_ARCHITECTURES + CANONICAL_AUTHORS_BY_ARCH)
… sentinel (TransformerLensOrg#1491)

Consolidates the tanh soft-capping (Gemma-2 style) used for attention
scores and output logits into apply_softcap()/softcap_enabled() in
transformer_lens/utilities/activation_functions.py, replacing five
independent cap * tanh(x / cap) implementations across HookedTransformer,
abstract_attention, and the native TransformerBridge model, plus the
center_unembed compatibility checks in both systems.

Also replaces the bare -1.0 disabled sentinel with a named
SOFTCAP_DISABLED constant in both HookedTransformerConfig and
TransformerBridgeConfig.

Pure refactor, no behavior change. A naive 'if cap:' truthiness check
would treat -1.0 as enabled (computing -1.0 * tanh(x / -1.0) == tanh(x),
saturating every large score to 1.0); softcap_enabled() makes that
mistake structurally impossible instead of relying on every call site
getting 'cap > 0' right independently.

Verified: uv run mypy . clean (307 files), make unit-test clean
(3204 passed, 32 skipped, 9 xfailed, 0 failed), docstring tests on
changed files clean.

Fixes TransformerLensOrg#1489
…alLM (TransformerLensOrg#1488)

* Add Ouro architecture adapter and integration tests for OuroForCausalLM

* add missing quarantine row for the gated ouro integration test

* feat: added Ouro architecture 'support_fold_ln = False' and test verification updates. Chore:  formatting updates in .MD
Adds a converter and loader that bring checkpoints from an external
decoder-only pretraining framework (RoPE with adjacent-pair rotation,
RMSNorm, SwiGLU, optional dropless top-k MoE) into HookedTransformer so
their residual streams can be read with the standard hook API.

- convert_maritime_pretrain_weights: splits the fused QKV projection into
  per-head W_Q/W_K/W_V, maps SwiGLU gate/up/down onto W_gate/W_in/W_out,
  and emits router + per-expert weights under the MoE component's names.
- maritime_pretrain_loader: builds the matching HookedTransformerConfig
  (rotary_adjacent_pairs=True, normalization_type=RMS, gated_mlp=True,
  norm_topk_prob=True for MoE), merges tensor-parallel shards, returns a
  ready HookedTransformer.
- Tests assert bit-exact logit equivalence (max|delta| < 1e-4) for both
  dense and MoE configs plus converter key/shape correctness.
@Cacapice Cacapice closed this Jul 11, 2026
@Cacapice Cacapice deleted the add-from-scratch-pretrain-converter branch July 11, 2026 06:44
@Cacapice Cacapice restored the add-from-scratch-pretrain-converter branch July 11, 2026 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants