Add from scratch pretrain converter#1504
Closed
Cacapice wants to merge 12 commits into
Closed
Conversation
Adds TransformerBridge support for Arcee AI's AFM-4.5B (ArceeForCausalLM), a
Llama-style dense decoder whose only architectural novelty is an ungated
squared-ReLU (ReLU^2) MLP (up_proj -> ReLU^2 -> down_proj) instead of the gated
SiLU/GeLU used by Llama.
- Register the "relu2" activation (relu(x)**2) in SUPPORTED_ACTIVATIONS so the
config guard and ActivationFunctionFactory accept HF's hidden_act="relu2".
- New ArceeArchitectureAdapter: Llama attention (RoPE, RMSNorm, GQA, no QK-norm,
no biases) + ungated MLPBridge. Post-activation ReLU^2 neurons are inspectable
via blocks.{i}.mlp.hook_post.
- Register at the four sites (factory, supported_architectures __init__,
model_registry HF_SUPPORTED_ARCHITECTURES + CANONICAL_AUTHORS_BY_ARCH,
generate_report descriptions) and add the two checkpoints to the registry.
- Unit + integration parity tests (fp32 + eager); integration asserts bridge
logits match HF and hook_post exposes non-negative d_mlp-width activations.
Closes TransformerLensOrg#1467
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…red-relu-mlp Add Arcee squared-ReLU MLP adapter (ArceeForCausalLM)
…o dev) (TransformerLensOrg#1482) * Added minimal TransformerBridge support skeleton for Qwen2MoeForCausalLM * Fix return type annotation in Qwen2MoeRouterBridge.forward method * Enhance tests for Qwen2-MoE bridge to validate cache capturing of MoE hooks --------- Co-authored-by: Canonik <alessandro006@icloud.com>
* Normalized SSM mixer across existing hybrids * GatedDeltaNet and compute_ssm_state improvements * Prepping Mamba1 mixer * Unify vocabulary * SSM2 interp support * Mamba1 read parity * Fixed Mamba1 bugs * Fix GPU device mismatch * Additional cleanup and testing * make format * Model verification + comment cleanup * documentation updates * Final SSM pass
…LM) (TransformerLensOrg#1480) * Add Falcon H1 Hybrid Parallel Mamba TransformerBridge Adapter * Add Tiny-90M parity tests and attention-branch ablation coverage for Falcon-H1 adapter * Drop redundant CI skipif from Falcon-H1 integration tests; rely on slow marker * Fix mypy attr-defined for Falcon-H1 eps_attr assignment --------- Co-authored-by: jlarson4 <jonahalarson@comcast.net>
* Add BD3LM architecture adapter * Address review: fix dead hook_attn_out, elif ordering, add integration test and verify_models support --------- Co-authored-by: Jonah Larson <jonahalarson@comcast.net>
…#1483) * Add RecurrentGemma (Griffin) architecture adapter Adds a TransformerBridge adapter for RecurrentGemmaForCausalLM (Griffin): RG-LRU real-gated linear recurrence interleaved with local sliding-window attention (config.block_types, default recurrent/recurrent/attention). Because the temporal_block substructure varies per layer (recurrent vs. attention), the adapter wraps each decoder layer whole with residual-stream hooks only (RecurrentGemmaBlockBridge), mirroring Lfm2MoeArchitectureAdapter — rather than pretending every layer has a homogeneous attention/MLP substructure. applicable_phases=[4] (generation); finer-grained RG-LRU state hooks are a follow-up. Sets the Gemma-family numerics on the bridge config: RMSNorm (1.0 + weight) offset, gated MLP, final_rms, and the tanh final-logit soft cap (config.logits_soft_cap=30.0). ln_final maps to model.final_norm. Registers the adapter at the factory + supported_architectures __init__ sites and adds an offline structural unit test. Closes TransformerLensOrg#1472 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Apply black formatting to RecurrentGemma adapter test Collapse two test method signatures onto single lines per black (line-length 100). No behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Register RecurrentGemma in the model-registry sync tables The RegistrySyncedWithFactory unit tests require every factory architecture to appear in HF_SUPPORTED_ARCHITECTURES and CANONICAL_AUTHORS_BY_ARCH (or INTENTIONAL_EXCLUDES). Add RecurrentGemmaForCausalLM to both (canonical author: google). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: re-trigger checks (flaky nbval notebook) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…1486) * feat: add Zamba2ForCausalLM architecture adapter Closes TransformerLensOrg#1463 Supports Zyphra/Zamba2-1.2B, Zamba2-7B, Zamba2-7B-Instruct via TransformerBridge. Zamba2 is a hybrid Mamba-2 / shared-attention model where layers_block_type alternates between "mamba" (pure SSM) and "hybrid" (SSM + shared global-attention block, weight-tied across num_mem_blocks slots). Key design decisions: - SSMBlockBridge for all layers: hook_in/hook_out fire regardless of layer type. Mamba layers expose mixer submodule hooks (in_proj, conv1d, inner_norm, out_proj); hybrid layers have norm and mixer marked optional=True so component_setup skips them gracefully. - is_stateful=False: Zamba2 threads its unified cache via past_key_values (standard KV path), not the Mamba cache_params path. Setting is_stateful=True collided with Zamba2's own cache call. - positional_embedding_type="none": RoPE is handled inside the HF attention block; no model-level rotary bridge needed. - Added mamba_expand, mamba_ngroups, num_mem_blocks, layers_block_type, use_shared_attention_adapter to _HF_PASSTHROUGH_ATTRS in both sources/transformers.py and sources/_bridge_builder.py so these config fields reach the adapter reliably. 21 integration tests pass against Zyphra/Zamba2-1.2B (38 layers: 32 mamba + 6 hybrid). Forward parity is exact (max_diff == 0.0); greedy generation matches HF bit-for-bit across both layer types. * fix: add Zamba2ForCausalLM to model registry (HF_SUPPORTED_ARCHITECTURES + CANONICAL_AUTHORS_BY_ARCH)
… sentinel (TransformerLensOrg#1491) Consolidates the tanh soft-capping (Gemma-2 style) used for attention scores and output logits into apply_softcap()/softcap_enabled() in transformer_lens/utilities/activation_functions.py, replacing five independent cap * tanh(x / cap) implementations across HookedTransformer, abstract_attention, and the native TransformerBridge model, plus the center_unembed compatibility checks in both systems. Also replaces the bare -1.0 disabled sentinel with a named SOFTCAP_DISABLED constant in both HookedTransformerConfig and TransformerBridgeConfig. Pure refactor, no behavior change. A naive 'if cap:' truthiness check would treat -1.0 as enabled (computing -1.0 * tanh(x / -1.0) == tanh(x), saturating every large score to 1.0); softcap_enabled() makes that mistake structurally impossible instead of relying on every call site getting 'cap > 0' right independently. Verified: uv run mypy . clean (307 files), make unit-test clean (3204 passed, 32 skipped, 9 xfailed, 0 failed), docstring tests on changed files clean. Fixes TransformerLensOrg#1489
…alLM (TransformerLensOrg#1488) * Add Ouro architecture adapter and integration tests for OuroForCausalLM * add missing quarantine row for the gated ouro integration test * feat: added Ouro architecture 'support_fold_ln = False' and test verification updates. Chore: formatting updates in .MD
Adds a converter and loader that bring checkpoints from an external decoder-only pretraining framework (RoPE with adjacent-pair rotation, RMSNorm, SwiGLU, optional dropless top-k MoE) into HookedTransformer so their residual streams can be read with the standard hook API. - convert_maritime_pretrain_weights: splits the fused QKV projection into per-head W_Q/W_K/W_V, maps SwiGLU gate/up/down onto W_gate/W_in/W_out, and emits router + per-expert weights under the MoE component's names. - maritime_pretrain_loader: builds the matching HookedTransformerConfig (rotary_adjacent_pairs=True, normalization_type=RMS, gated_mlp=True, norm_topk_prob=True for MoE), merges tensor-parallel shards, returns a ready HookedTransformer. - Tests assert bit-exact logit equivalence (max|delta| < 1e-4) for both dense and MoE configs plus converter key/shape correctness.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a weight converter and loader that bring checkpoints from an external
from-scratch pretraining framework into
HookedTransformer, so afoundation model trained outside the HuggingFace ecosystem can be analysed with
the standard TransformerLens hook API (
run_with_cache,resid_post, etc.).The source architecture is a decoder-only transformer with RoPE (adjacent-pair
rotation), RMSNorm, SwiGLU MLPs, and optional dropless top-k Mixture-of-Experts.
Its checkpoints are not loadable through
from_pretrainedbecause they are nota HuggingFace model; this PR follows the same pattern as the existing
nanogpt/mixtralconverters to support them.Motivation: this is the base-model half of an interpretability project — the
model whose residual stream downstream linear probes read. Loading it into
TransformerLens means the probing side can use the library's tooling directly
instead of a bespoke activation-extraction path.
Fixes # (no issue — new-feature contribution)
Type of change
to add a
docs/loading example if the maintainers would like one)What's included
weight_conversions/maritime_pretrain.py—convert_maritime_pretrain_weights.Splits the fused QKV projection into per-head
W_Q/W_K/W_V, maps SwiGLUgate/up/downonto TransformerLens'sW_gate/W_in/W_out, and (for MoE) emitsthe router plus every expert under the names
MoE/MoEGatedMLPexpect. NoQ/K reordering is needed because RoPE is applied inside attention, not baked
into the weights.
pretrained/maritime_pretrain_loader.py—build_configandload_maritime_pretrain. Builds the matchingHookedTransformerConfig(
positional_embedding_type="rotary",rotary_adjacent_pairs=True,normalization_type="RMS",final_rms=True,gated_mlp=True, andnum_experts/experts_per_token/norm_topk_probfor MoE), mergestensor-parallel weight shards, and returns a ready
HookedTransformer.weight_conversions/__init__.py— export the new converter (alphabetical).tests/unit/pretrained_weight_conversions/test_maritime_pretrain.py.Key correctness detail
The source rotates adjacent RoPE pairs (
[x0, x1] -> [-x1, x0]), which maps torotary_adjacent_pairs=True(the GPT-NeoX-styled -> (d 2)frequency layout),not the default half-split convention. Getting this wrong still loads without
error but silently corrupts every activation, so the equivalence tests below are
the real guardrail.
Checklist
Test evidence
test_maritime_pretrain.pyasserts bit-exact logit equivalence between thesource model and its converted
HookedTransformer, parametrised across the axesa future change is most likely to break, plus structural, failure-path and
tensor-parallel coverage (13 tests):
(d_model, n_heads)combinationsValueError(not an opaquereshape error)
N ∈ {1, 2, 4} shards
Observed
max|Δlogit|between source and converted model ≈1e-5(floating-point noise) across every parametrisation. Converter validates each
source weight's shape before reshaping, so a mismatched config fails with an
actionable message. Files formatted with the repo's pinned
black==23.3.0/isort==5.8.0;pylint10.00/10 andpyrightclean on the new modules.Notes / open questions for reviewers
maritime_pretrainto mirrornanogpt. If you'd prefer a generic name (e.g.swiglu_moe_gpt) since theconverter isn't domain-specific, I'm glad to rename.
pretrained/maritime_pretrain_loader.pyrather than being wired into
loading_from_pretrained.py, because thesecheckpoints load from a local run directory, not a hub ID. If you'd rather
register it as an official model source, point me at the preferred hook.
devper the contribution guidelines (this is a feature, not afix against the released version).