Skip to content

Add Lfm2 support#1510

Open
TensorCruncher wants to merge 5 commits into
TransformerLensOrg:devfrom
TensorCruncher:add-lfm2-support
Open

Add Lfm2 support#1510
TensorCruncher wants to merge 5 commits into
TransformerLensOrg:devfrom
TensorCruncher:add-lfm2-support

Conversation

@TensorCruncher

Copy link
Copy Markdown
Contributor

Description

Adds support for LFM2. Includes:

  1. New Lfm2ShortConv bridge
  2. Lfm2 adapter
  3. Unit tests for adapter

Verify model test ran on [LiquidAI/LFM2.5-230M, LiquidAI/LFM2-350M, LiquidAI/LFM2-1.2B]. Tests pass except harness fails on conv blocks due to difference in shape (channels last vs first). This could be fixed by adding transpose step for conv layers in test harness.

Also, compatibility mode test fails. I will paste test results for 1.2B model in comment below for reference.

Questions:

  1. Does the new bridge look okay? I let it inherit the forward method from the base class.
  2. Should I limit applicable tests to [4] or leave it empty line nemotron?
  3. Should I add an integration test as nemotron does since verify models does not fully work with non transformer / attention models?
  4. Once this is done, shall I update the existing Lfm2Moe model to use the new bridge along with gatedMoE bridge? It currently wraps around the decoder layer with no support for hooks around conv/attn or MoE. This can be done in a seperate PR.

Fixes #1461

Type of change

  • New feature (non-breaking change which adds functionality)

@TensorCruncher

Copy link
Copy Markdown
Contributor Author
adil@Syeds-MacBook-Pro TransformerLens % uv run python -m transformer_lens.tools.model_registry.verify_models \
  --model LiquidAI/LFM2-1.2B \
  --max-memory 16 \
  --device cpu \
  --dtype float32 \
  --no-ht-reference
Model list mode: 1 model(s)
Selected 1 models for verification
2026-07-12 15:34:00,685 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/config.json "HTTP/1.1 200 OK"
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-07-12 15:34:00,686 [WARNING] Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-07-12 15:34:00,906 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/tokenizer_config.json "HTTP/1.1 200 OK"
2026-07-12 15:34:01,120 [INFO] HTTP Request: GET https://huggingface.co/api/models/gpt2/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:34:01,334 [INFO] HTTP Request: GET https://huggingface.co/api/models/openai-community/gpt2/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-07-12 15:34:01,558 [INFO] HTTP Request: GET https://huggingface.co/api/models/gpt2/tree/main?recursive=true&expand=false "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:34:01,810 [INFO] HTTP Request: GET https://huggingface.co/api/models/openai-community/gpt2/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
2026-07-12 15:34:02,219 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/config.json "HTTP/1.1 200 OK"
Loading weights: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 148/148 [00:00<00:00, 18110.54it/s]
2026-07-12 15:34:02,556 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/generation_config.json "HTTP/1.1 200 OK"
Pre-loaded GPT-2 scoring model for Phase 4

======================================================================
[1/1] LiquidAI/LFM2-1.2B (Lfm2ForCausalLM)
======================================================================
2026-07-12 15:34:02,776 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:34:02,793 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-12 15:34:02,812 [INFO] HTTP Request: GET https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
config.json: 1.00kB [00:00, 1.60MB/s]
  Estimated parameters: 1,207,959,552
  Estimated benchmark memory: 10.8 GB (limit: 16.0 GB)
  Running phases [1, 2, 3, 4] in a single benchmark call...

================================================================================
Running TransformerBridge Benchmark Suite
Model: LiquidAI/LFM2-1.2B
Device: cpu
================================================================================


================================================================================
PHASE 1: HuggingFace + TransformerBridge (unprocessed)
================================================================================

Detecting model configuration...
2026-07-12 15:34:03,035 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:34:03,050 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-12 15:34:21,395 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:34:21,412 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-12 15:34:21,658 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:34:21,948 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/tokenizer_config.json "HTTP/1.1 200 OK"
2026-07-12 15:34:22,007 [INFO] HTTP Request: GET https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/tokenizer_config.json "HTTP/1.1 200 OK"
tokenizer_config.json: 91.6kB [00:00, 135MB/s]
2026-07-12 15:34:22,242 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-07-12 15:34:22,494 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
2026-07-12 15:34:22,712 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/tokenizer.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:34:22,733 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/tokenizer.json "HTTP/1.1 200 OK"
2026-07-12 15:34:22,755 [INFO] HTTP Request: GET https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/tokenizer.json "HTTP/1.1 200 OK"
tokenizer.json: 4.73MB [00:00, 79.0MB/s]
2026-07-12 15:34:23,038 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/tokenizer.model "HTTP/1.1 404 Not Found"
2026-07-12 15:34:23,279 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/added_tokens.json "HTTP/1.1 404 Not Found"
2026-07-12 15:34:23,491 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/special_tokens_map.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:34:23,506 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/special_tokens_map.json "HTTP/1.1 200 OK"
2026-07-12 15:34:23,527 [INFO] HTTP Request: GET https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/special_tokens_map.json "HTTP/1.1 200 OK"
special_tokens_map.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 434/434 [00:00<00:00, 569kB/s]
2026-07-12 15:34:23,746 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/chat_template.jinja "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:34:23,761 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/chat_template.jinja "HTTP/1.1 200 OK"
2026-07-12 15:34:23,775 [INFO] HTTP Request: GET https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/chat_template.jinja "HTTP/1.1 200 OK"
chat_template.jinja: 1.38kB [00:00, 3.89MB/s]
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
  getattr(module, "_register_aliases")()
✓ Detected attn_implementation=eager
Loading HuggingFace reference model...
Using attn_implementation=eager
2026-07-12 15:34:24,644 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:34:24,661 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-12 15:34:24,879 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:34:24,898 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-12 15:34:25,144 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:34:25,156 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-12 15:34:25,379 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/model.safetensors "HTTP/1.1 302 Found"
2026-07-12 15:34:25,614 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/xet-read-token/933cee00d754fb3bfe06c644c0cb95453f2d8bb2 "HTTP/1.1 200 OK"
model.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 2.34G/2.34G [02:59<00:00, 13.0MB/s]
Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 148/148 [00:02<00:00, 53.35it/s]
2026-07-12 15:37:28,258 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/generation_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:37:28,272 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/generation_config.json "HTTP/1.1 200 OK"
2026-07-12 15:37:28,291 [INFO] HTTP Request: GET https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/generation_config.json "HTTP/1.1 200 OK"
generation_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 137/137 [00:00<00:00, 227kB/s]
Detected dtype=torch.float32
✓ HuggingFace model loaded
Loading TransformerBridge (unprocessed)...
2026-07-12 15:37:28,527 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:37:28,541 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 148/148 [00:01<00:00, 97.92it/s]
2026-07-12 15:37:30,379 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/generation_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:37:30,396 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/generation_config.json "HTTP/1.1 200 OK"
2026-07-12 15:37:30,621 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:37:30,634 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-12 15:37:30,867 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:37:30,882 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/tokenizer_config.json "HTTP/1.1 200 OK"
2026-07-12 15:37:31,106 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-07-12 15:37:31,425 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
✓ TransformerBridge loaded (unprocessed)

Running Phase 1 benchmarks...

1. Component-Level Benchmarks
❌ [FAIL] all_components: 10/114 components failed (10 critical)
  total_components: 114
  passed_components: 104
  failed_components: 10
  pass_rate: 91.22807017543859
  failures: {'blocks.0.conv.conv': {'max_diff': inf, 'mean_diff': inf, 'severity': 'critical', 'error': 'Given groups=2048, weight of size [2048, 1, 3], expected input[2, 8, 2048] to have 2048 channels, but got 8 channels instead'}, 'blocks.1.conv.conv': {'max_diff': inf, 'mean_diff': inf, 'severity': 'critical', 'error': 'Given groups=2048, weight of size [2048, 1, 3], expected input[2, 8, 2048] to have 2048 channels, but got 8 channels instead'}, 'blocks.3.conv.conv': {'max_diff': inf, 'mean_diff': inf, 'severity': 'critical', 'error': 'Given groups=2048, weight of size [2048, 1, 3], expected input[2, 8, 2048] to have 2048 channels, but got 8 channels instead'}, 'blocks.4.conv.conv': {'max_diff': inf, 'mean_diff': inf, 'severity': 'critical', 'error': 'Given groups=2048, weight of size [2048, 1, 3], expected input[2, 8, 2048] to have 2048 channels, but got 8 channels instead'}, 'blocks.6.conv.conv': {'max_diff': inf, 'mean_diff': inf, 'severity': 'critical', 'error': 'Given groups=2048, weight of size [2048, 1, 3], expected input[2, 8, 2048] to have 2048 channels, but got 8 channels instead'}, 'blocks.7.conv.conv': {'max_diff': inf, 'mean_diff': inf, 'severity': 'critical', 'error': 'Given groups=2048, weight of size [2048, 1, 3], expected input[2, 8, 2048] to have 2048 channels, but got 8 channels instead'}, 'blocks.9.conv.conv': {'max_diff': inf, 'mean_diff': inf, 'severity': 'critical', 'error': 'Given groups=2048, weight of size [2048, 1, 3], expected input[2, 8, 2048] to have 2048 channels, but got 8 channels instead'}, 'blocks.11.conv.conv': {'max_diff': inf, 'mean_diff': inf, 'severity': 'critical', 'error': 'Given groups=2048, weight of size [2048, 1, 3], expected input[2, 8, 2048] to have 2048 channels, but got 8 channels instead'}, 'blocks.13.conv.conv': {'max_diff': inf, 'mean_diff': inf, 'severity': 'critical', 'error': 'Given groups=2048, weight of size [2048, 1, 3], expected input[2, 8, 2048] to have 2048 channels, but got 8 channels instead'}, 'blocks.15.conv.conv': {'max_diff': inf, 'mean_diff': inf, 'severity': 'critical', 'error': 'Given groups=2048, weight of size [2048, 1, 3], expected input[2, 8, 2048] to have 2048 channels, but got 8 channels instead'}}
✗ 10/114 components failed (10 critical)

Capturing HF reference outputs to CPU...
2026-07-12 15:37:37,943 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:37:37,963 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
✓ Captured HF logits torch.Size([1, 28, 65536]), loss=2.5977

Cleaning up HuggingFace model...
2. Forward Pass Benchmarks
🟢 [PASS] forward_pass_logits: Tensors match within tolerance
  atol: 0.001
  rtol: 0.03
✓ Saved Phase 1 reference data (logits: torch.Size([1, 28, 65536]))

================================================================================
PHASE 2: TransformerBridge (unprocessed) + HookedTransformer (unprocessed)
================================================================================

Running Phase 2 benchmarks...

2026-07-12 15:37:42,550 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:37:42,571 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
1. Generation Benchmarks (unprocessed)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00,  3.58it/s]
🟢 [PASS] generation: Generation successful: 28 -> 38 tokens (176 -> 234 chars)
  input_tokens: 28
  output_tokens: 38
  input_chars: 176
  output_chars: 234
  max_new_tokens: 10
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00,  3.88it/s]
🟢 [PASS] generation_with_kv_cache: KV cache generation successful (238 chars)
  output_len: 238
  max_new_tokens: 10
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00,  2.83it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00,  2.88it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00,  2.76it/s]
🟢 [PASS] multiple_generation_calls: All 3 generation calls successful
  prompt_count: 3
  max_new_tokens: 5
  output_lens: [43, 27, 62]
2. Running Unprocessed Model Comparison Benchmarks

2. Model Equivalence Benchmarks (Forward Pass)
⏭️ Skipped (no HookedTransformer reference)

⏭️ [SKIPPED] logits_equivalence: Skipped (HookedTransformer not available for this model)
⏭️ [SKIPPED] loss_equivalence: Skipped (HookedTransformer not available for this model)
3. Hook Registration Benchmarks
🟢 [PASS] hook_registry: Bridge has 588 registered hooks
  hook_count: 588
4. Forward Hook Functionality Benchmarks
🟡 [PASS] hook_functionality: Ablation had minimal effect: 0.000000
  effect: 0.0
🟢 [PASS] critical_forward_hooks: Bridge captured 8/14 critical hooks
  captured: 8
  expected: 14
🟡 [PASS] forward_hooks: 78/787 hooks didn't fire during forward pass
  captured: 709
  registered: 787
  didnt_fire: ['blocks.10.attn.hook_v_input', 'blocks.5.hook_mlp_in', 'blocks.2.hook_attn_in', 'blocks.14.hook_k_input', 'blocks.5.hook_attn_in', 'blocks.12.hook_v_input', 'blocks.2.attn.hook_v_input', 'blocks.14.attn.hook_attn_in', 'blocks.1.hook_mlp_in', 'blocks.10.hook_attn_in']
🟢 [PASS] gated_hooks_fire: All gated hooks fired on their supporting layers (54 activations across 5 hook stems, 3 flags tested)
  fired_counts: {'hook_result': 6, 'hook_q_input': 12, 'hook_k_input': 12, 'hook_v_input': 12, 'hook_attn_in': 12}
  tested_flags: ['use_attn_result', 'use_split_qkv_input', 'use_attn_in']
  skipped: []
5. Activation Cache Benchmarks
🟢 [PASS] run_with_cache: run_with_cache successful with 709 cached activations
  cache_size: 709
🟢 [PASS] activation_cache: Activation cache created with 709 entries
  cache_size: 709
6. Backward Gradient Benchmarks
🟢 [PASS] gradient_computation: Gradients computed successfully
🟢 [PASS] critical_backward_hooks: Bridge captured 7/12 critical backward gradients
  captured: 7
  expected: 12
🟢 [PASS] backward_hooks: Bridge captured 429 backward hook gradients
  gradient_count: 429
2026-07-12 15:38:13,909 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:38:13,928 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-12 15:38:14,151 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:38:14,170 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"

================================================================================
PHASE 2.5: Text Quality (GPT-2 perplexity scoring)
================================================================================

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:10<00:00,  4.96it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00,  5.29it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00,  5.30it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00,  5.36it/s]
🟢 [PASS] text_quality: Text quality score: 99.3/100 (avg perplexity: 22.9, 4 prompts)
  score: 99.3
  avg_perplexity: 22.87
  avg_repetition_penalty: 0.99
  num_prompts: 4
  per_prompt: ppl=15.6 score=100.0 rep=1.00 | ppl=9.9 score=97.6 rep=0.98 | ppl=34.7 score=99.5 rep=1.00 | ppl=31.3 score=100.0 rep=1.00
  scoring_model: gpt2
  max_new_tokens: 50
  generated_text: Natural language processing tasks, such as question answering, machine translation, reading comprehension, and summarization, are typically approached with supervised learning. However, these tasks often require large amounts of labeled data, which can be hard to come by, especially for niche domains. Self-supervised learning (SSL) has emerged as a powerful technique to address this challenge by leveraging unlabeled data to
2026-07-12 15:38:53,001 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-12 15:38:53,014 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"

================================================================================
PHASE 3: TransformerBridge (processed) + HookedTransformer (processed)
================================================================================

Processing weights on existing bridge (reusing Phase 1 instance)...
✓ TransformerBridge compatibility mode enabled (processed)

Running Phase 3 benchmarks...

1. Weight Processing Benchmarks (Foundation)
🟢 [PASS] no_nan_inf: No NaN or Inf values found in weights
  num_tensors_checked: 150
🟢 [PASS] weight_magnitudes: All weight magnitudes in reasonable ranges
  min_threshold: 1e-06
  max_threshold: 1000.0
🟡 [PASS] layer_norm_folding: Partial LN folding: ln2 (mean=1.000000) folded; ln1 (mean=0.235300) preserved (expected for combined QKV models)
  folded: ['ln2']
  not_folded: ['ln1']
🟢 [PASS] attention_output_centering: Attention output centering verified (worst_mean=0.000390) (6/16 blocks have attention)
  mean: 0.00039015396032482386
  tolerance: 0.01
  n_attn_blocks: 6
🟢 [PASS] mlp_output_centering: MLP output centering verified (mean=0.000185)
  mean: 0.00018529227236285806
  tolerance: 0.01
🟢 [PASS] unembed_centering: Unembed centering verified (mean=0.000000)
  mean: 8.756500524675914e-10
  tolerance: 0.01
  key: unembed.weight
🟢 [PASS] value_bias_folding: Skipped for GQA models (n_key_value_heads != n_heads)
  is_gqa: True
  n_heads: 32
  n_kv_heads: 8
⏭️ weight_processing and weight_sharing skipped (no HT reference)
⏭️ [SKIPPED] weight_processing: Skipped (HookedTransformer not available for this model)
⏭️ [SKIPPED] weight_sharing: Skipped (HookedTransformer not available for this model)
🟢 [PASS] weight_modification: Weight modification propagates correctly (change: 0.101187)
  change: 0.10118675231933594
2. Model Equivalence Benchmarks (Forward Pass)
Using saved Phase 1 bridge reference for equivalence comparison
🔴 [FAIL] logits_equivalence: Tensors differ: max_diff=23.349289, mean_rel=0.575189
  max_diff: 23.349288940429688
  mean_diff: 6.9689178466796875
  mean_rel: 0.5751886367797852
  atol: 0.01
  rtol: 0.0001
🔴 [FAIL] loss_equivalence: Scalars differ: 12.080062 vs 2.597675
  diff: 9.48238730430603
  atol: 0.001
3. Hook Registration Benchmarks
🟢 [PASS] hook_registry: Bridge has 787 registered hooks
  hook_count: 787
4. Forward Hook Functionality Benchmarks
🟡 [PASS] hook_functionality: Ablation had minimal effect: 0.000000
  effect: 0.0
🟢 [PASS] critical_forward_hooks: Bridge captured 8/14 critical hooks
  captured: 8
  expected: 14
🟡 [PASS] forward_hooks: 126/787 hooks didn't fire during forward pass
  captured: 661
  registered: 787
  didnt_fire: ['blocks.10.attn.hook_v_input', 'blocks.13.mlp.in.hook_in', 'blocks.2.hook_attn_in', 'blocks.10.mlp.out.hook_out', 'blocks.14.hook_k_input', 'blocks.5.hook_mlp_in', 'blocks.5.hook_attn_in', 'blocks.12.hook_v_input', 'blocks.2.attn.hook_v_input', 'blocks.2.mlp.out.hook_out']
🟢 [PASS] gated_hooks_fire: All gated hooks fired on their supporting layers (54 activations across 5 hook stems, 3 flags tested)
  fired_counts: {'hook_result': 6, 'hook_q_input': 12, 'hook_k_input': 12, 'hook_v_input': 12, 'hook_attn_in': 12}
  tested_flags: ['use_attn_result', 'use_split_qkv_input', 'use_attn_in']
  skipped: []
5. Activation Cache Benchmarks
🟢 [PASS] run_with_cache: run_with_cache successful with 661 cached activations
  cache_size: 661
🟢 [PASS] activation_cache: Activation cache created with 661 entries
  cache_size: 661
6. Backward Gradient Benchmarks
🟢 [PASS] gradient_computation: Gradients computed successfully
🟢 [PASS] critical_backward_hooks: Bridge captured 7/12 critical backward gradients
  captured: 7
  expected: 12
🟢 [PASS] backward_hooks: Bridge captured 555 backward hook gradients
  gradient_count: 555
Cleaning up TransformerBridge (processed)...

================================================================================
BENCHMARK SUMMARY
================================================================================

Phase 1: 2 tests run
  Passed: 1/2 (50.0%)
  Failed: 1/2 (50.0%)

Phase 2: 13 tests run
  Passed: 13/13 (100.0%)
  Failed: 0/13 (0.0%)
  Skipped: 2

Phase 3: 20 tests run
  Passed: 18/20 (90.0%)
  Failed: 2/20 (10.0%)
  Skipped: 2

Phase 4: 1 tests run
  Passed: 1/1 (100.0%)
  Failed: 0/1 (0.0%)

Overall:
Total: 40 tests
Run: 36 tests
Skipped: 4 tests
Passed: 33/36 (91.7%)
Failed: 3/36 (8.3%)
================================================================================
  FAILED: Below threshold: P1=50.0% < 100.0% (failed: all_components); P3=90.0% but required tests failed: log — 10/114 components failed (10 critical)
  Partial scores saved: P1=50.0%, P2=100.0%, P3=90.0%, P4=99.3%, P7=None%, P8=None%

======================================================================
Verification Summary
======================================================================
  Total tested:  1
  Verified:      0
  Skipped:       0
  Failed:        1

  Failed models:
    - LiquidAI/LFM2-1.2B

Total time: 325.5s
Checkpoint cleared (run complete)
adil@Syeds-MacBook-Pro TransformerLens % 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant