Skip to content

[Attention] Lazy-load LLM and ViT attention backends - #1668

Merged
shihaobai merged 1 commit into
mainfrom
bsh/att_backend
Oct 30, 2026
Merged

shihaobai merged 1 commit into
mainfrom
bsh/att_backend

Conversation

@shihaobai

@shihaobai shihaobai commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

No description provided.

@coderabbitai

coderabbitai Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1db91f77-2ff7-4b0e-8409-2d371d1aec11
Commits

Reviewing files that changed from the base of the PR and between 6a6e980 and 2c4b9b3.

Files selected for processing: 11
  • lightllm/common/basemodel/attention/__init__.py
  • lightllm/common/basemodel/attention/create_linear_utils.py
  • lightllm/common/basemodel/attention/create_utils.py
  • lightllm/common/basemodel/attention/import_utils.py
  • lightllm/common/basemodel/attention/linear/__init__.py
  • lightllm/common/basemodel/attention/nsa/__init__.py
  • lightllm/common/basemodel/attention/nsa/dsv4_fp8_flashmla_sparse.py
  • lightllm/common/basemodel/attention_vit/create_utils.py
  • lightllm/models/qwen3_dflash/model.py
  • unit_tests/common/basemodel/attention/linear/test_gdn.py
  • unit_tests/common/basemodel/attention/test_lazy_import.py
Files with no reviewable changes: 3
  • lightllm/common/basemodel/attention/linear/init.py
  • lightllm/common/basemodel/attention/init.py
  • lightllm/common/basemodel/attention/nsa/init.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 6 remain after this review.

Walkthrough

Attention backend registries now store import paths instead of eagerly imported classes. Selection functions load the chosen class. Package-level backend re-exports were removed, and tests cover lazy loading across regular, MLA, NSA, Neo, linear, and ViT selection.

Changes: Lazy Attention Backend Loading

Layer / File(s) Summary
Backend import contract
lightllm/common/basemodel/attention/import_utils.py, lightllm/common/basemodel/attention/__init__.py, lightllm/common/basemodel/attention/linear/__init__.py, lightllm/common/basemodel/attention/nsa/__init__.py, lightllm/models/qwen3_dflash/model.py
Added load_backend_class to resolve module-and-class paths. Removed package-level backend re-exports and changed the Qwen model to import FA3 classes from their backend modules.
Attention backend selection
lightllm/common/basemodel/attention/create_utils.py, lightllm/common/basemodel/attention/create_linear_utils.py, lightllm/common/basemodel/attention/nsa/dsv4_fp8_flashmla_sparse.py, unit_tests/common/basemodel/attention/test_lazy_import.py, unit_tests/common/basemodel/attention/linear/test_gdn.py
Regular, MLA, NSA, Neo, and linear backend maps now use import paths. Selectors load configured, validated, or fallback backends when selected. Tests cover selection order, import isolation, and fallback behavior.
ViT backend selection
lightllm/common/basemodel/attention_vit/create_utils.py, unit_tests/common/basemodel/attention/test_lazy_import.py
ViT backend maps now use import paths, and the selected class is loaded during selection. Tests cover explicit and automatic selection.

Priority: Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Refactor

Sequence Diagram(s)

sequenceDiagram
  participant PrefillSelector
  participant _auto_select_backend
  participant load_backend_class
  participant importlib
  PrefillSelector->>_auto_select_backend: Validate candidates in priority order
  _auto_select_backend->>load_backend_class: Load selected import path
  load_backend_class->>importlib: Import module relative to attention package
  importlib-->>load_backend_class: Return imported module
  load_backend_class-->>_auto_select_backend: Return backend class
  _auto_select_backend-->>PrefillSelector: Return selected backend class
Loading

Suggested reviewers: sufubao, sangchengc, wandy666

Merge Risk: Minimal · up to 2c4b9

No actionable issue was established in the changed backend-loading paths. The PR is ready to merge after normal checks.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 6.25% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: lazy loading of LLM and ViT attention backends.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 6.25% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 16 functions across 8 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches: 💡 1
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@shihaobai
shihaobai merged commit ccf4e33 into main Oct 30, 2026
2 checks passed
@shihaobai
shihaobai deleted the bsh/att_backend branch October 30, 2026 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant