Skip to content

feat(extract): add Twig and SCSS extractors - #2240

Open
amslezak wants to merge 2 commits into
Graphify-Labs:v8from
amslezak:feat/twig-scss-extractors
Open

amslezak wants to merge 2 commits into
Graphify-Labs:v8from
amslezak:feat/twig-scss-extractors

Conversation

@amslezak

@amslezak amslezak commented Jul 27, 2026 •

Copy link
Copy Markdown

What does this PR do?

Adds extractors for Twig templates and SCSS/Sass stylesheets. Neither extension is in any FileType set today, so detection skips both. On a Drupal or Symfony project the whole template layer is missing from the graph, and on a Sass project the stylesheet module graph and the design tokens are missing too.

Refs #3473 (the SCSS and design-token half; .css stays out of scope, see Limitations).

Both extractors follow extractors/blade.py: regex only, no new dependency. They route through FileType.CODE, so they cost no LLM tokens and produce EXTRACTED edges.

extractors/twig.py resolves references to real files:

  • Drupal SDC: {% include 'mytheme:card' %} resolves through the nearest <provider>.info.yml and an index of components/**/<name>/<name>.twig.
  • Plain paths and Symfony: {% extends 'layout.html.twig' %} and @Namespace/path.html.twig resolve against the referring file's directory and any enclosing templates/ root.
  • Relations: includes, embeds, extends, imports_macro, uses_template, defines_block, attaches_library. It reads the {% tag %} form, the {{ include('x') }} function form and {% from 'x' import y %}.

extractors/scss.py covers three graphs:

  • Modules: @use, @forward and @import, through Sass's partial (_name.scss) and index (name/_index.scss) rules. @import 'a', 'b' lists work. Built-ins like sass:math stay stubs.
  • Mixins: @mixin and @include key on the bare name, so @include mx.button-reset links to its definition.
  • Design tokens: --token: declarations and var(--token) references share one node.

A second commit fixes a resolution bug that the extractors exposed. An SDC keeps card.twig beside card.scss. Both get the card file id, so _disambiguate_colliding_node_ids salts them apart. An {% include %} edge from a third template carries neither salt's source_key, so it dangled on the retired card id. This is the #1475 C/ObjC header case (foo.h beside foo.c), and the fix mirrors that one: template-composition edges repoint to the .twig variant. The target_file gate also now covers these relations. It was limited to imports, imports_from and re_exports.

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Tests or CI
  • Refactor
  • Security fix

Verification & Invariants

  • No ghost nodes. A resolved reference emits the edge only, stamped with target_file, and mints no stub, following Absolute-path node ids on 0.9.25 for in-root relative import targets (residual of #1899 Variant B; ghost duplicates + dangling endpoints) #2195. A second node with the same id would duplicate the real file node.
  • No self-loops from comments. Both extractors blank out comments before they scan, and keep the newlines so line numbers stay correct. Component docblocks often show a usage example ({# Usage: {% include 'mytheme:card' %} #}). Read naively, that gives the component an includes edge to itself, which diagnose_extraction flags. test_twig.py and test_scss.py cover this case.
  • Deterministic. The SDC component index walks components/ in sorted order, so a component name that appears twice resolves the same way on every filesystem. test_duplicate_sdc_name_resolves_independent_of_filesystem_order runs both walk orders and fails without the sort. The extractors read no environment and make no network calls.
  • No cross-language stub remap. .twig, .scss and .sass are now in _LANG_FAMILY_BY_EXT. Without a family, _lang_family returns None, and the stub-remap pass could bind an SCSS mixin stub to a function with the same label in another language.

Measured on a Drupal theme with 141 templates and 109 stylesheets:

before after
includes dangling 198 / 234 0 / 234
embeds dangling 14 / 14 0 / 14
extends dangling 1 / 1 0 / 1
Sass imports resolved n/a 84 / 84
self-loop edges n/a 0

The only Twig references left unresolved point at base-theme templates outside the scanned directory. They correctly stay stubs.

Limitations

  • .css is out of scope. In a Sass project the CSS is compiled output of the .scss files, so indexing both duplicates the stylesheet layer. Hand-written CSS projects need a separate discussion, which No extraction path for .css/.scss: design-token systems are invisible to the graph (measured: 714 tokens, 1334 edges recoverable) #3473 has started.

  • The extractors use regex, like Blade, not tree-sitter. This keeps them free of new dependencies, but Twig expressions that build a template name at runtime (include(prefix ~ '-card')) are not resolved.

  • I have not run this on Windows. Path handling goes through pathlib.

  • Read the CONTRIBUTING.md guide.

  • Reproduced the issue and identified the invariant.

  • Made the smallest fix necessary.

  • Added a regression test (if bug fix) or isolated boundary test.

  • Kept the PR description synchronized with the final implementation.

  • Documented any limitations / unsupported cases explicitly.

How was this tested?

Rebased onto v8 at 0.9.92, with uv sync --all-extras:

uv run pytest tests -q
  2 failed, 8428 passed, 14 skipped
  Neither failure comes from this change. On a clean v0.9.92 checkout on the same macOS machine:
    test_skill_no_backtick_substitution.py::test_control_a_backticked_unknown_command_is_caught
      fails the same way
    test_hooks_rewrite.py fails one test per run with "the background rebuild did not settle",
      a different test each run (here test_range_cherry_pick_lands_every_new_function)
uv run pytest tests -q -k "twig or scss"
  38 passed
uv run ruff check .
  All checks passed!
uv run pyright
  641 errors, the same count as clean v0.9.92; 0 in the new files

I also ran the installed build against the Drupal theme above and checked the include edges against git grep.

Graphify-specific checklist

  • I updated generated skill artifacts (uv run python -m tools.skillgen --bless) when changing their source fragments. (No skill fragments changed.)
  • I confirmed that AST/structural extraction remains deterministic (no ambient state dependencies like ENV variables).
  • I reviewed changes for security implications (no unsafe interpolation into shell/Python).
  • I confirmed no API keys or local-only graph data are included.
  • (If applicable) I disclosed AI authorship in my commit messages.

nothariharan added a commit to nothariharan/graphify that referenced this pull request Oct 14, 2026
…s/.css/.scss/.less and dotfile configs (Graphify-Labs#2498, Graphify-Labs#2698, Graphify-Labs#3473)

An explicit wildcard-free '!path' entry in .graphifyignore now overrides only the Stage-3 secrets-keyword heuristic, never the .env/key-file or credential-store stages; a rescued file is kept and the skip report names the escape hatch (Graphify-Labs#2498).

Well-known extensionless config/build files (.htaccess, Dockerfile, Makefile, ...) are classified as documents instead of dropped; .less routes to the CSS extractor (Graphify-Labs#2698).

Combines and supersedes Graphify-Labs#2747/Graphify-Labs#3505 (.gs Apps Script), Graphify-Labs#3480/Graphify-Labs#3067/Graphify-Labs#2240 (CSS/SCSS design tokens) and Graphify-Labs#1921 (allowlist) against current v8.
@amslezak
amslezak force-pushed the feat/twig-scss-extractors branch from 234579d to 7651768 Compare October 17, 2026 19:54
@amslezak
amslezak requested a review from safishamsi as a code owner October 17, 2026 19:54
@github-actions github-actions Bot added the needs:title PR title should follow the conventional-commit format label Oct 17, 2026
@github-actions

Copy link
Copy Markdown

Thanks @amslezak! One small thing: our changelog and release notes are built from PR titles, so we use the conventional-commits format, for example:

fix(python): resolve a bare stdlib import correctly

Prefixes we use: feat, fix, docs, chore, refactor, test, perf, build, ci. Editing the PR title clears this automatically. This is just a nudge and does not block review.

amslezak and others added 2 commits October 17, 2026 16:04
Neither extension was in any FileType set, so Twig templates and Sass
stylesheets were silently skipped: on a Drupal theme that dropped the entire
component-composition and design-token layer from the graph.

Both are regex extractors in the shape of extractors/blade.py — no new
dependency, and routing them through FileType.CODE means they cost no LLM
tokens and yield EXTRACTED-confidence edges.

twig.py resolves references to real files rather than stubs:
- Drupal SDC (`{% include 'mytheme:card' %}`) via the nearest
  `<provider>.info.yml` plus a components/ index
- plain paths and Symfony's `@Namespace/...` via the referring file's dir
  and any enclosing templates/ root
- relations: includes, embeds, extends, imports_macro, uses_template,
  defines_block, attaches_library

scss.py covers the module, mixin, and design-token graphs:
- @use/@forward/@import through Sass partial (`_name.scss`) and index
  (`name/_index.scss`) conventions
- @mixin/@include keyed on the bare name so a namespaced use
  (`@include mx.button-reset`) links to its definition
- `--token:` declarations and `var(--token)` references share one node, so
  token-defining stylesheets connect to every consumer

Both strip comments before scanning, via a newline-preserving blank pass that
keeps reported line numbers accurate. Without it a component docblock showing
example usage ("Usage: {% include 'mytheme:card' %}") became a real dependency
and gave the component an edge to itself.

.twig/.scss/.sass are also added to _LANG_FAMILY_BY_EXT so their stubs cannot
be remapped onto same-labeled symbols in an unrelated language.

.css is deliberately out of scope: in a Sass project it is compiled output of
the .scss already indexed, so indexing both duplicates the stylesheet layer.

Verified against a 141-template / 109-stylesheet Drupal theme: 97% of Twig
references and 100% of Sass imports resolve to real files, no self-loops. The
unresolved remainder are base-theme templates outside the scanned corpus.

Refs: Graphify-Labs#3473
Co-Authored-By: Claude <noreply@anthropic.com>
…iguation

A Single Directory Component keeps `card.twig` beside `card.scss`, so both
collapse to the same `card` file id and _disambiguate_colliding_node_ids salts
them apart. An `{% include %}` edge from a THIRD template carries neither
salt's source_key, so the (target, edge_source_key) lookup missed and the edge
dangled on the retired `card` id.

That is the Graphify-Labs#1475 C/ObjC header failure exactly — `foo.h` beside `foo.c`, with
the import edge from a third file. Mirror its fix: repoint template-composition
edges to the .twig variant, since an include always targeted the template.

Also honor the target_file stamp for those relations. It was gated to
imports/imports_from/re_exports, so a stamped `includes` edge fell back to the
importer's own source_key.

On a 141-template Drupal theme this takes dangling composition edges from
198/234 to 0/234, and embeds from 14/14 to 0/14. The extractors emit the edge
only (no stub node) for a resolved reference, matching the documented Graphify-Labs#2195
pattern, so no ghost node duplicates the real file node either.

Co-Authored-By: Claude <noreply@anthropic.com>
@amslezak
amslezak force-pushed the feat/twig-scss-extractors branch from 7651768 to 9cb498b Compare October 19, 2026 15:52
@amslezak amslezak changed the title Add Twig and SCSS extractors feat(extract): add Twig and SCSS extractors Oct 19, 2026
@github-actions github-actions Bot removed the needs:title PR title should follow the conventional-commit format label Oct 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant