Progress towards tree-sitter feature by saniyafatima07 · Pull Request #3102 · mandiant/capa

saniyafatima07 · 2026-06-10T18:39:59Z

Addresses PR #1080 and #2931

This PR updates capa’s Tree-sitter support for script analysis and fixes the related integration issues so the feature works correctly in both source and packaged builds.

Changes include:

Updated few outdated tree-sitter queries
Ensured that all the CI tests pass
Fixed few import errors
Added Python test scripts
Added PyInstaller packaging support so the Tree-sitter signature JSON files are included in bundled builds.

Checklist

CHANGELOG updated

Few tests added

No documentation update needed currently.

This submission includes AI-generated code and I have provided details in the description.

Parts of this implementation were assisted using AI tools (Codex and ChatGPT).

AI was used for:

writing the test script for python
fixing few errors

All code was reviewed, modified and tested manually before submission.

Testing:
I built it using PyInstaller and tested the capabilities on script inputs. It functions as intended.

gemini-code-assist

Code Review

This pull request introduces Tree-Sitter Script Analysis to capa, enabling feature extraction from script languages such as C#, Python, HTML, and ASPX templates. It adds a new Tree-Sitter-based feature extractor, auto-detection capabilities, and signature-based tools, along with comprehensive tests and updated dependencies. The code review feedback primarily addresses compatibility issues with the upgraded tree-sitter library (version 0.25.0), specifically pointing out that QueryCursor has been removed and the Parser instantiation has changed in tree-sitter versions >= 0.21.0. The feedback provides actionable suggestions to execute queries directly and update parser usage. Additionally, it identifies a bug in integer suffix parsing and recommends replacing a deprecated importlib.resources API.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

saniyafatima07 · 2026-06-23T18:44:44Z

@mike-hunhoff @larchchen @Maijin The first version of the script analysis feature is ready for review.
Could you please take a look when you have a chance?

mr-tz

Initial review of the first few files

mr-tz · 2026-06-24T17:22:21Z

+
+def get_language_from_ext(path: str) -> str:
+    if path.endswith(EXT_ASPX):
+        return LANG_TEM


Embedded template is not immediately clear to me, some doc on that would be good (maybe it is further down).

I haven’t added documentation yet, but I’ll add a section describing the script analysis feature in the near future.

mr-tz · 2026-06-24T17:23:17Z

+            buf = f.read()
+        return get_language_ts(buf)
+    except ValueError:
+        return get_language_from_ext(str(path))


I guess the order is debatable here, so flagging for further discussion

My thinking here was to prefer Tree-sitter’s language detection when possible, since it can inspect the file contents rather than relying only on the extension. The extension fallback is mainly for cases where content-based detection fails.

That said, I agree that the order is debatable. Happy to discuss more on this!

What's the performance difference if we switch the order? My concern is that Tree-sitter's language detection could be slow for large or heavily obfuscated scripts, while simply checking the extension won't be bogged down by script size or complexity.

I ran a small benchmark test with simple Python scripts of different file sizes. There is actually a huge difference in the time taken by content-based detection compared to extension-based detection.

Extension-based: 0.0005 ms

Content-based: 7.3845 ms

So, it would be optimal for us to have extension-based detection first. Thank you for pointing it out @mr-tz and @mike-hunhoff.

I will correct it in the next commit.

Maijin · 2026-06-25T08:22:00Z

Kudo for making this huge pr work 👍, I have two high-level comments mainly:

Since automated detection may struggle with heavily obfuscated scripts, it might be beneficial to allow users to force specific language detection on the CLI. This would not only improve user experience but also serve as a useful tool for regression testing. I suggest considering this for the next PR.
Regarding the cs.json, py.json etc. signature files, what is the intended process for updating these as new malicious script patterns emerge? Since these are separated from capa-rules, we should document how rule authors can contribute new signatures. It may also be worth having an architectural discussion regarding the long-term management of these files tagging @mike-hunhoff for visibility on this.

Maijin · 2026-06-25T08:23:01Z

@@ -0,0 +1,95 @@
+{
+    "classes" : [
+        "System.Data.SqlClient.SqlCommand",


Would recommend sorting all of these signatures everywhere. This will ease contributions.

Maijin · 2026-06-25T08:23:13Z

@@ -0,0 +1,47 @@
+{
+    "classes": [
+        "socket.socket",


Same here - sort everywhere

saniyafatima07 · 2026-06-26T06:44:45Z

Kudo for making this huge pr work 👍, I have two high-level comments mainly:

* Since automated detection may struggle with heavily obfuscated scripts, it might be beneficial to allow users to force specific language detection on the CLI. This would not only improve user experience but also serve as a useful tool for regression testing. I suggest considering this for the next PR.

* Regarding the `cs.json`, `py.json` etc. signature files, what is the intended process for updating these as new malicious script patterns emerge? Since these are separated from `capa-rules`, we should document how rule authors can contribute new signatures. It may also be worth having an architectural discussion regarding the long-term management of these files tagging @mike-hunhoff for visibility on this.

Sure, Maxime. We can discuss about the architecture in the upcoming meets.

saniyafatima07 · 2026-07-21T21:00:26Z

Thank you @saniyafatima07 , I know this is a large PR to work with, so thank you for all you've done so far. I've left comments for your review. Mostly it appears that the PR still contains unrelated changes, likely from some rebase conflicts?

Thank you for the review, Mike.

I looked into all the comments. It seems like I messed up a few things during the rebase, which caused a lot of unrelated changes.

I will carefully look into all the unrelated changes and resolve them.

… sigs)

…heels

Revives and supersedes old PR mandiant#1080. Resolves merge conflicts and brings up to date with current master.

saniyafatima07 · 2026-07-30T20:27:03Z

@mike-hunhoff I have addressed all the comments and this pr is ready for another review.
Could you please review?

gemini-code-assist Bot reviewed Jun 10, 2026

View reviewed changes

saniyafatima07 marked this pull request as ready for review June 10, 2026 18:58

saniyafatima07 marked this pull request as draft June 10, 2026 18:58

saniyafatima07 force-pushed the script-feature branch 2 times, most recently from e71ca37 to 5c19383 Compare June 17, 2026 16:41

mr-tz reviewed Jun 18, 2026

View reviewed changes

Comment thread capa/capabilities/common.py

saniyafatima07 force-pushed the script-feature branch 12 times, most recently from 19308f7 to 291975a Compare June 22, 2026 22:41

saniyafatima07 marked this pull request as ready for review June 23, 2026 14:15

saniyafatima07 requested review from Maijin, larchchen and mike-hunhoff June 23, 2026 14:16

mr-tz reviewed Jun 24, 2026

View reviewed changes

Maijin reviewed Jun 25, 2026

View reviewed changes

mr-tz reviewed Jun 27, 2026

View reviewed changes

Comment thread capa/helpers.py Outdated

mr-tz reviewed Jun 27, 2026

View reviewed changes

Comment thread capa/helpers.py Outdated

saniyafatima07 force-pushed the script-feature branch from 0582aff to d12e8d5 Compare July 22, 2026 19:48

saniyafatima07 and others added 23 commits July 30, 2026 22:20

Fix issues

8f30c60

fix: address PR review feedback (regex optimization, typos, duplicate…

43aaf0d

… sigs)

Update CHANGELOG.md

3b956d6

Fix outdated code

bbdb885

Fix outdated code

a103a63

Modernize Tree-sitter: Replace local C-compilation with native PyPI w…

f2bfdd1

…heels

feat: revive script analysis ts feature

303f730

Revives and supersedes old PR mandiant#1080. Resolves merge conflicts and brings up to date with current master.

Fix: CI builds

83c595b

Fix import errors

ab8edd7

Fix CI: trial-1

596d194

Fix tree-sitter packaging

c9ead9a

Address valid gemini reviews

f392f72

Minor fixes

423597d

Address comments

f591b87

Sort json files

766c287

Remove few unnecessary changes

9a35abc

Computer hashes for scripts

cb5f9c9

Cleanup: Revert the unrelated changes

3f485af

Cleanup2: Revert the unrelated changes

65c7c03

Cleanup3: Revert unrelated changes

4a3d948

Cleanup4: Resolve unrelated changes

5b5ff51

Fixture: restore, add tree-sitter

838a271

Update changelog

d76cd1b

saniyafatima07 force-pushed the script-feature branch from 3e330be to d76cd1b Compare July 30, 2026 16:57

saniyafatima07 added 2 commits July 30, 2026 23:04

Update tests/data submodule

cc5135b

Restore rules submodule pointer

42d6e94

Add test helpers for feature extraction

19cbb25

Uh oh!

Conversation

saniyafatima07 commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

saniyafatima07 commented Jun 23, 2026

Uh oh!

mr-tz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Maijin commented Jun 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saniyafatima07 commented Jun 26, 2026

Uh oh!

Uh oh!

Uh oh!

saniyafatima07 commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saniyafatima07 commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

saniyafatima07 commented Jun 10, 2026 •

edited

Loading

saniyafatima07 commented Jul 21, 2026 •

edited

Loading

saniyafatima07 commented Jul 30, 2026 •

edited

Loading