Proposal: Arabic & Multilingual Security Contributions to PyRIT

[AbdaullahAG]

Hi PyRIT team 👋

I recently published SemGuard, a research project focused on
detecting prompt attacks across Arabic, Arabizi, and English —
accepted at IEEE AEECT 2026.

As part of it, I built a validated Arabic prompt attack dataset
(807 samples) covering injection/jailbreak, phishing, privacy
leakage, Unicode-based attacks, and safe educational prompts.

Given PyRIT's red-teaming focus, would there be interest in an
Arabic dataset loader or Arabic-focused attack scenarios? Happy
to share more details (including a semantic evaluation approach
we used) if useful — just don't want to front-load everything
before knowing what's actually relevant to the roadmap.

Thanks!

[romanlutz]

Abdallah Abughallous (@AbdaullahAG) thanks for the proposal! I do think it's potentially relevant. We have similar datasets in our datasets module. We typically exclude safe prompts unless they have a specific purpose, e.g., showing overrefusal.

Is the dataset available somewhere to look at?

[AbdaullahAG]

Roman Lutz (@romanlutz) Thanks for the response!

Here's the dataset:
https://huggingface.co/datasets/AG-31625874/SemGuard-Dataset

And the repo (code, preprocessing utilities, etc.):
https://github.com/AbdaullahAG/SemGuard

Good point on safe prompts — in our case they weren't meant as
a standalone benchmark item, but as reference anchors for a
semantic classifier (comparing input intent against attack/safe/
destructive sets rather than binary classification). Happy to
strip them out or restructure if that's a better fit for a PyRIT
dataset loader — whatever aligns best with your existing module's
conventions.

[AbdaullahAG]

Hi Roman Lutz (@romanlutz), just checking in on this , happy to move forward with restructuring the dataset per your loader conventions once you've had a chance to take a look, or let me know if you need anything else from my end to evaluate fit.

[romanlutz]

I think it's a good idea to add. We will probably have some kind of enum to decide which partition/category of the data to load.

My only hesitation: It might be hard to use for people who don't speak the languages as they won't be able to read the prompts.

Feel free to create a PR and we will have someone review.

[AbdaullahAG]

Hi Roman Lutz (@romanlutz) , Draft PR is up: #2468

Went with SemGuardCategory as a filterable enum (SAFE included
as an explicit opt-in value rather than default-loaded, per your
point). Added a language field to each seed's metadata for the
accessibility concern — flagged a couple of implementation
questions directly in the PR description (harm_categories mapping
for prompt injection specifically, since there's no direct
taxonomy entry for it as an attack technique).

All 9 unit tests passing, notebook regenerated and executes clean.
Happy to adjust based on review.