FEAT: Scorer Backend API

Is your feature request related to a problem? Please describe.

The dataset explorer's custom-scenario flow needs configured scorer discovery instead of a hardcoded list of scorer classes. PyRIT already has ScorerRegistry and initializer-provided instances, but these needed a general backend API before the GUI and scenario consumers could build on them.

Work item 12 of 17 in #2744. Completed by #2990, merged on October 5, 2026. This was the independent scorer API track; it did not depend on dataset browsing or provider loading.

This issue originally described objective-scorer presets. Rich's October 4 direction changed the first deliverable to the general registry backend, with objective selection and the GUI left as follow-ups. The body now reflects that implemented scope. The overlapping general-API issue #2970 is satisfied by the same PR, not a second implementation.

Describe the solution you'd like

Completed backend scope

#2990 adds thin routes and a service over the existing ScorerRegistry:

Route	Purpose
`GET /api/scorers/types`	Discover scorer types and shared constructor `Parameter` metadata without constructing them.
`GET /api/scorers`	List named, configured instances with bounded pagination.
`GET /api/scorers/{scorer_registry_name}`	Retrieve one configured instance by registry name.
`POST /api/scorers`	Create a named instance from `name`, `type`, and `params` through the registry and shared resolver.

The shared ScorerInstance response contains scorer_registry_name, a full typed ScorerIdentifier, and description. Per the final maintainer decision, identifiers include nested child scorers and target identifiers. This supersedes the earlier flat-summary proposal, but it does not permit credential disclosure. A full identifier does not itself expose registry tags or prove objective suitability.

Type discovery is separate from configured-instance discovery. Read endpoints do not construct scorers, score content, call targets, or load datasets. Registry construction and reference resolution remain in the core registry/resolver; the backend validates HTTP input and translates errors. Failed construction does not register a partial instance, and duplicate-name creation must not replace an existing one.

Acceptance criteria completed in #2990:

Discoverable types and configured instances have distinct typed API responses backed by the existing registry.
List, detail, and create use the same configured-instance model, including full nested identifiers.
Explicit named creation uses the shared resolver; failed construction and duplicate-name handling are covered.
Read endpoints perform no scoring, model calls, or provider loading.
Focused route/service coverage includes nested identifiers, credential exclusion, pagination, registry lifecycle changes, and concurrent duplicate creation using local fakes.
No new scorer registry, persistent scorer store, memory migration, or execution engine is introduced.

Remaining dataset-scenario requirements

The original dataset-specific criteria are retained in these open work items, not claimed as implemented by #2990:

FEAT GUI: Accept exact seed selections in scenario requests #2757: carry the selected configured scorer's registry name alongside exact seed selections; preserve one effective configuration across preview, estimate, and launch; reject missing/stale/incompatible references without silent substitution.
FEAT: Support custom scenarios over selected dataset examples #2760: reuse shared objective-compatibility and default-selection policy, including default_objective_scorer, configured composites/inverters, prerequisite handling, and correct objective-result meaning. Raw refusal=true is not attack success; raise_if_scorer_blocks governs blocked judging, not refusal polarity.
FEAT GUI: Configure a scenario from selected dataset examples #2759: build the configured-instance picker and its shared typed metadata projection for tags, compatible defaults, compatibility/prerequisites, and setup guidance. Those fields are not supplied merely by returning a full identifier.

#2757 remains gated on #2756 and maintainer promotion. Completing this parallel API track does not release the remaining explorer work out of order.

Describe alternatives you've considered, if relevant

#2834 attempted a specialized objective-presets endpoint and closed without merging. The agreed path was the general registry API first, then objective-selection and GUI consumers. Do not reimplement that backend, hardcode a frontend scorer list, create a second registry, or add an unscored scenario mode. A general scorer-builder GUI and arbitrary new multi-scorer composition language are outside this dataset series; an existing configured scorer may already be a composite.

Additional context

Implementation: #2990, merge commit 5f619c0b87e342ba15b5dd0af29df4858ecce5e9. The contributor also explicitly recorded the backend/follow-up boundary in this comment.

Starting points for the remaining consumers are pyrit\backend\models\scorers.py, pyrit\backend\services\scorer_service.py, pyrit\models\catalog\scorer.py, ScorerRegistry, ScorerInitializer, and Scenario._get_default_objective_scorer(). The existing /scores routes concern score operations, not this configured-scorer catalog. Follow doc\code\framework.md and applicable model/Python/scorer/scenario/test instructions.

I would like to take this.

As described in #2834 we should follow something closer to #2834 (review)

Thanks for pointing me to the #2834 review; I read it. I understand the objective-preset endpoint in #2755 would be premature until the named Scorer API and scenario-run selection share one contract. I’ll pause that specialized endpoint and would like to work on the general Scorer API first.

My starting shape is:

GET /api/scorers/types: registered type names and constructor-parameter schemas.
GET /api/scorers: paginated safe summaries of configured instances.
GET /api/scorers/{name}: one safe instance summary.
POST /api/scorers: create through ScorerRegistry from a registered type and parameters, returning the same summary shape.

For summaries, I propose only a stable public name, scorer type, and safe display description—no prompts, nested target configuration, credentials, or raw scorer identifiers. Objective-scorer selection would remain out of scope until the run API consumes the same names.

Does that contract match what you have in mind, especially the summary fields and create request, or would you prefer a narrower first slice? I’ll hold implementation until we agree on the response shape.

Richard Lundeen (@richlundeen)

To be honest, I'm not sure if Richard Lundeen (@richlundeen) is taking this up as part of his scoring refactoring work so I'd hold off on that for a minute.

Thanks for the feedback! A maintainer will take a look soon.

Chen Yufeiyang (@feiiiiii5)

This looks good! Thank you for tackling! Assigning to you Chen Yufeiyang (@feiiiiii5)

No one on our team is looking at this for at least a couple weeks, so let us know if your timelines change (and I'll comment here at least a couple days in advance if this ever ends up in our critical path and we need it/to take it).

I agree with the routes and use of the existing registry. I also like that you use the shared Parameter metadata and resolver. Two pieces of feedback:

I think we should return the full ScorerIdentifier, including child scorers and target identifiers. This matches targets and converters and lets clients inspect the complete configured scorer. A separate summary model would duplicate information and add maintenance work. I understand the concern about exposing configuration, but I think the benefits outweigh the costs here. Credentials must remain excluded from identifiers - but the target APIs should already take care of this so I don't think anything new here.
I would not add scorer-specific parameter or compatibility validation in the backend. The shared registry resolver and scorer classes should own those checks. The backend should validate the HTTP request and translate errors into API responses. If resolution or construction fails, the instance must not be registered, and the error should be returned to the caller.

I'm also updating the title. I think the backend is a good solo-PR, and as a followup you could have a separate issue (or just continue this one) around adding a "Scorer" tab next to converter/targets in the GUI registry

Taking this. I will start from scorer_registry.py, the scorer initializers and Scenario._get_default_objective_scorer(), and keep the registry responsible for the catalog rather than pushing it into service DTOs.

Two things I want to flag before I write code, so we do not go back and forth later:

not ready yet is still on this issue. I saw your note that nobody on the team will be looking at it for a couple of weeks. That is not a reason for me to hold off — you assigned it and the scope is clear enough to build against — but if not ready yet means something specific (a design review still pending, or a dependency on the explorer sequence in #2748), tell me and I will park it. The issue says the API track is independent of the explorer sequence, which is what I am assuming.

The one acceptance criterion I will need to make a judgment call on is "incompatible result families". A scorer's result family is not something the registry currently records, so deriving it means reading each scorer's declared output type. I will do that from the existing declarations rather than adding a new registry field, unless you would rather the family be declared explicitly on the scorer. If I guess wrong here the API shape is wrong, so I would rather ask now than rework it.

I will come back with a PR that satisfies the five criteria as written — fake registered scorers in the tests, no paid services, no credentials in the response, and listing that performs no scoring or model calls. I will keep the diff to this work item; no scorer builder, no new registry, no changes to the existing /scores routes.

Before I open a PR I need to flag a conflict with my own earlier attempt, because building it anyway would be repeating something you already asked me to close.

The history: on 2026-09-25 I opened #2834 for this same work — an objective-scorer preset endpoint. You closed it with:

We have a scorer registry, but we have not settled the backend Scorer API contract or how scenario runs will select scorers. A separate objective-preset endpoint could give clients a shape that we need to replace soon.

and pointed me at the general scorer endpoints instead (GET /api/scorers/types, GET /api/scorers, GET /api/scorers/{name}, POST /api/scorers), with objective-scorer selection to come after those, "once the run API can resolve and use those same registry names".

The prerequisite is still unmet on main today (2596f34):

pyrit/backend/routes/targets.py        present  <- the API to mirror
pyrit/backend/services/target_service.py present
pyrit/backend/routes/scores.py        present  <- score operations, not a scorer catalog
pyrit/backend/routes/scorers.py       absent

There is no routes/scorers.py, and no open PR adds one. So the sequencing you set on 09-25 still applies — which puts the work you assigned me on 10-04 behind a step that has not happened.

What I have ready, so nothing is lost either way. Branch feat/scorer-backend-api at 19cf37d7:

pyrit/models/catalog/scorer.py — three wire models in the same shape as catalog/initializer.py and catalog/target.py, no registry knowledge.
pyrit/registry/components/scorer_catalog.py — a projection over ScorerRegistry.instances that never reads the class catalog, never scores and never calls a target.
17 tests with fake registered scorers, no paid services. ruff check/format clean.

What the branch reports

It reports, per configured preset: stable registry name, description, result family, tags, whether it carries default_objective_scorer, whether it needs a chat target, and raise_if_scorer_blocks as the refusal polarity. Only true_false presets are offered — the same constraint _get_default_objective_scorer() already enforces — and anything else is returned under incompatible with the reason.

Note that this is deliberately not an endpoint: it is the catalog and the projection underneath one, so it can back whichever route shape you settle on rather than freezing one.

Two ways forward, and I will follow whichever you say:

If the general scorer endpoints should come first, say so and I will drop this branch and start on GET /api/scorers/types instead — the small one you suggested — and we agree the response shape before I write it, as you offered on 09-25.
If the 09-25 sequencing no longer applies and you want objective-scorer selection now, I will open the branch as it stands and treat default_objective_scorer as the field to rename whenever the general endpoint lands.

I did not want to guess, because a field name or a route frozen now is exactly the cost you pointed at. The two questions I asked earlier (whether not ready yet blocks anything, and whether "refusal polarity" means raise_if_scorer_blocks) are still open and both are cheap to answer alongside this one.

You mentioned nobody is on this for a couple of weeks, so there is no rush — I would rather get the sequencing right than land the wrong shape early.

Richard Lundeen (@richlundeen)

Thanks Richard Lundeen (@richlundeen) — your October 4 review settles the general scorer API contract. I am implementing the four registry routes with the full ScorerIdentifier, including child scorers and target identifiers, and using the shared resolver for construction and validation. The backend will handle request validation and HTTP error translation; failed construction will leave the registry unchanged. Objective-scorer selection and the GUI remain follow-ups.

Implemented in #2990: the four general registry routes return the full ScorerIdentifier, use the shared resolver, and translate failures to HTTP errors without registering failed constructions. The 273 targeted unit tests passed, including nested identifiers, target references, and concurrent duplicate names. Objective presets and the GUI remain follow-ups.

Chen Yufeiyang (@feiiiiii5)

Closing as completed by #2990. I updated the body to match the agreed general scorer backend scope and full-identifier contract, rather than marking the original objective-preset requirements as shipped.

The remaining requirements are now explicit in #2757 (selected scorer references and consistent preview/estimate/launch), #2760 (compatible defaults and objective-result semantics), and #2759 (picker metadata and setup guidance). #2757 still waits for #2756, so this does not release the downstream work yet. Thanks Chen Yufeiyang (@feiiiiii5)!

Navigation Menu

Repository navigation

FEAT: Scorer Backend API #2755

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Completed backend scope

Remaining dataset-scenario requirements

Describe alternatives you've considered, if relevant

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

FEAT: Scorer Backend API #2755

Description

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Completed backend scope

Remaining dataset-scenario requirements

Describe alternatives you've considered, if relevant

Additional context

Activity

feiiiiii5 commented on Sep 25, 2026

romanlutz commented on Sep 26, 2026

feiiiiii5 commented on Sep 26, 2026

romanlutz commented on Sep 28, 2026

github-actions commented on Sep 28, 2026

richlundeen commented on Oct 4, 2026

feiiiiii5 commented on Oct 4, 2026

feiiiiii5 commented on Oct 4, 2026

feiiiiii5 commented on Oct 5, 2026

feiiiiii5 commented on Oct 5, 2026

romanlutz commented on Oct 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions