Skip to content

feat: add PROMPT subject support for NGT (Agentforce Studio) test creation @W-23524165@ - #353

Open
niharpatel-salesforce wants to merge 4 commits into
forcedotcom:mainfrom
niharpatel-salesforce:feat/w-23524165-prompt-subject-test-support
Open

feat: add PROMPT subject support for NGT (Agentforce Studio) test creation @W-23524165@#353
niharpatel-salesforce wants to merge 4 commits into
forcedotcom:mainfrom
niharpatel-salesforce:feat/w-23524165-prompt-subject-test-support

Conversation

@niharpatel-salesforce

@niharpatel-salesforce niharpatel-salesforce commented Aug 26, 2026

Copy link
Copy Markdown

What does this PR do?

Demo video link

Adds client-side support for the PROMPT subject type (prompt template testing) to the NGT (Agentforce Studio) sf agent test create code path, alongside the existing AGENT subject type. Covers the full round trip:

  • Types (src/types.ts, src/index.ts): new AiPromptTestCase, AiTestCasePromptInputXml, NgtPromptInputSet, NgtPromptTestCase, discriminated by subjectType: 'AGENT' | 'PROMPT'.
  • YAML → metadata XML (convertToTestingMetadata + new toPromptInputsXml): serializes a PROMPT spec's promptInput slots into AiTestingDefinition XML.
  • Metadata XML → spec (convertToNgtSpec + new parsePromptTestCaseXml; collapseContiguousTestCases genericized): parses a deployed PROMPT AiTestingDefinition back into an NgtTestSpec, including multi-invocation fan-out/collapse.
  • Validation (validateNgtSpec + new validatePromptTestCases, shared validateScorers extracted): PROMPT specs get structural + promptInput content checks (non-blank referenceName/value), and correctly skip the AGENT-only multi-agent-handoff / task_resolution+conversationHistory / conversationHistory-index rules.
  • Scorer validation (src/ngtScorerCatalog.ts, validateScorers): each catalog scorer now declares which subject types it supports. AGENT keeps all 11 existing scorers; PROMPT supports 5 (coherence, conciseness, factuality, completeness, response_match). Using any other catalog scorer on a PROMPT spec now emits a Lifecycle warning instead of passing through silently — the deploy still proceeds and is validated server-side. AGENT behavior is unchanged.
  • Wiring (AgentTest.create()): skips the BotDefinition.IsMultiAgent org lookup for PROMPT subjects, since multi-agent handoff is structurally an AGENT-only concept.
  • Reorganized the subject-type dispatch in convertToTestingMetadata, convertToNgtSpec, and validateNgtSpec into small per-subject handler functions routed through one shared helper. The default path is the existing AGENT path; the PROMPT path is only taken when subjectType is exactly PROMPT. No behavior change for the AGENT path.
  • 3 new user-facing message keys in messages/agentTest.md, 2 new fixtures (test/fixtures/ngt-prompt-*), and unit tests for every new code path.

Example PROMPT spec:

name: SummarizeCasePromptSuite
description: Validates the SummarizeCase prompt template.
subjectType: PROMPT
subjectName: SummarizeCasePrompt
subjectVersion: v3
testCases:
  - inputs:
      - promptInput:
          - referenceName: CaseDescription
            value: Customer reports login failures since yesterday.
          - referenceName: CustomerTone
            value: frustrated
    scorers:
      - name: factuality
      - name: completeness
      - name: response_match
        expected: A concise, accurate case summary
sf agent test create --spec ./summarize-case-spec.yaml --test-runner agentforce-studio --api-name SummarizeCasePromptSuite --target-org myOrgAlias

Testing:

  • Unit: yarn test — 435 passing, 1 pending, 0 failing, coverage thresholds met.
  • Manually verified end-to-end against a live org (sf agent test create --test-runner agentforce-studio, linked build of this branch into plugin-agent) across 16 scenarios spanning both subject types — validation failures, server-side rejections, and successful deploys with XML inspection:
# Scenario Result As expected?
1 AGENT subject: full 11-scorer catalog on a single test case Client-side ngtTaskResolutionRequiresConversationHistory (fixture's task_resolution scorer needs conversationHistory) Yes
2 AGENT subject: baseline smoke test Deploy failed: BotVersion not found (org-state — bot has no deployed version) Yes
3 AGENT subject: baseline smoke test (second bot) Deploy failed: BotVersion not found (org-state) Yes
4 PROMPT subject: baseline multi-invocation spec Deployed successfully Yes
5 PROMPT subject: baseline multi-invocation spec (variant) Deployed successfully Yes
6 PROMPT subject: AGENT-only scorer (topic_sequence_match) Client warning ("not supported for subject type 'PROMPT'"); deploy also rejected server-side Yes
7 PROMPT subject: blank referenceName Client-side ngtPromptInputMissingReferenceName Yes
8 PROMPT subject: blank value Client-side ngtPromptInputMissingValue Yes
9 PROMPT subject: empty promptInput: list Client-side ngtPromptInputSetEmpty Yes
10 Misspelled subjectType: PROMT Deploy failed: BotVersion not found (took the AGENT path) Yes — the default path is AGENT; the PROMPT path is only taken when subjectType is exactly PROMPT
11 PROMPT subject: scorer missing expected: Client-side ngtScorerMissingExpected Yes
12 PROMPT subject: multi-invocation fan-out Deployed successfully Yes
13 PROMPT subject: non-adjacent matching scorer sets Deployed successfully (no false collapse) Yes
14 PROMPT subject: unquoted numeric value Deployed successfully Yes
15 PROMPT subject: quality scorer (coherence) with expected: supplied anyway XML omits <expectedValue>; deployed successfully Yes
16 PROMPT subject: scorer unknown to the local catalog Client warning only; deploy rejected server-side with the org's full scorer allow-list Yes

What issues does this PR fix or reference?

@W-23524165@

…tch @W-23524165@

- NgtScorerCatalog entries now declare supportedSubjects; PROMPT specs
  using an AGENT-only scorer emit a Lifecycle warning instead of
  passing through silently. AGENT is unaffected (all 11 scorers keep
  supportedSubjects: ['AGENT']).
- Replaced the per-function subjectType if/else in
  convertToTestingMetadata, convertToNgtSpec, and validateNgtSpec with
  small per-subject handler functions routed through one
  dispatchBySubjectType helper. No behavior change for AGENT.
…ySubjectType @W-23524165@

dispatchBySubjectType checked subjectType === 'AGENT' and fell through
to PROMPT otherwise, inverting the original fallback direction (the
prior if/else checked === 'PROMPT' and fell through to AGENT). A
malformed subjectType (e.g. a "PROMT" typo) now coerces to AGENT again,
matching pre-refactor behavior. Caught via live-org testing: the
malformed-subjectType corner case unexpectedly deployed successfully
as PROMPT instead of failing on the AGENT BotDefinition lookup.
@niharpatel-salesforce
niharpatel-salesforce marked this pull request as ready for review August 26, 2026 09:54
@niharpatel-salesforce
niharpatel-salesforce requested a review from a team as a code owner August 26, 2026 09:54
@niharpatel-salesforce
niharpatel-salesforce marked this pull request as draft August 26, 2026 10:09
@niharpatel-salesforce
niharpatel-salesforce marked this pull request as ready for review August 26, 2026 10:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant