feat: add PROMPT subject support for NGT (Agentforce Studio) test creation @W-23524165@ - #353
Open
niharpatel-salesforce wants to merge 4 commits into
Conversation
…ation @W-23524165@
…tch @W-23524165@ - NgtScorerCatalog entries now declare supportedSubjects; PROMPT specs using an AGENT-only scorer emit a Lifecycle warning instead of passing through silently. AGENT is unaffected (all 11 scorers keep supportedSubjects: ['AGENT']). - Replaced the per-function subjectType if/else in convertToTestingMetadata, convertToNgtSpec, and validateNgtSpec with small per-subject handler functions routed through one dispatchBySubjectType helper. No behavior change for AGENT.
…ySubjectType @W-23524165@ dispatchBySubjectType checked subjectType === 'AGENT' and fell through to PROMPT otherwise, inverting the original fallback direction (the prior if/else checked === 'PROMPT' and fell through to AGENT). A malformed subjectType (e.g. a "PROMT" typo) now coerces to AGENT again, matching pre-refactor behavior. Caught via live-org testing: the malformed-subjectType corner case unexpectedly deployed successfully as PROMPT instead of failing on the AGENT BotDefinition lookup.
niharpatel-salesforce
marked this pull request as ready for review
August 26, 2026 09:54
niharpatel-salesforce
marked this pull request as draft
August 26, 2026 10:09
niharpatel-salesforce
marked this pull request as ready for review
August 26, 2026 10:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Demo video link
Adds client-side support for the PROMPT subject type (prompt template testing) to the NGT (Agentforce Studio)
sf agent test createcode path, alongside the existing AGENT subject type. Covers the full round trip:src/types.ts,src/index.ts): newAiPromptTestCase,AiTestCasePromptInputXml,NgtPromptInputSet,NgtPromptTestCase, discriminated bysubjectType: 'AGENT' | 'PROMPT'.convertToTestingMetadata+ newtoPromptInputsXml): serializes a PROMPT spec'spromptInputslots intoAiTestingDefinitionXML.convertToNgtSpec+ newparsePromptTestCaseXml;collapseContiguousTestCasesgenericized): parses a deployed PROMPTAiTestingDefinitionback into anNgtTestSpec, including multi-invocation fan-out/collapse.validateNgtSpec+ newvalidatePromptTestCases, sharedvalidateScorersextracted): PROMPT specs get structural +promptInputcontent checks (non-blankreferenceName/value), and correctly skip the AGENT-only multi-agent-handoff /task_resolution+conversationHistory / conversationHistory-index rules.src/ngtScorerCatalog.ts,validateScorers): each catalog scorer now declares which subject types it supports. AGENT keeps all 11 existing scorers; PROMPT supports 5 (coherence,conciseness,factuality,completeness,response_match). Using any other catalog scorer on a PROMPT spec now emits a Lifecycle warning instead of passing through silently — the deploy still proceeds and is validated server-side. AGENT behavior is unchanged.AgentTest.create()): skips theBotDefinition.IsMultiAgentorg lookup for PROMPT subjects, since multi-agent handoff is structurally an AGENT-only concept.convertToTestingMetadata,convertToNgtSpec, andvalidateNgtSpecinto small per-subject handler functions routed through one shared helper. The default path is the existing AGENT path; the PROMPT path is only taken whensubjectTypeis exactlyPROMPT. No behavior change for the AGENT path.messages/agentTest.md, 2 new fixtures (test/fixtures/ngt-prompt-*), and unit tests for every new code path.Example PROMPT spec:
sf agent test create --spec ./summarize-case-spec.yaml --test-runner agentforce-studio --api-name SummarizeCasePromptSuite --target-org myOrgAliasTesting:
yarn test— 435 passing, 1 pending, 0 failing, coverage thresholds met.sf agent test create --test-runner agentforce-studio, linked build of this branch intoplugin-agent) across 16 scenarios spanning both subject types — validation failures, server-side rejections, and successful deploys with XML inspection:ngtTaskResolutionRequiresConversationHistory(fixture'stask_resolutionscorer needs conversationHistory)BotVersion not found(org-state — bot has no deployed version)BotVersion not found(org-state)topic_sequence_match)referenceNamengtPromptInputMissingReferenceNamevaluengtPromptInputMissingValuepromptInput:listngtPromptInputSetEmptysubjectType: PROMTBotVersion not found(took the AGENT path)subjectTypeis exactlyPROMPTexpected:ngtScorerMissingExpectedcoherence) withexpected:supplied anyway<expectedValue>; deployed successfullyWhat issues does this PR fix or reference?
@W-23524165@