diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..58c7ff5e --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,92 @@ +# Changelog + +## v1.6.0 + +A major step-up for the local validator: it now mirrors what actually loads and runs in Provar, surfaces severity through validity, and ships a higher default quality bar — plus two new MCP resources that expose the validator's contract to AI clients. + +### Highlights + +- **The local validator now mirrors Provar's own load/runtime behaviour.** Dozens of structural and best-practice checks that previously lived only in the Quality Hub backend now run locally, so `provar_testcase_validate` catches load-blocking and runtime defects with no API key required. +- **Validity now reflects severity.** A `critical` best-practice violation (e.g. a hallucinated `apiId`, a non-integer `testItemId`) now gates `is_valid` instead of quietly passing — an AI agent can trust `is_valid: true` again. +- **A single, tri-state verdict.** Every validate tool returns `status: valid | needs_improvement | invalid`, alongside the effective `quality_threshold` and `meets_quality_threshold`, so agents have one unambiguous signal to gate on. +- **A higher default quality bar.** The default quality threshold is raised **80 → 90**, tunable per call (`quality_threshold`) or globally (`PROVAR_MCP_QUALITY_THRESHOLD`). +- **Two new MCP resources** expose the validator's contract: the structured Provar test-step schema and a canonical Validation Rule Registry. + +### New + +- **MCP resource `provar://schema/test-step`** — the structured JSON contract for the Provar test-case XML (root, generic `apiCall`, every step type with required/optional args and value classes). +- **MCP resource `provar://docs/validation-rules`** — the canonical registry of every validation rule across both layers (id, severity, weight, what it checks, and whether it gates `is_valid`). +- **`status`, `quality_threshold`, `meets_quality_threshold`** output fields on `provar_testcase_validate` and the suite/plan/project validate tools. +- **`quality_threshold` input** plus the **`PROVAR_MCP_QUALITY_THRESHOLD`** env var (precedence: per-call arg → env → 90). +- **Context-aware `comparisonType` validation** — the valid comparison set is scoped by step type (AssertValues, UI Assert, …) instead of one flat list. +- **Project-aware `test_case_id` allocation** — generated test cases take the next id in the surrounding Provar project rather than a hard-coded `id="1"`; the chosen id is surfaced on the response. +- **`PROVAR_PLUGIN_NOT_FOUND`** error code when the Provar Automation plugin is missing. + +### Changed + +- **Validity bridge:** critical best-practice violations are surfaced as `is_valid`-gating issues, deduplicated against the Layer-1 rule that already owns the same concept. +- Severity alignment with Quality Hub: `UI-BINDING-ORDER-001` and `VAR-NAMING-001` reclassified critical → major (they hard-fail at runtime but do not block loading). +- Test-case generator fidelity: `UiDoAction` is serialised as `uiInteraction`, and `UiAssert` field assertions are nested for correct Provar IDE rendering. + +### Fixed + +- Best-practices engine no longer crashes on numeric tag values. +- `RENDER-CASE-001` scoped to the six real `valueClass` values, removing false positives. +- `TC_010` accepts any integer test-case id and treats id as optional (the `guid` is the real identifier). +- Windows: the `sf` executable and its arguments are quoted so project paths containing spaces work. + +### Upgrade notes + +- **Mostly non-breaking.** New inputs, env vars, and output fields are additive. +- **Behaviour change to note:** with the validity bridge and the higher default threshold (90), a test case that previously returned `is_valid: true` may now report `status: "needs_improvement"` (score below 90) or `"invalid"` (a critical violation now gates validity). Set `quality_threshold` per call or `PROVAR_MCP_QUALITY_THRESHOLD` globally to restore the previous 80 bar if needed. + +## v1.5.1 + +### Highlights + +- **Smaller, faster MCP handshake** — opt into compact tool schemas and load only the tool groups you need. **~36% fewer handshake tokens with compact mode alone, up to ~57% when combined with group filtering.** +- **Smarter validation loops** — agents get tunable response detail, run-over-run diffs, and a single completeness signal that's safe to gate on. +- **Single-call test authoring** — test-case generation is now a true one-shot construction, with a runtime guard so agents stop iterating in the wrong direction. +- **Reliable connection + environment resolution** in `.testproject` files. + +### Tool-catalog footprint + +Tokens sent to the LLM on `tools/list` (≈4 chars/token): + +| Configuration | Tools | ~Tokens | Savings vs default | +| ------------------------------------------ | ----: | ------: | -----------------: | +| Standard (all groups, full descriptions) | 41 | 18,355 | — | +| Compact (all groups, compact descriptions) | 41 | 11,758 | **−36%** | +| Authoring profile (compact + 4 groups) | 21 | 7,906 | **−57%** | + +Per-tool savings are largest where they matter most — `testcase_generate` alone drops from ~2,070 tokens of description to a fraction of that in compact mode. + +### New + +- Compact schema mode and tool-group filtering for trimmed startup payloads. +- `detail`, `baseline_run_id`, `run_id`, and `completeness_score` on validation tools. +- `fields` parameter on inspect / list tools to scope responses to only what's needed. +- Depth guard and token-attribution middleware across all tools. +- Construct-vs-amend contract carried into test-case tool titles, descriptions, and a runtime check on empty steps. + +### Guidance & prompt improvements + +- Test-case authoring rewritten as a **single-call construction** contract — agents now produce a complete test case in one call instead of looping through construct → amend → re-amend cycles. End-to-end authoring of a multi-step Salesforce flow drops from typically **3–5 tool calls to 1**. +- Construct-vs-amend semantics surfaced at three layers (tool title, description, runtime guard) so agents that skim only the title still get the contract. +- Validation tools now return a single `completeness_score` (0–100) so agents have one number to gate on, instead of inferring stop/continue from violation arrays. +- Compact tool descriptions are tuned to keep the _contract_ (when to call, prerequisites, common failure modes) while dropping prose — the signal agents actually use stays intact. + +### Fixed + +- Validation stop decisions now account for all violation levels (plan metadata, suite, best-practices) instead of stopping while issues remain. +- Read-only validation diffs work without writing new results. +- Validation baselines are now scoped to their original project context, so a baseline from one project can't silently diff against another. +- Unknown tool-group names now warn instead of silently disabling everything. +- Release builds now reliably fetch the latest NitroX schemas instead of falling back to a bundled copy. +- Connection + environment resolution in `.testproject` files. +- Various agent-loop and review-pass hardening for the test-case authoring path. + +### Upgrade notes + +- **Non-breaking.** All new parameters and env vars are opt-in. +- Existing callers see no behavior change. diff --git a/README.md b/README.md index d6012540..43469cb9 100644 --- a/README.md +++ b/README.md @@ -100,7 +100,7 @@ sf provar auth login claude mcp add provar -s user -- sf provar mcp start --allowed-paths /path/to/your/provar/project ``` -📖 **[docs/mcp.md](https://github.com/ProvarTesting/provardx-cli/blob/main/docs/mcp.md) — full setup, all 35+ tools, 11 MCP prompts, troubleshooting.** +📖 **[docs/mcp.md](https://github.com/ProvarTesting/provardx-cli/blob/main/docs/mcp.md) — full setup, all 42 tools, 6 resources, 11 MCP prompts, troubleshooting.** --- @@ -251,42 +251,10 @@ DESCRIPTION Note: --json is not available on this command — stdout is reserved for MCP traffic. TOOLS EXPOSED - provar.project.inspect — inspect project folder inventory - provar.pageobject.generate — generate Java Page Object skeleton - provar.pageobject.validate — validate Page Object quality (30+ rules) - provar.testcase.generate — generate XML test case skeleton - provar.testcase.validate — validate test case XML (validity + best-practices scores) - provar.testsuite.validate — validate test suite hierarchy - provar.testplan.validate — validate test plan with metadata completeness checks - provar.project.validate — validate full project: cross-cutting rules, connections, environments - provar.properties.generate — generate provardx-properties.json from the standard template - provar.properties.read — read and parse a provardx-properties.json file - provar.properties.set — update fields in a provardx-properties.json file - provar.properties.validate — validate a provardx-properties.json file against the schema - provar.ant.generate — generate an ANT build.xml for CI/CD pipeline execution - provar.ant.validate — validate an ANT build.xml for structural correctness - provar.qualityhub.connect — connect to a Quality Hub org - provar.qualityhub.display — display connected Quality Hub org info - provar.qualityhub.testrun — trigger a Quality Hub test run - provar.qualityhub.testrun.report — poll test run status - provar.qualityhub.testrun.abort — abort an in-progress test run - provar.qualityhub.testcase.retrieve — retrieve test cases by user story / component - provar.automation.setup — detect or download/install Provar Automation binaries - provar.automation.testrun — trigger a Provar Automation test run (LOCAL) - provar.automation.compile — compile Page Objects after changes - provar.automation.config.load — register a provardx-properties.json as the active config (required before compile/testrun) - provar.automation.metadata.download — download Salesforce metadata into the project - provar.qualityhub.defect.create — create Quality Hub defects from failed test executions - provar.testrun.report.locate — resolve artifact paths (JUnit.xml, HTML reports) for a completed test run - provar.testrun.rca — analyse a completed test run: classify failures, extract page objects, detect pre-existing issues - provar.testplan.add-instance — wire a test case into a plan suite by writing a .testinstance file - provar.testplan.create-suite — create a new test suite directory with .planitem inside a plan - provar.testplan.remove-instance — remove a .testinstance file from a plan suite - provar.nitrox.discover — discover projects containing NitroX (Hybrid Model) page objects - provar.nitrox.read — read NitroX .po.json files and return parsed content - provar.nitrox.validate — validate a NitroX .po.json against schema rules - provar.nitrox.generate — generate a new NitroX .po.json from a component description - provar.nitrox.patch — apply a JSON merge-patch to an existing NitroX .po.json file + 42 tools across: project inspection & org describe, Page Object and test-case + authoring/validation, test-suite/plan validation, properties files, Quality Hub + (test runs, defects, corpus examples), Provar Automation, ANT build, and NitroX + components. See docs/mcp.md for the full catalogue with schemas and examples. EXAMPLES Start MCP server (accepts stdio connections from Claude Desktop / Cursor): diff --git a/docs/VALIDATION_RULE_REGISTRY.md b/docs/VALIDATION_RULE_REGISTRY.md new file mode 100644 index 00000000..2e323736 --- /dev/null +++ b/docs/VALIDATION_RULE_REGISTRY.md @@ -0,0 +1,225 @@ +# Provar Validation Rule Registry + +> **Generated** by `scripts/build-validation-rule-registry.cjs`. Do not edit by hand — re-run the script after changing a rule. + +Provar test-case validation runs in two layers. This registry is the single canonical list of every rule across both. + +- **Layer 1 — structural validity** (hand-coded in `testCaseValidate.ts`): emits `issues[]` with `ERROR`/`WARNING`. `is_valid = error_count === 0`. +- **Layer 2 — best practices** (`provar_best_practices_rules.json`, same engine/weights as the Quality Hub API): emits `best_practices_violations[]` with `critical`/`major`/`minor`/`info` and a weighted `quality_score`. + +**Severity taxonomy:** `critical` = the test will not load/render in Provar; `major` = a runtime ERROR (loads, fails at execution); `minor` = warning; `info` = advisory. + +**The validity bridge (PDX-509):** a `critical` best-practice violation is surfaced into `issues[]` as an `ERROR` and therefore gates `is_valid` — EXCEPT where a Layer-1 check already owns the concept (then it is suppressed to avoid double-reporting). `major`/`minor`/`info` affect `quality_score` (and the `needs_improvement` status) only. The `status` field is tri-state: `invalid` (a critical) / `needs_improvement` (loads but `quality_score < quality_threshold`) / `valid`. + +**Counts:** Layer 1 — 23 rules (18 gating). Layer 2 — 178 rules (critical 64 / major 67 / minor 29 / info 18; 58 bridged to `is_valid`). + +## Layer 1 — Structural validity rules + +| Rule ID | Severity | Gates is_valid? | Applies to | Checks | +| ------------------------- | -------- | --------------- | ---------- | ---------------------------------------------------------------------------------------------- | +| `TC_001` | ERROR | Yes | document | XML declaration present ( first line). | +| `TC_002` | ERROR | Yes | document | XML is well-formed (parses without error). | +| `TC_003` | ERROR | Yes | document | Root element is . | +| `TC_010` | ERROR | Yes | testCase | testCase id, when present, is a non-negative integer (id is optional; guid is the identifier). | +| `TC_011` | ERROR | Yes | testCase | testCase has a guid attribute. | +| `TC_012` | ERROR | Yes | testCase | testCase guid is a valid UUID v4. | +| `TC_020` | ERROR | Yes | testCase | testCase has a element. | +| `TC_030` | ERROR | Yes | apiCall | Each apiCall has a guid attribute. | +| `TC_031` | ERROR | Yes | apiCall | Each apiCall guid is a valid UUID v4. | +| `TC_032` | ERROR | Yes | apiCall | Each apiCall has an apiId attribute. | +| `TC_033` | WARNING | No | apiCall | Each apiCall has a descriptive name attribute. | +| `TC_034` | ERROR | Yes | apiCall | Each apiCall has a testItemId attribute. | +| `TC_035` | ERROR | Yes | apiCall | apiCall testItemId is a whole number. | +| `DATA-001` | WARNING | No | testCase | only iterates under a test plan; flags direct testCase-mode execution. | +| `VAR-REF-001` | WARNING | No | argument | A whole-token {Var} stored as valueClass="string" (use class="variable"). | +| `VAR-REF-002` | WARNING | No | argument | {Var} tokens embedded in a plain string (use class="compound"). | +| `UI-TARGET-001` | ERROR | Yes | apiCall | UiWithScreen/UiWithRow target uses class="uiTarget". | +| `UI-LOCATOR-001` | ERROR | Yes | apiCall | UI action locator uses class="uiLocator". | +| `UI-INTERACTION-001` | ERROR | Yes | apiCall | UiDoAction interaction uses class="uiInteraction". | +| `UI-ASSERT-STRUCTURE-001` | ERROR | Yes | apiCall | UiAssert uses nested field/column/page assertion containers, not a flat argument. | +| `SETVALUES-STRUCTURE-001` | ERROR | Yes | apiCall | SetValues values argument uses class="valueList" with . | +| `ASSERT-001` | WARNING | No | apiCall | AssertValues namedValues format flagged for variable/Apex comparisons. | +| `COMPARISON-TYPE-001` | ERROR | Yes | apiCall | comparisonType is within the step-scoped enum subset (load-blocking otherwise). | + +## Layer 2 — Best-practice rules + +| Rule ID | Category | Severity | Weight | Gates is_valid? | Checks | +| ---------------------------------------- | -------------------------- | -------- | ------ | --------------- | ---------------------------------------------------------------------------------------------------- | +| `APEX-ASSERT-LAYOUT-001` | ApexAPI | major | 5 | No | ApexAssertLayout must have object and expected file. | +| `APEX-CONNECTION-REF-001` | ApexAPI | major | 5 | No | Apex API steps must reference a valid connection. | +| `APEX-CREATE-FIELDS-001` | ApexAPI | major | 5 | No | ApexCreateObject with fields must populate at least one field. | +| `APEX-CREATE-METADATA-001` | ApexAPI | major | 5 | No | ApexCreateObject and ApexUpdateObject must include parameter metadata. | +| `APEX-CREATE-UPDATE-STRUCTURE-001` | ApexAPI | major | 5 | No | ApexUpdateObject/ApexCreateObject field arguments must be direct, not nested in uiObjectFieldValue. | +| `APEX-DELETE-ID-001` | ApexAPI | major | 5 | No | ApexDeleteObject must have valid record ID. | +| `APEX-EXTRACT-LAYOUT-001` | ApexAPI | major | 5 | No | ApexExtractLayout must have object, file type, and path. | +| `APEX-OBJECT-TYPE-001` | ApexAPI | major | 5 | No | Apex CRUD operations must have valid object types. | +| `APEX-PARAM-GEN-URI-001` | ApexAPI | minor | 2 | No | Apex CRUD operations should include parameterGeneratorUri. | +| `APEX-READ-ASSERTIONS-001` | ApexAPI | minor | 3 | No | ApexReadObject should use resultAssertions instead of separate AssertValues. | +| `APEX-READ-FIELDS-STRUCTURE-001` | ApexAPI | major | 5 | No | ApexReadObject must use generatedParameters for field references, not fields argument with textType. | +| `APEX-READ-ID-001` | ApexAPI | major | 5 | No | ApexReadObject must have valid record ID. | +| `APEX-UPDATE-FIELDS-001` | ApexAPI | major | 5 | No | ApexUpdateObject must specify fields to update. | +| `APEX-UPDATE-ID-001` | ApexAPI | major | 5 | No | ApexUpdateObject must have valid record ID. | +| `CONN-ARG-001` | ApexAPI | minor | 2 | No | Connection arguments must use correct naming convention. | +| `BUILD-PLAN-001` | BuildAndCI | major | 5 | No | Regression Test Plan exists for CI. | +| `APEX-AUTOCLEANUP-001` | ConnectionsAndEnvironments | minor | 2 | No | Prefer autoCleanup over manual ApexDeleteObject steps. | +| `CONN-APEX-001` | ConnectionsAndEnvironments | critical | 8 | Yes | Apex API calls reference valid connections. | +| `CONN-DB-001` | ConnectionsAndEnvironments | critical | 8 | Yes | Database operations reference valid connections. | +| `CONN-DB-002` | ConnectionsAndEnvironments | major | 5 | No | DbConnect resultName must match dbConnectionName in DB operations. | +| `CONN-UI-001` | ConnectionsAndEnvironments | critical | 8 | Yes | UI operations reference valid connections. | +| `DB-CONNECT-001` | ConnectionsAndEnvironments | critical | 8 | Yes | DbConnect has connectionName. | +| `DB-CONNECT-002` | ConnectionsAndEnvironments | critical | 8 | Yes | DbConnect has resultName. | +| `ENV-CONN-001` | ConnectionsAndEnvironments | major | 5 | No | Admin connection supports Login-As. | +| `ENV-CONN-002` | ConnectionsAndEnvironments | minor | 2 | No | Connection names should not contain environment specifiers. | +| `REST-CONN-001` | ConnectionsAndEnvironments | critical | 8 | Yes | WebConnect has connectionName. | +| `REST-CONN-002` | ConnectionsAndEnvironments | critical | 8 | Yes | WebConnect has resultName. | +| `UI-CONN-LITERAL-001` | ConnectionsAndEnvironments | critical | 8 | Yes | uiConnectionName must be a literal string. | +| `UI-CONNECT-ARGS-001` | ConnectionsAndEnvironments | critical | 10 | Yes | UiConnect has invalid arguments (ApexConnect arguments used). | +| `UI-NITROX-CONNECT-ARGS-001` | ConnectionsAndEnvironments | critical | 10 | Yes | NitroX MS connect step has invalid arguments. | +| `UI-NITROX-VARIANT-ARG-001` | ConnectionsAndEnvironments | minor | 2 | No | NitroX MS connect step missing variant-specific argument. | +| `DDT-EXCEL-001` | DataDrivenTesting | major | 5 | No | Excel headers match field label or API name. | +| `DDT-NO-FUNC-001` | DataDrivenTesting | major | 5 | No | No Excel functions in data. | +| `DDT-VAR-001` | DataDrivenTesting | minor | 3 | No | No hardcoded values in steps. | +| `VAR-NAMING-001` | DataDrivenTesting | major | 5 | No | Variable names must use valid identifiers. | +| `VAR-USAGE-001` | DataDrivenTesting | minor | 2 | No | Variable references use correct syntax. | +| `UI-BINDING-ORDER-001` | LocatorPatterns | major | 5 | No | UI binding parameter order must have object= first. | +| `MAINT-FOLDER-001` | MaintenanceAndFolders | minor | 3 | No | Folder-level setup test per application segment. | +| `MAINT-VERSION-001` | MaintenanceAndFolders | info | 1 | No | Consistent Provar/OS/browser versions. | +| `APEX-RESULTNAME-001` | NamingConventions | minor | 2 | No | ApexConnect resultName is unique. | +| `CUSTOM-FIELD-001` | NamingConventions | major | 5 | No | Custom fields end with \_\_c. | +| `NC-FIELD-001` | NamingConventions | minor | 2 | No | Field names use camelCase. | +| `NC-FOLDER-001` | NamingConventions | major | 5 | No | Folder names are modular and title-cased. | +| `NC-PARAM-001` | NamingConventions | major | 5 | No | Parameters and variables use camelCase. | +| `NC-PO-001` | NamingConventions | major | 5 | No | Page Objects use PascalCase. | +| `NC-TESTCASE-001` | NamingConventions | minor | 2 | No | Test case names use consistent naming convention. | +| `SETVALUES-NAME-001` | NamingConventions | critical | 8 | Yes | SetValues namedValue elements have name attribute. | +| `CALLABLE-VISIBILITY-001` | ReusabilityAndCallables | critical | 8 | Yes | Called test cases are marked as Callable. | +| `REUSE-CALL-001` | ReusabilityAndCallables | minor | 2 | No | Callable tests reside in Callables folder. | +| `REUSE-CALL-002` | ReusabilityAndCallables | minor | 2 | No | Callable tests are parameterized. | +| `REUSE-CALL-003` | ReusabilityAndCallables | minor | 2 | No | Callable tests executable in isolation. | +| `ASSERT-STR-VAR-001` | StructureAndGrouping | major | 5 | No | AssertValues must not use string literal to reference a variable. | +| `BDD-AND-LIMIT-001` | StructureAndGrouping | info | 1 | No | Limit And/But chain length. | +| `BDD-GIVEN-FIRST-001` | StructureAndGrouping | info | 1 | No | BDD scenario should start with Given. | +| `BDD-ORDER-001` | StructureAndGrouping | info | 1 | No | BDD steps should follow logical order. | +| `CONTROL-FINALLY-001` | StructureAndGrouping | major | 5 | No | Finally block should be at end of test. | +| `RENDER-ARG-001` | StructureAndGrouping | critical | 10 | Yes | All arguments must have value elements. | +| `RENDER-BOOL-001` | StructureAndGrouping | critical | 10 | Yes | Boolean values must use lowercase. | +| `RENDER-CASE-001` | StructureAndGrouping | critical | 10 | Yes | valueClass attributes must use lowercase. | +| `RENDER-ROOT-001` | StructureAndGrouping | minor | 3 | No | Test case root element should not have unknown attributes. | +| `SETVALUES-FUNC-STR-001` | StructureAndGrouping | major | 5 | No | SetValues must not use string interpolation for function calls. | +| `SETVALUES-INVALID-ELEMENT-001` | StructureAndGrouping | critical | 10 | Yes | SetValues must not contain invalid child elements. | +| `SETVALUES-ZERO-IDX-001` | StructureAndGrouping | major | 5 | No | SetValues string expression must not use [0] index. | +| `STEP-NAMES-001` | StructureAndGrouping | minor | 2 | No | Custom step names for UI asserts and sets. | +| `STRUCT-GROUP-001` | StructureAndGrouping | minor | 2 | No | All steps are inside Group steps or BDD structure. | +| `STRUCT-SUMMARY-001` | StructureAndGrouping | info | 1 | No | Test case has top-level summary. | +| `UI-ASSERT-STRUCT-001` | StructureAndGrouping | critical | 8 | Yes | UiAssert steps must include all required arguments. | +| `UI-ASSERT-STRUCT-002` | StructureAndGrouping | critical | 10 | Yes | UiAssert steps must NOT contain generatedParameters. | +| `VALUE-CLASS-001` | StructureAndGrouping | critical | 10 | Yes | Value elements must use valid class attribute. | +| `AI-CONVERSATION-SESSION-001` | TestCaseDesign | critical | 8 | Yes | AIAgentConversation requires valid session. | +| `AI-IMAGE-CONFIDENCE-001` | TestCaseDesign | major | 5 | No | ImageValidator confidence should be 0.0-1.0. | +| `AI-SESSION-WEBCONNECT-001` | TestCaseDesign | critical | 8 | Yes | AIAgentSession requires WebConnect first. | +| `AI-UTTERANCE-COUNT-001` | TestCaseDesign | info | 1 | No | GenerateUtterance count should be reasonable. | +| `APEX-BULK-LIMIT-001` | TestCaseDesign | info | 1 | No | ApexBulk should be used for large data volumes. | +| `APEX-EXECUTE-SYNTAX-001` | TestCaseDesign | critical | 8 | Yes | ApexExecute code should be valid Apex syntax. | +| `APEX-REUSE-CONN-001` | TestCaseDesign | major | 5 | No | ApexConnect reuseConnectionName should be left blank. | +| `ASSERT-ACTUAL-001` | TestCaseDesign | critical | 8 | Yes | AssertValues has actualValue. | +| `ASSERT-API-001` | TestCaseDesign | critical | 8 | Yes | Must use AssertValues API, not deprecated Assert API. | +| `ASSERT-ARG-ORDER-001` | TestCaseDesign | info | 1 | No | AssertValues arguments must be in correct order. | +| `ASSERT-COMPARISON-001` | TestCaseDesign | critical | 8 | Yes | AssertValues has comparisonType. | +| `ASSERT-DATE-FORMAT-001` | TestCaseDesign | minor | 4 | No | Date/DateTime assertions should use proper format functions. | +| `ASSERT-EXPECTED-001` | TestCaseDesign | critical | 8 | Yes | AssertValues has expectedValue. | +| `ASSERT-VALUES-COMPARISON-001` | TestCaseDesign | major | 5 | No | AssertValues should have meaningful expected values. | +| `BDD-GIVEN-001` | TestCaseDesign | major | 5 | No | Given steps have description. | +| `BDD-THEN-001` | TestCaseDesign | major | 5 | No | Then steps have description. | +| `BDD-WHEN-001` | TestCaseDesign | major | 5 | No | When steps have description. | +| `CLEANUP-CONSISTENCY-001` | TestCaseDesign | major | 5 | No | Manual cleanup matches object creation. | +| `CLEANUP-ORDER-001` | TestCaseDesign | minor | 2 | No | Cleanup deletes objects in reverse order. | +| `CONTROL-FINALLY-001` | TestCaseDesign | major | 5 | No | Finally block must have description and be at end. | +| `CONTROL-FOREACH-001` | TestCaseDesign | major | 4 | No | ForEach loops have valid source collection. | +| `CONTROL-FOREACH-002` | TestCaseDesign | critical | 8 | Yes | ForEach loops have valueName to store current item. | +| `CONTROL-IF-001` | TestCaseDesign | critical | 8 | Yes | If statements have conditions. | +| `CONTROL-SLEEP-001` | TestCaseDesign | major | 5 | No | Sleep step duration and frequency issues. | +| `CONTROL-SLEEP-001` | TestCaseDesign | info | 1 | No | Sleep duration should be under 5 seconds. | +| `CONTROL-SLEEP-002` | TestCaseDesign | critical | 8 | Yes | Sleep steps have duration specified. | +| `CONTROL-SWITCH-001` | TestCaseDesign | critical | 8 | Yes | Switch statements have value expression. | +| `CONTROL-WAITFOR-001` | TestCaseDesign | critical | 8 | Yes | WaitFor steps have condition. | +| `CONTROL-WAITFOR-002` | TestCaseDesign | major | 5 | No | WaitFor steps have max iterations limit. | +| `CONTROL-WHILE-001` | TestCaseDesign | critical | 8 | Yes | While loops have exit conditions. | +| `CONTROL-WHILE-MAX-001` | TestCaseDesign | major | 5 | No | While loop must have termination condition. | +| `CREATE-RESULT-001` | TestCaseDesign | major | 5 | No | ApexCreateObject steps specify resultIdName. | +| `DATA-DB-WHERE-001` | TestCaseDesign | critical | 8 | Yes | DbDelete and DbUpdate should have WHERE clause. | +| `DATA-REST-BODY-001` | TestCaseDesign | major | 5 | No | POST/PUT/PATCH should have request body. | +| `DATA-REST-METHOD-001` | TestCaseDesign | critical | 8 | Yes | RestRequest method should be valid HTTP method. | +| `DATA-REST-STATUS-001` | TestCaseDesign | info | 1 | No | Validate REST response status. | +| `DATA-SOAP-XML-001` | TestCaseDesign | critical | 8 | Yes | SOAP request body should be well-formed XML. | +| `DATA-TYPE-BOOL-001` | TestCaseDesign | critical | 8 | Yes | Boolean values are 'true' or 'false'. | +| `DATA-TYPE-NUMBER-001` | TestCaseDesign | info | 0 | No | Numeric values are valid numbers. | +| `DESIGN-APIUI-001` | TestCaseDesign | minor | 3 | No | Prefer API for setup where possible. | +| `DESIGN-GROUP-001` | TestCaseDesign | minor | 2 | No | Use Group Steps or BDD structure for logical phases. | +| `FILE-READ-PATH-001` | TestCaseDesign | critical | 8 | Yes | Read dataUrl should be valid file path. | +| `FILE-WRITE-PATH-001` | TestCaseDesign | critical | 8 | Yes | Write dataUrl should be writable. | +| `LOG-LEVEL-001` | TestCaseDesign | info | 1 | No | Log messages use appropriate log levels. | +| `MESSAGING-SUBSCRIBE-BEFORE-RECEIVE-001` | TestCaseDesign | critical | 8 | Yes | Subscribe before ReceiveMessage. | +| `MESSAGING-TIMEOUT-001` | TestCaseDesign | info | 1 | No | ReceiveMessage timeout should be reasonable. | +| `PICKLIST-001` | TestCaseDesign | major | 7 | No | Picklist values should match Salesforce metadata. | +| `PO-FIELD-EXISTS-001` | TestCaseDesign | major | 5 | No | Page Object locator references non-existent field. | +| `REST-REQUEST-001` | TestCaseDesign | critical | 8 | Yes | RestRequest has connectionName. | +| `SETVALUES-STRUCTURE-001` | TestCaseDesign | critical | 8 | Yes | SetValues steps have namedValues container. | +| `SETVALUES-VALUE-001` | TestCaseDesign | critical | 8 | Yes | SetValues namedValue elements have value element. | +| `SF-CONVERT-LEAD-STATUS-001` | TestCaseDesign | critical | 8 | Yes | ConvertLead status must be valid. | +| `SF-LAYOUT-EXTRACT-BEFORE-ASSERT-001` | TestCaseDesign | minor | 2 | No | ExtractSalesforceLayout before AssertSalesforceLayout. | +| `SOQL-IN-LOOP-001` | TestCaseDesign | major | 5 | No | SOQL queries must not be inside loops. | +| `SOQL-QUERY-001` | TestCaseDesign | critical | 8 | Yes | ApexSoqlQuery has soqlQuery argument. | +| `SOQL-RESULT-001` | TestCaseDesign | critical | 8 | Yes | SOQL queries specify resultListName. | +| `SOQL-SELECT-ID-001` | TestCaseDesign | minor | 2 | No | SOQL queries include Id and Name. | +| `SOQL-STRUCTURE-001` | TestCaseDesign | critical | 8 | Yes | SOQL queries have SELECT and FROM clauses. | +| `SOQL-WHERE-001` | TestCaseDesign | major | 5 | No | SOQL queries include WHERE or LIMIT clause. | +| `SQL-QUERY-001` | TestCaseDesign | critical | 8 | Yes | SqlQuery has query argument. | +| `SQL-QUERY-002` | TestCaseDesign | critical | 8 | Yes | SqlQuery has dbConnectionName. | +| `STEP-DISABLED-001` | TestCaseDesign | minor | 2 | No | Disabled test steps should be removed. | +| `STEP-ITEMID-001` | TestCaseDesign | critical | 8 | No | testItemId values are whole numbers. _(Layer-1 owns this concept; not bridged)_ | +| `TEST-LENGTH-001` | TestCaseDesign | minor | 3 | No | Test case should not be excessively long. | +| `UI-ALERT-HANDLE-001` | TestCaseDesign | info | 1 | No | UiHandleAlert should capture alert text. | +| `UI-ASSERT-COMPOUND-001` | TestCaseDesign | major | 6 | No | UiAssert must use compound fields for component field assertions. | +| `UI-ASSERT-FIELDLOCATOR-001` | TestCaseDesign | major | 5 | No | UiAssert fieldLocator uses object+field binding. | +| `UI-ASSERT-FIELDLOCATOR-002` | TestCaseDesign | major | 5 | No | UiAssert fieldAssertion must not wrap fieldLocator in uiLocator. | +| `UI-ASSERT-FIELDLOCATOR-003` | TestCaseDesign | critical | 10 | Yes | UiAssert bare locator in Salesforce metadata context causes render failure. | +| `UI-ASSERT-TYPE-001` | TestCaseDesign | minor | 2 | No | UiAssert steps specify assertion type. | +| `UI-DOACTION-VALUE-001` | TestCaseDesign | critical | 8 | Yes | UiDoAction Set requires value argument. | +| `UI-FIELD-METADATA-001` | TestCaseDesign | major | 5 | No | UiDoAction/UiAssert fields should exist in Salesforce metadata. | +| `UI-FILL-VERIFY-001` | TestCaseDesign | info | 1 | No | Verify fields after UiFill. | +| `UI-LOCATOR-ACTION-001` | TestCaseDesign | major | 5 | No | UiDoAction locator URIs must use valid patterns. | +| `UI-LOCATOR-BINDING-001` | TestCaseDesign | major | 5 | No | Ui locator built-in actions use object binding. | +| `UI-LOCATOR-BUTTON-CASING-001` | TestCaseDesign | major | 5 | No | Standard Salesforce flow buttons must use correct locator pattern. | +| `UI-LOCATOR-RECORDTYPE-001` | TestCaseDesign | major | 5 | No | Record Type field locator must use name=RecordType not name=recordTypeId. | +| `UI-LOCATOR-SAVE-001` | TestCaseDesign | major | 5 | No | Save button locator must use correct pattern. | +| `UI-LOOKUP-ID-001` | TestCaseDesign | major | 6 | No | UiDoAction lookup fields should use Name values, not IDs. | +| `UI-NAVIGATE-PREFER-SCREEN-001` | TestCaseDesign | info | 1 | No | Prefer UiWithScreen over UiNavigate for Salesforce. | +| `UI-SCREEN-NAV-001` | TestCaseDesign | major | 5 | No | First UiWithScreen must use navigate=Always or IfNeccessary. | +| `UI-SCREEN-NAV-002` | TestCaseDesign | minor | 2 | No | First UiWithScreen should prefer navigate=Always over IfNeccessary. | +| `UI-SCREEN-OBJID-001` | TestCaseDesign | major | 5 | No | UiWithScreen with navigate=Always for Edit/View must have sfUiTargetObjectId. | +| `UI-SCREEN-TARGET-001` | TestCaseDesign | major | 5 | No | UiWithScreen target URIs must use valid patterns. | +| `UI-TARGET-ACTION-001` | TestCaseDesign | major | 5 | No | UiWithScreen target uses invalid action value. | +| `UI-WAIT-VALUECLASS-001` | TestCaseDesign | major | 5 | No | Wait arguments must use uiWait value class. | +| `UTIL-MATCH-REGEX-001` | TestCaseDesign | critical | 8 | Yes | Match regex pattern should be valid. | +| `UTIL-REPLACE-EMPTY-001` | TestCaseDesign | major | 5 | No | Replace searchString should not be empty. | +| `UTIL-SPLIT-DELIMITER-001` | TestCaseDesign | major | 5 | No | Split delimiter should not be empty. | +| `VALID-GUID-001` | TestCaseDesign | critical | 8 | No | Test case has valid identifier. _(Layer-1 owns this concept; not bridged)_ | +| `VALID-STEPS-001` | TestCaseDesign | critical | 8 | No | Test case has steps element. _(Layer-1 owns this concept; not bridged)_ | +| `VAR-PROPERTY-001` | TestCaseDesign | major | 6 | No | Variable property references must be valid. | +| `VAR-REFERENCE-001` | TestCaseDesign | major | 5 | No | Variables are defined before use. | +| `VAR-STRING-LITERAL-001` | TestCaseDesign | major | 5 | No | Variable reference stored as plain string. | +| `APEX-APIPARAM-HALLUCINATION-001` | XMLSchema | critical | 10 | Yes | Apex CRUD apiParam elements must be self-closing without summary/type children. | +| `APEX-CONNECT-ARGS-001` | XMLSchema | critical | 10 | Yes | ApexConnect - Only valid argument IDs allowed. | +| `APEX-CONNECT-CONNID-001` | XMLSchema | critical | 10 | Yes | ApexConnect connectionId must use valueClass='id'. | +| `API-UNKNOWN-001` | XMLSchema | critical | 10 | Yes | API identifier must be a valid Provar API. | +| `FUNCCALL-VALID-001` | XMLSchema | major | 6 | No | funcCall id must be a valid Provar function. | +| `RENDER-DATE-VALUECLASS-001` | XMLSchema | critical | 10 | Yes | valueClass='date' requires epoch timestamp, not date string. | +| `SCHEMA-EMPTY-001` | XMLSchema | minor | 2 | No | Test case should not be empty. | +| `SCHEMA-ID-001` | XMLSchema | critical | 10 | No | Test case must have valid identifier. _(Layer-1 owns this concept; not bridged)_ | +| `SCHEMA-LEGACY-001` | XMLSchema | info | 1 | No | Consider migrating from registryId to id or guid. | +| `SCHEMA-ROOT-001` | XMLSchema | critical | 10 | No | Test case root element must be testCase. _(Layer-1 owns this concept; not bridged)_ | +| `SCHEMA-STEPS-001` | XMLSchema | critical | 10 | No | Test case must have steps element. _(Layer-1 owns this concept; not bridged)_ | +| `SCHEMA-URI-001` | XMLSchema | critical | 10 | Yes | URI attributes must properly encode ampersands. | +| `SCHEMA-VALUE-001` | XMLSchema | critical | 10 | Yes | Value elements must not use text attribute. | +| `STRUCT-ATTR-001` | XMLSchema | info | 1 | No | Test case should have failureBehaviour attribute. | +| `UI-NEST-STRUCT-001` | XMLSchema | major | 7 | No | UI action steps must be nested inside a UiWithScreen substeps clause. | diff --git a/docs/mcp-pilot-guide.md b/docs/mcp-pilot-guide.md index f2ce6b6f..06f5e6bc 100644 --- a/docs/mcp-pilot-guide.md +++ b/docs/mcp-pilot-guide.md @@ -238,6 +238,9 @@ Prompt your AI assistant: **What to look for:** - `validity_score` and `quality_score` both returned (0–100) +- A tri-state `status` — `valid`, `needs_improvement` (structurally valid but `quality_score` below the threshold), or `invalid` (a critical issue gates validity) +- `quality_threshold` (the effective threshold, default **90**) and `meets_quality_threshold` returned alongside the score +- Critical best-practice violations surface as `is_valid: false` issues (the validity bridge) — e.g. a hallucinated `apiId` or a non-integer `testItemId` - Specific rule violations called out (e.g. TC_010 missing test case ID, TC_001 missing XML declaration) - Best-practices suggestions (e.g. hardcoded credentials, missing step descriptions) - `validation_source: "local"` if no API key is configured, `"quality_hub"` if authenticated diff --git a/docs/mcp.md b/docs/mcp.md index 1a3d045d..826898ad 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -75,6 +75,9 @@ The Provar DX CLI ships with a built-in **Model Context Protocol (MCP) server** - [provar.loop.db](#provarloopdb) - [MCP Resources](#mcp-resources) - [provar://docs/step-reference](#provardocsstep-reference) + - [provar://schema/test-step](#provarschematest-step) + - [provar://docs/validation-rules](#provardocsvalidation-rules) + - [provar://docs/tool-guide](#provardocstool-guide) - [provar://nitrox/component-catalog](#provarnitroxcomponent-catalog) - [provar://nitrox/catalog-source](#provarnitroxcatalog-source) - [AI loop pattern](#ai-loop-pattern) @@ -189,18 +192,19 @@ sf provar mcp start -a /workspace/project-a -a /workspace/project-b The MCP server reads the following environment variables at startup or during tool invocation. Internal/dev-only variables (license bypass, ALGAS dev credentials) are intentionally not documented here — they remain source-only and are not supported for production use. -| Variable | Purpose | Default | -| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | -| `PROVAR_HOME` | Provar Automation install root. Used to locate license files (`/.licenses/*.properties`) and resolve home-relative tool defaults. | `~/Provar` (`%USERPROFILE%\Provar` on Windows) | -| `PROVAR_API_KEY` | API key for Quality Hub validation. Takes priority over any stored key in `~/.provar/credentials.json`. Must start with `pv_k_` — any other value is ignored. | None — falls back to stored credentials | -| `PROVAR_QUALITY_HUB_URL` | Override the Quality Hub API base URL. Set when pointing at a non-default Quality Hub environment. | Dev API Gateway URL (`/dev`) | -| `PROVAR_MCP_TOOLS` | Comma-separated list of tool groups to register at startup. Deep-dive: [Tool group filtering](#tool-group-filtering-provar_mcp_tools). | All groups registered | -| `PROVAR_MCP_SCHEMA_MODE` | Set to `compact` to shorten all tool descriptions. Deep-dive: [Compact descriptions](#compact-descriptions-provar_mcp_schema_mode). | Standard (full) descriptions | -| `PROVAR_MCP_MAX_TOOL_DEPTH` | Agentic loop guard — max tool calls per MCP session before further calls return `TOOL_BUDGET_EXCEEDED`. Deep-dive: [Agentic loop guard](#agentic-loop-guard-provar_mcp_max_tool_depth). | `50` | -| `PROVAR_MCP_EMIT_TOKEN_META` | When `true`, appends a `_meta` token-attribution block to every tool response. Deep-dive: [Per-call token attribution](#per-call-token-attribution-provar_mcp_emit_token_meta). | unset (no `_meta` block) | -| `PROVAR_MCP_VALIDATION_DIR` | Override the directory where `provar_testcase_validate` writes validation diff artifacts. | `/.provar-mcp/validation/` | -| `PROVAR_NO_UPDATE_CHECK` | When set (any non-empty value), skips the startup npm-registry update check. Same effect as `--no-update-check`. | unset (check runs) | -| `PROVAR_AUTO_DEFECTS` | When `1`, enables the Quality Hub auto-defect creation flow. Normally set by passing the `--auto-defects` flag rather than directly. | unset (auto-defects disabled) | +| Variable | Purpose | Default | +| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | +| `PROVAR_HOME` | Provar Automation install root. Used to locate license files (`/.licenses/*.properties`) and resolve home-relative tool defaults. | `~/Provar` (`%USERPROFILE%\Provar` on Windows) | +| `PROVAR_API_KEY` | API key for Quality Hub validation. Takes priority over any stored key in `~/.provar/credentials.json`. Must start with `pv_k_` — any other value is ignored. | None — falls back to stored credentials | +| `PROVAR_QUALITY_HUB_URL` | Override the Quality Hub API base URL. Set when pointing at a non-default Quality Hub environment. | Dev API Gateway URL (`/dev`) | +| `PROVAR_MCP_TOOLS` | Comma-separated list of tool groups to register at startup. Deep-dive: [Tool group filtering](#tool-group-filtering-provar_mcp_tools). | All groups registered | +| `PROVAR_MCP_SCHEMA_MODE` | Set to `compact` to shorten all tool descriptions. Deep-dive: [Compact descriptions](#compact-descriptions-provar_mcp_schema_mode). | Standard (full) descriptions | +| `PROVAR_MCP_MAX_TOOL_DEPTH` | Agentic loop guard — max tool calls per MCP session before further calls return `TOOL_BUDGET_EXCEEDED`. Deep-dive: [Agentic loop guard](#agentic-loop-guard-provar_mcp_max_tool_depth). | `50` | +| `PROVAR_MCP_EMIT_TOKEN_META` | When `true`, appends a `_meta` token-attribution block to every tool response. Deep-dive: [Per-call token attribution](#per-call-token-attribution-provar_mcp_emit_token_meta). | unset (no `_meta` block) | +| `PROVAR_MCP_VALIDATION_DIR` | Override the directory where `provar_testcase_validate` writes validation diff artifacts. | `/.provar-mcp/validation/` | +| `PROVAR_MCP_QUALITY_THRESHOLD` | Minimum `quality_score` for a test case to count as `valid` (vs `needs_improvement`) across all validation tools. A per-call `quality_threshold` argument overrides it. Out-of-range or unparseable values are ignored. | `90` | +| `PROVAR_NO_UPDATE_CHECK` | When set (any non-empty value), skips the startup npm-registry update check. Same effect as `--no-update-check`. | unset (check runs) | +| `PROVAR_AUTO_DEFECTS` | When `1`, enables the Quality Hub auto-defect creation flow. Normally set by passing the `--auto-defects` flag rather than directly. | unset (auto-defects disabled) | ### Setting these in your MCP client config @@ -561,7 +565,7 @@ Paste the [standard config](#the-standard-config-recommended) into either file u } ``` -> **Tool limit:** Agentforce Vibes loads approximately 20 tools per MCP server at runtime. The Provar MCP server exposes 38 tools — you may need to restart or re-enable the server between tasks if the active tool list gets out of date. Salesforce is tracking this limit; consult the [Agentforce Vibes MCP documentation](https://developer.salesforce.com/docs/platform/einstein-for-devs/guide/devagent-mcp.html) for the latest guidance. +> **Tool limit:** Agentforce Vibes loads approximately 20 tools per MCP server at runtime. The Provar MCP server exposes 42 tools — you may need to restart or re-enable the server between tasks if the active tool list gets out of date. Salesforce is tracking this limit; consult the [Agentforce Vibes MCP documentation](https://developer.salesforce.com/docs/platform/einstein-for-devs/guide/devagent-mcp.html) for the latest guidance. @@ -892,13 +896,13 @@ The tool's chip-level `title` — `Generate Test Case (full steps in one call)` ```xml - + ... ``` -- `id` is always the integer literal `"1"` — Provar ignores any other value +- `id` is a numeric integer **label**, not a uniqueness key — Provar identifies the test case by its `guid`. When the output path sits inside an existing Provar project (a directory tree containing a `.testproject` marker, within the allowed roots), the generator auto-allocates the next integer after the highest `id` already in use, so a new case does not land on a duplicate `id="1"`. With no surrounding project — preview/`dry_run` runs, or output outside the allowed roots — it defaults to `1`. Regenerating over an existing file preserves that file's id. The chosen value is echoed back as `test_case_id`. - No `name` attribute on `` — Provar derives the name from the file name - `` must appear before `` - `standalone="no"` is required in the XML declaration @@ -954,7 +958,7 @@ AssertValues uses **flat** argument structure (`expectedValue`, `actualValue`, ` | Mode | Behaviour | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `auto` (default) | When a `UiWithScreen` is followed by UI action siblings (any of `UiDoAction`, `UiAssert`, `UiRead`, `UiFill`, `UiNavigate`, `UiWithRow`, `UiHandleAlert`), those siblings are absorbed into the screen's `` block. The grouping run stops at the next `UiWithScreen`, any non-UI step (`SetValues`, `ApexConnect`, …), or end of list. `UiWithRow` plays a dual role: when it follows a `UiWithScreen` it is pulled in as a child container and absorbs its own following UI actions. When the payload contains screen containers but no `UiWithScreen` at root (e.g. starts with `UiWithRow`), the generator synthesizes a root `UiWithScreen` wrapper (`target` = `target_uri` or `sf:ui:target`) so the output still satisfies `UI-NEST-STRUCT-001` — without that wrapper, the root `UiWithRow` itself would fail validation. `testItemId`s are assigned depth-first: parent screen, then its substeps slot, then its children. Numbering remains sequential and gap-free. | -| `flat` | Legacy behaviour: every step is emitted as a root sibling, no `` block is generated. Use this for payloads that are already structured correctly by the caller, or when debugging the pre-PDX-495 shape. | +| `flat` | Legacy behaviour: every step is emitted as a root sibling, no `` block is generated. Use this for payloads that are already structured correctly by the caller, or when debugging the legacy flat shape. | | `single-screen` | Wraps every step in one synthetic `UiWithScreen` whose `target` is `sf:ui:target` (or the URI passed via `target_uri`). Matches the existing `ui:pageobject:target` semantics. Use for tests that all live on a single screen. | If `target_uri` is `ui:pageobject:target?pageId=…` the single-screen wrap takes precedence regardless of `grouping_mode` — this is the pre-existing non-SF nesting behaviour. @@ -979,9 +983,9 @@ If `target_uri` is `ui:pageobject:target?pageId=…` the single-screen wrap take Validation rules: `UI-NITROX-CONNECT-ARGS-001` (critical, bans ApexConnect-only and cross-variant args), `UI-NITROX-VARIANT-ARG-001` (minor, requires variant-specific arg unless declared in ``). -**Output** — `{ xml_content: string, file_path?: string, written: boolean, validation?: ValidationResult }` +**Output** — `{ xml_content: string, file_path?: string, written: boolean, test_case_id: number, validation?: ValidationResult }` -`validation` is present when `validate_after_edit=true` (default). If the generated XML fails validation the tool returns `TESTCASE_INVALID` with the `validation` field in `details`. +`test_case_id` is the `id` written into the `` element (auto-allocated as highest-in-project + 1 on the write path; `1` for preview runs). `validation` is present when `validate_after_edit=true` (default). If the generated XML fails validation the tool returns `TESTCASE_INVALID` with the `validation` field in `details`. **Error codes** @@ -1010,43 +1014,51 @@ Validates an XML test case for schema correctness (validity score) and best prac **Input** -| Parameter | Type | Required | Description | -| ----------------- | --------------------------------- | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `content` | string | one of `content`/`xml`/`file_path` required | XML content to validate (MCP field name) | -| `xml` | string | one of `content`/`xml`/`file_path` required | XML content to validate (API-compatible alias) | -| `file_path` | string | one of `content`/`xml`/`file_path` required | Path to the `.testcase` XML file | -| `detail` | `summary` \| `standard` \| `full` | no | Response verbosity. `"summary"`: is_valid, scores, and stop signal only. `"standard"`/`"full"`: full issues list (default). | -| `baseline_run_id` | string | no | `run_id` from a previous call. Returns only new/resolved issues since that run (`{ added, resolved, unchanged_count, run_id }`). Returns `BASELINE_NOT_FOUND` if the run ID is unknown. | +| Parameter | Type | Required | Description | +| ------------------- | --------------------------------- | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `content` | string | one of `content`/`xml`/`file_path` required | XML content to validate (MCP field name) | +| `xml` | string | one of `content`/`xml`/`file_path` required | XML content to validate (API-compatible alias) | +| `file_path` | string | one of `content`/`xml`/`file_path` required | Path to the `.testcase` XML file | +| `detail` | `summary` \| `standard` \| `full` | no | Response verbosity. `"summary"`: is_valid, scores, and stop signal only. `"standard"`/`"full"`: full issues list (default). | +| `quality_threshold` | number (0–100) | no | Minimum `quality_score` for `status` to be `"valid"` rather than `"needs_improvement"`. Does **not** affect `is_valid` (only critical defects do). Precedence: this arg → `PROVAR_MCP_QUALITY_THRESHOLD` env → `90`. | +| `baseline_run_id` | string | no | `run_id` from a previous call. Returns only new/resolved issues since that run (`{ added, resolved, unchanged_count, run_id }`). Returns `BASELINE_NOT_FOUND` if the run ID is unknown. | **Output** -| Field | Type | Description | -| -------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------ | -| `run_id` | string | Stable identifier for this validation run. Pass as `baseline_run_id` in the next call to receive only new/resolved issues. | -| `completeness_score` | number (0–1) | Ratio of valid test cases to total test cases validated (`0.0`–`1.0`). | -| `recommended_next_action` | string | `"stop"` (all passing), `"continue"` (issues remain), or `"escalate"` (no baseline yet — run without `baseline_run_id` first). | -| `is_valid` | boolean | `true` if zero ERROR-level schema violations | -| `validity_score` | number (0–100) | Schema compliance score (100 − errorCount × 20) | -| `quality_score` | number (0–100) | Best-practices score (weighted deduction formula) | -| `error_count` | integer | Schema error count | -| `warning_count` | integer | Schema warning count | -| `step_count` | integer | Number of `` steps | -| `test_case_id` | string | Value of the `id` attribute | -| `test_case_name` | string | Value of the `name` attribute | -| `issues` | array | Schema issues with `rule_id`, `severity`, `message` | -| `best_practices_violations` | array | Best-practices violations with `rule_id`, `severity`, `weight`, `message` | -| `best_practices_rules_evaluated` | integer | How many best-practices rules were checked | -| `validation_source` | string | `quality_hub`, `local`, or `local_fallback` — see Authentication section | -| `validation_warning` | string | Present when `validation_source` is `local` (onboarding) or `local_fallback` (explains why API failed) | +| Field | Type | Description | +| -------------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `run_id` | string | Stable identifier for this validation run. Pass as `baseline_run_id` in the next call to receive only new/resolved issues. | +| `completeness_score` | number (0–1) | Ratio of valid test cases to total test cases validated (`0.0`–`1.0`). | +| `recommended_next_action` | string | `"stop"` (all passing), `"continue"` (issues remain), or `"escalate"` (no baseline yet — run without `baseline_run_id` first). | +| `is_valid` | boolean | `true` if the test case will load in Provar — i.e. zero ERROR-level schema violations **and** zero `critical` best-practice violations (the latter are bridged into `issues[]`; see note below) | +| `status` | string | Tri-state verdict: `"invalid"` (a critical defect — will not load), `"needs_improvement"` (loads but `quality_score` is below `quality_threshold`), or `"valid"` (loads and clears the bar) | +| `quality_threshold` | number (0–100) | The effective threshold applied (resolved from the `quality_threshold` arg, the `PROVAR_MCP_QUALITY_THRESHOLD` env var, or the default `90`) | +| `meets_quality_threshold` | boolean | `true` when `quality_score >= quality_threshold` | +| `validity_score` | number (0–100) | Schema compliance score (100 − errorCount × 20) | +| `quality_score` | number (0–100) | Best-practices score (weighted deduction formula) | +| `error_count` | integer | Schema error count | +| `warning_count` | integer | Schema warning count | +| `step_count` | integer | Number of `` steps | +| `test_case_id` | string | Value of the `id` attribute | +| `test_case_name` | string | Value of the `name` attribute | +| `issues` | array | Schema issues with `rule_id`, `severity`, `message` | +| `best_practices_violations` | array | Best-practices violations with `rule_id`, `severity`, `weight`, `message` | +| `best_practices_rules_evaluated` | integer | How many best-practices rules were checked | +| `validation_source` | string | `quality_hub`, `local`, or `local_fallback` — see Authentication section | +| `validation_warning` | string | Present when `validation_source` is `local` (onboarding) or `local_fallback` (explains why API failed) | **Key schema rules:** TC_001 (missing XML declaration), TC_002 (malformed XML), TC_003 (wrong root element), TC_010/011/012 (missing/invalid id/guid), TC_031 (invalid apiCall guid), TC_034/035 (non-integer testItemId). +**Validity bridge.** A `critical` best-practice violation means "Provar will not load/render the test case" (e.g. an unknown/hallucinated `apiId`, a missing required control or connection argument, an invalid render value-class). These now gate `is_valid` the same way a structural schema error does — each is surfaced into `issues[]` as an `ERROR` and flips `is_valid` to `false`. `major` (runtime ERROR), `minor` (warning), and `info` violations do **not** flip `is_valid`; they affect `quality_score` and therefore the `needs_improvement` verdict. Best-practice rules covering concepts the schema rules already own (root element, identifier, steps presence, `testItemId` integers, comparisonType enums) are intentionally **not** double-reported — the hand-coded schema rule is authoritative there. + **Warning rules:** - **DATA-001** — `testCase` declares a `` element. When the validator is called with `file_path` and the project's `provardx-properties.json` references that test case directly via top-level `testCase` or `testCases` (rather than via a `.testinstance` inside a plan), the warning carries the `WARNING [DATA-001]:` prefix and recommends wiring the test into a plan via `provar_testplan_add-instance`. When `file_path` is not supplied (or the project context cannot be resolved), the warning falls back to a structural advisory recommending `SetValues` (Test scope) steps. The warning is suppressed entirely when a `.testinstance` references the test case, because data-driven iteration works in that mode. See also [Data-driven execution](#data-driven-execution). - **ASSERT-001** — An `AssertValues` step uses the `argument id="values"` (namedValues) format, which is designed for UI element attribute assertions. For Apex/SOQL result or variable comparisons this silently passes as `null=null`. Use separate `expectedValue`, `actualValue`, and `comparisonType` arguments instead. - **UI-TARGET-001** — A UiWithScreen or UiWithRow `target` argument uses the wrong XML class (e.g. `class="value"`). Must be `class="uiTarget"` or the screen binding is silently ignored at runtime. - **UI-LOCATOR-001** — A locator-bearing UI step (`UiDoAction`, `UiAssert`, `UiRead`, `UiFill`) has a `locator` argument that uses the wrong XML class. Must be `class="uiLocator"` or Provar cannot resolve the element. +- **UI-INTERACTION-001** (ERROR) — A UI action step (e.g. `UiDoAction`) has an `interaction` argument that uses the wrong XML class (e.g. `class="value"`). Must be `class="uiInteraction"` (``). A plain string runs green from the CLI but renders the Action field blank in the Provar IDE step editor. The `interaction` attribute is converted automatically by `provar_testcase_generate`. +- **UI-ASSERT-STRUCTURE-001** (ERROR) — A `UiAssert` step carries a flat top-level field-assertion argument (`fieldLocator`, `attributeName`, `comparisonType`, or `expectedValue`) instead of the nested `fieldAssertions` → `uiFieldAssertion` structure the Provar IDE Result Assertions tab binds from. The flat shape runs green from the CLI but renders the Result Assertions tab blank in the IDE. The correct form nests a `` containing a **bare** `` element (NOT `class="uiLocator"`) and ``, plus empty `columnAssertions`/`pageAssertions`. `provar_testcase_generate` builds this structure automatically when you pass `fieldLocator`/`attributeName`/`comparisonType`/`expectedValue` as flat attributes. - **SETVALUES-STRUCTURE-001** (ERROR) — A `SetValues` step's `values` argument uses `class="value"` (plain string) instead of `class="valueList"` with `` children. This causes an immediate `ClassCastException` at runtime. - **COMPARISON-TYPE-001** (ERROR) — A `comparisonType` value is used outside the subset its step type allows. `comparisonType` is a single Provar enum but each step type accepts only a subset: **AssertValues** accepts the 16-value set (`EqualTo, NotEqualTo, GreaterThan, GreaterThanOrEqualTo, LessThan, LessThanOrEqualTo, IsPresent, IsEmpty, Matches, NotMatches, Contains, NotContains, StartsWith, NotStartsWith, EndsWith, NotEndsWith`); a **UI Assert** (`uiAttributeAssertion`) accepts only the 6-value set (`EqualTo, Contains, StartsWith, EndsWith, Matches, None`). A value outside the step's subset (e.g. `NotEqualTo` on a UI Assert) is load-blocking — the whole test case fails to load at runtime with `IllegalArgumentException: No enum constant com.provar.core.model.base.java.ComparisonType.`. This local check runs even offline / in `local_fallback`, so the load-blocker is caught without the Quality Hub back-end. Only literal `comparisonType` values are checked; variable / compound references are skipped. See [`provar://docs/step-reference`](#resources) for the full step-scoped tables. - **UI-NEST-STRUCT-001** (severity `major`, weight 7, category `XMLSchema`) — A UI action step (`UiDoAction`, `UiAssert`, `UiRead`, `UiFill`, `UiNavigate`, `UiWithRow`, or `UiHandleAlert`) is emitted outside a screen ancestor. To pass, every UI action must descend from a `UiWithScreen` or `UiWithRow` `apiCall` through a `` path. Control-flow wrappers (`If`/`ForEach`/`DoWhile`/`WaitFor`/`Switch`) between the screen ancestor and the UI action are allowed; steps inside `` are exempt (disabled / settings blocks). One violation is emitted per offending step, so `(rule_id, test_item_id)` de-duplicates cleanly against the Quality Hub API. Provar IDE cannot bind flat-emitted UI actions to a screen context and they will not render in the editor. Wrap each offending step in the canonical chain: @@ -1066,6 +1078,39 @@ Validates an XML test case for schema correctness (validity score) and best prac `UiWithRow` plays a dual role: it is itself a UI action that must be nested, and a container whose `` satisfies the rule for its descendants. Mirrors Quality Hub's `UiActionNestingStructureValidator`. - **VAR-REF-001** — An argument value looks like a variable reference (`{VarName}` or `{Obj.Field}`) but is stored as `class="value" valueClass="string"`. Provar will treat it as a literal string, not resolve the variable. Replace with `class="variable"` and `` elements. - **VAR-REF-002** — A `{VarName}` token is embedded inside a larger plain string (e.g. `SELECT Id FROM Account WHERE Id = '{AccountId}'`). Provar does not perform `{…}` interpolation in string values at runtime; the braces are emitted literally. Use `class="compound"` with `` children to split the literal text and variable references. In `provar_testcase_generate`, pass the value with `{VarName}` placeholders — the generator emits compound XML automatically. +- **Required-argument rules** (`mustContainArgument`) — A step type that requires a named argument is flagged when that argument is absent **or present but empty** (a self-closing ``, an empty ``, or a bare `` with no ``). A value counts as present when it has text, a `funcCall`, a comparison/logic operator (`gt`/`lt`/`eq`/…), a `variable` with a ``, or a `compound` with ``. Each rule pins one `(step apiId, required argument)` pair; apiId matching is exact, steps nested inside control-flow containers are checked, and disabled steps are **not** exempt (a missing required argument is load/exec-blocking regardless). `If`/`DoWhile` steps that carry the condition in the step `title` (legacy `If: {expr}` format) instead of a `condition` argument are accepted. One violation per rule (the message names every offending step) — matching the Quality Hub back-end, so the weighted-deduction score stays in parity with the Lambda. These rules run offline / in `local_fallback` so the missing argument is caught without the back-end. Currently enforced: + - **Control flow** — `CONTROL-IF-001` (`If` → `condition`), `CONTROL-WHILE-001` (`DoWhile` → `condition`), `CONTROL-WAITFOR-001`/`-002` (`WaitFor` → `condition` / `maxIterations`), `CONTROL-FOREACH-001`/`-002` (`ForEach` → `list` / `valueName`), `CONTROL-SWITCH-001` (`Switch` → `value`), `CONTROL-SLEEP-002` (`Sleep` → `sleepSecs`). + - **Assertions** — `ASSERT-COMPARISON-001` (`AssertValues` → `comparisonType`), `ASSERT-EXPECTED-001` (`AssertValues` → `expectedValue`). + - **BDD** — `BDD-GIVEN-001` / `BDD-WHEN-001` / `BDD-THEN-001` (`Given`/`When`/`Then` → `description`). + - **Apex / database** — `SOQL-QUERY-001` (`ApexSoqlQuery` → `soqlQuery`), `DB-CONNECT-001`/`-002` (`DbConnect` → `connectionName` / `resultName`), `SQL-QUERY-001`/`-002` (`SqlQuery` → `query` / `dbConnectionName`). + - **Web service** — `REST-CONN-001`/`-002` (`WebConnect` → `connectionName` / `resultName`), `REST-REQUEST-001` (`RestRequest` → `connectionName`). +- **Render / load-blocking rules** — Structural checks that catch XML which prevents a test case from rendering or loading in the Provar IDE. They run offline / in `local_fallback` and emit one violation per rule (the message names the first offender; `count` is set only when more than one element offends), matching the Quality Hub back-end so the weighted-deduction score stays in parity with the Lambda. Currently enforced: + - **`RENDER-CASE-001`** (`valueClassCasing`, critical) — a `valueClass` spelled with the wrong case (e.g. `Boolean` instead of `boolean`). Only a known class with bad casing fires; an unknown class does not. + - **`RENDER-BOOL-001`** (`booleanCasing`, critical) — a `` whose text is `True`/`False`/`TRUE`/`FALSE`; it must be lowercase `true`/`false`. + - **`VALUE-CLASS-001`** (`invalidValueClass`, critical) — a `` whose `class` is not a recognised Provar value class (e.g. the hallucinated `class="null"`), or a `class="value"` whose `valueClass` is not one of `string`/`boolean`/`decimal`/`id`/`date`/`dateTime`. Empty/optional arguments must omit the `` entirely. + - **`RENDER-DATE-VALUECLASS-001`** (`dateValueClassFormat`, critical) — a `valueClass="date"`/`"dateTime"` value whose text is an ISO date string (e.g. `2025-01-15`) instead of an epoch-milliseconds integer; store dates as `valueClass="string"` or convert to epoch millis. + - **`SETVALUES-INVALID-ELEMENT-001`** (`setValuesInvalidElements`, critical) — a `SetValues` step containing hallucinated `` or `` elements, or a `` child other than ``. The correct shape is `` with `` children. + - **`APEX-CONNECT-ARGS-001`** (`apexConnectValidArguments`, critical) — an `ApexConnect` step using an argument id outside the 20-id whitelist (overridable per rule via `check.validArgumentIds`). + - **`APEX-CONNECT-CONNID-001`** (`apexConnectConnectionIdValueClass`, critical) — an `ApexConnect` `connectionId` value that uses `valueClass="string"` (or any non-`id` class) instead of `valueClass="id"`; leave it empty if there is no GUID. + - **`APEX-REUSE-CONN-001`** (`apexConnectReuseConnection`, major) — an `ApexConnect` `reuseConnectionName` argument that carries a value; it must be left blank. +- **Runtime anti-pattern rules** — Major (`weight 5`) checks for XML that loads but silently misbehaves at runtime — typically variable references or expressions written as plain string literals. Like the rules above they run offline / in `local_fallback` and keep score parity with the Quality Hub back-end. Currently enforced: + - **`VAR-STRING-LITERAL-001`** (`varStringLiteral`) — an argument value is a whole-token `{VarName}` / `{Obj.Field}` stored as `class="value" valueClass="string"`; Provar passes the literal text instead of resolving the variable. Use ``. (The interpolation-tolerant target args `sfUiTargetObjectId` / `sfUiTargetResultName` are exempt.) Emits one violation per offending value. + - **`ASSERT-STR-VAR-001`** (`assertValuesStringExpr`) — an `AssertValues` `expectedValue`/`actualValue` is a whole `{…}` string literal, so the assertion compares the literal text instead of the variable's value. Use `class="variable"`. + - **`SETVALUES-FUNC-STR-001`** (`setValuesFuncCallString`) — a `SetValues` assigned value embeds a `{Func(args)}` call as a string literal; Provar will not evaluate it. Use a `class="funcCall"` value. + - **`SETVALUES-ZERO-IDX-001`** (`setValuesZeroIndexString`) — a `SetValues` string-template expression uses a `[0]` index; Provar string templates are 1-indexed, so this is out-of-bounds at runtime. Use `[1]` for the first item (or the XML variable-path structure). + - **`CONN-DB-002`** (`dbConnectResultNameMismatch`) — a DB operation (`SqlQuery`/`DbRead`/`DbInsert`/`DbUpdate`/`DbDelete`) sets a `dbConnectionName` that does not match any `DbConnect` `resultName` in the test, so the open connection can't be found. Defers to `CONN-DB-001` when there is no `DbConnect` at all. + - **`UI-LOCATOR-BUTTON-CASING-001`** (`uiLocatorButtonCasing`) — a `UiDoAction` locator uses a wrong-cased standard-button name: `name=Cancel` (use lowercase `name=cancel`) or `name=Continue`/`name=continue` on the record-type screen (use `name=save&path=selectRecordType`). + - **`UI-LOCATOR-RECORDTYPE-001`** (`uiLocatorRecordTypeField`) — a `UiDoAction` Record Type picker locator uses `name=recordTypeId`/`name=recordType` instead of `name=RecordType` with `field=RecordTypeId` in the binding. +- **Structural correctness rules** — Checks on element shape / required structure that the Provar IDE depends on to render and run a step. Mostly critical; like the rules above they run offline / in `local_fallback` and keep score parity with the Quality Hub back-end. Currently enforced: + - **`SETVALUES-STRUCTURE-001`** (`setValuesStructure`, critical) — a `SetValues` step has no `` container. + - **`SETVALUES-NAME-001`** (`namedValueName`, critical) — a `` inside a `SetValues` step is missing its `name` attribute. + - **`SETVALUES-VALUE-001`** (`namedValueValue`, critical) — a `` inside a `SetValues` step is missing its child `` element. + - **`UI-ASSERT-STRUCT-002`** (`uiAssertHallucinatedGeneratedParameters`, critical) — a `UiAssert` step contains a `` element, which is never valid on `UiAssert` and blocks validation. + - **`UI-ASSERT-STRUCT-001`** (`uiAssertMissingArguments`, critical) — a `UiAssert` step is missing one or more required arguments (`fieldAssertions`, `columnAssertions`, `pageAssertions`, `resultScope`, `captureAfter`, `beforeWait`, `autoRetry` — present even if empty). + - **`UI-BINDING-ORDER-001`** (`bindingParameterOrder`, critical) — a `uiLocator` binding URI lists `action=`/`field=` before `object=`; the corpus-majority convention is `object=` first. + - **`UI-CONN-LITERAL-001`** (`uiConnectionNameLiteral`, critical) — a UI step's `uiConnectionName` uses a `class="variable"` value; it must be a literal connection name. + - **`FUNCCALL-VALID-001`** (`validFuncCallId`, major) — a `` uses an `id` that is not one of Provar's built-in functions (e.g. the hallucinated `Concatenate`/`Substring`); use the documented set (`Count`, `DateAdd`, `StringReplace`, …) or `class="compound"` for concatenation. + - **`RENDER-ROOT-001`** (`rootAttributes`, minor) — the root `` element carries an attribute outside the allowed set (`guid`, `id`, `name`, `visibility`, `registryId`, `failureBehaviour`). **Error codes** @@ -1084,15 +1129,15 @@ Validates a Provar test suite — checks for empty suites, duplicate names (with **Input** -| Parameter | Type | Required | Description | -| ------------------- | --------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | -| `suite_name` | string | yes | Name of the test suite | -| `test_cases` | array | no | Test cases directly in this suite. Each item: `{ name, xml_content \| xml }` | -| `child_suites` | array | no | Child suites (up to 2 levels of nesting). Each item: `{ name, test_cases?, test_suites?, test_case_count? }` | -| `test_case_count` | integer | no | Override total count for the size check (useful when not sending full XML) | -| `quality_threshold` | number (0–100) | no | Minimum quality score for a test case to be "valid" (default: 80) | -| `detail` | `summary` \| `standard` \| `full` | no | Response verbosity. `"summary"`: name, scores, and stop signal only. `"standard"`/`"full"`: full violations and per-test-case results (default). | -| `baseline_run_id` | string | no | `run_id` from a previous call. Returns only new/resolved violations since that run. Returns `BASELINE_NOT_FOUND` if the run ID is unknown. | +| Parameter | Type | Required | Description | +| ------------------- | --------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `suite_name` | string | yes | Name of the test suite | +| `test_cases` | array | no | Test cases directly in this suite. Each item: `{ name, xml_content \| xml }` | +| `child_suites` | array | no | Child suites (up to 2 levels of nesting). Each item: `{ name, test_cases?, test_suites?, test_case_count? }` | +| `test_case_count` | integer | no | Override total count for the size check (useful when not sending full XML) | +| `quality_threshold` | number (0–100) | no | Minimum quality score for a test case to be `valid` rather than `needs_improvement`. Precedence: this arg → `PROVAR_MCP_QUALITY_THRESHOLD` env → `90`. | +| `detail` | `summary` \| `standard` \| `full` | no | Response verbosity. `"summary"`: name, scores, and stop signal only. `"standard"`/`"full"`: full violations and per-test-case results (default). | +| `baseline_run_id` | string | no | `run_id` from a previous call. Returns only new/resolved violations since that run. Returns `BASELINE_NOT_FOUND` if the run ID is unknown. | **Output** — `{ run_id, completeness_score, recommended_next_action, name, level: "suite", quality_score, violations[], test_cases[], test_suites[], summary }` @@ -1119,7 +1164,7 @@ Validates a Provar test plan — checks for empty plans, duplicate suite names, | `test_cases` | array | no | Test cases directly in this plan | | `test_suite_count` | integer | no | Override suite count for the size check | | `metadata` | object | no | Plan completeness metadata (see below) | -| `quality_threshold` | number (0–100) | no | Minimum quality score (default: 80) | +| `quality_threshold` | number (0–100) | no | Minimum quality score for `valid` vs `needs_improvement`. Precedence: this arg → `PROVAR_MCP_QUALITY_THRESHOLD` env → `90`. | | `detail` | `summary` \| `standard` \| `full` | no | Response verbosity. `"summary"`: name, scores, and stop signal only. `"standard"`/`"full"`: full violations and hierarchy results (default). | **`metadata` fields** @@ -1157,7 +1202,7 @@ Validates a Provar project directly from its directory on disk. Reads the plan/s | Parameter | Type | Required | Description | | ---------------------- | --------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `project_path` | string | yes | Absolute path to the Provar project root (directory containing `.testproject`) | -| `quality_threshold` | number (0–100) | no | Minimum quality score for a test case to be considered valid (default: 80) | +| `quality_threshold` | number (0–100) | no | Minimum quality score for a test case to be considered valid. Precedence: this arg → `PROVAR_MCP_QUALITY_THRESHOLD` env → `90`. | | `save_results` | boolean | no | Write a QH-compatible JSON report to `{project_path}/provardx/validation/` (default: true) | | `results_dir` | string | no | Override the output directory for the saved report (must be within `allowed-paths`) | | `detail` | `summary` \| `standard` \| `full` | no | Response verbosity. `"summary"`: key scores and stop signal only. `"standard"`: slim violation summary (default). `"full"`: full per-suite and per-test-case data. | @@ -1962,7 +2007,7 @@ This produces silent-pass behaviour that is hard to spot from a log: the run exi The plan-mode resolver consults the properties file registered by [`provar_automation_config_load`](#provar_automation_config_load) (`PROVARDX_PROPERTIES_FILE_PATH` in `~/.sf/config.json`), reads `projectPath`, then: 1. Walks `/plans/**/*.testinstance` for any `testCasePath="..."` referencing the test under validation. If found → `plan` mode → DATA-001 suppressed. -2. Otherwise checks `testCase` / `testCases` for a direct reference. If found → `direct` mode → DATA-001 with the PDX-489 advisory. +2. Otherwise checks `testCase` / `testCases` for a direct reference. If found → `direct` mode → DATA-001 with the direct-mode advisory. 3. Falls back to `unknown` mode when no project context is resolvable — DATA-001 still fires (structural fallback) so authors editing a test case in isolation are still warned. **Recommended workaround** @@ -2509,6 +2554,41 @@ The resource content is the same as `docs/PROVAR_TEST_STEP_REFERENCE.md` in this --- +### `provar://schema/test-step` + +Structured JSON reference describing the full Provar test case XML structure: the `` root, the generic `` shape, and every supported step type organised by category (Control, Data, Design, ProvarAI, ProvarLabs, Salesforce, UI, Utility) with its required/optional arguments and validation rules, plus the value-class types and common patterns. Where `provar://docs/step-reference` is the prose reference, this resource is the structured contract a client can parse to drive generation or validation programmatically. + +This is a **Provar-specific schema reference** — its top-level keys (`testCase`, `apiCall`, `apiCalls`, `value_types`, `common_patterns`) are Provar domain entities, not JSON-Schema keywords. Although the file carries a `$schema: draft-07` declaration inherited from its source, it is **not** a standards-compliant constraint JSON Schema; do not load it into a JSON-Schema validator expecting it to constrain documents. + +**URI:** `provar://schema/test-step` +**MIME type:** `application/json` + +The resource content is the bundled `src/mcp/rules/provar_test_step_schema.json`, compiled into the package at build time. It is the same schema the local best-practices validator's API-ID and value-class checks are derived from, so step structures that satisfy it are consistent with what `provar_testcase_validate` enforces. The handler parses the file once to confirm it is valid JSON before serving it; if the file is missing or unparseable, the resource returns a small `{ "error": "schema_not_found", "message": … }` object instead. + +--- + +### `provar://docs/validation-rules` + +The single canonical registry of every Provar test-case validation rule across both layers — the structural validity rules (**Layer 1**, hand-coded, gate `is_valid`) and the best-practice rules (**Layer 2**, the 178-rule engine, weighted `quality_score`). For each rule it lists the id, severity, weight, what it checks, and **whether it gates `is_valid`**. A `critical` best-practice violation gates `is_valid` via the validity bridge (except where a Layer-1 check already owns the concept); `major`/`minor`/`info` affect `quality_score` (and the `needs_improvement` status) only. Read this to understand why `provar_testcase_validate` returned a given issue, or why it marked a test `invalid` vs `needs_improvement`. + +**URI:** `provar://docs/validation-rules` +**MIME type:** `text/markdown` + +The resource content is `docs/VALIDATION_RULE_REGISTRY.md`, generated from the rule sources by `scripts/build-validation-rule-registry.cjs` and compiled into the package at build time. Re-run that script after changing any rule; a unit test guards the registry against drift. + +--- + +### `provar://docs/tool-guide` + +Tool-selection guide for the Provar MCP server, organised by what you want to accomplish (run tests, author tests, debug failures, manage config, …) rather than by tool name. Read this to choose the right tool and understand correct sequencing — e.g. which prerequisite tool must run before another — before making calls. + +**URI:** `provar://docs/tool-guide` +**MIME type:** `text/markdown` + +The resource content is the same as `docs/PROVAR_TOOL_GUIDE.md` in this repository, compiled into the package at build time. If the file is missing, the resource returns a short placeholder telling the client to reinstall or upgrade the plugin. + +--- + ### `provar://nitrox/component-catalog` Catalog of all shipped NitroX (Hybrid Model) base component packages. Lists every package with its components, types, tagNames, interactions, and attributes. Read this before calling `provar_nitrox_generate` to understand available component patterns and naming conventions. diff --git a/messages/sf.provar.automation.project.validate.md b/messages/sf.provar.automation.project.validate.md index 2a868d38..c3695df2 100644 --- a/messages/sf.provar.automation.project.validate.md +++ b/messages/sf.provar.automation.project.validate.md @@ -15,7 +15,7 @@ Path to the Provar project root (directory containing the .testproject file). De # flags.quality-threshold.summary -Minimum quality score (0-100) for a test case to pass validation (default: 80). +Minimum quality score (0-100) for a test case to pass validation (default: 90). # flags.save-results.summary diff --git a/messages/sf.provar.mcp.start.md b/messages/sf.provar.mcp.start.md index 4cf9c79d..8aae7c53 100644 --- a/messages/sf.provar.mcp.start.md +++ b/messages/sf.provar.mcp.start.md @@ -16,6 +16,7 @@ Project & inspection: - provar_project_inspect — inspect project folder inventory - provar_project_validate — validate full project from disk: coverage, quality scores - provar_connection_list — list connections and named environments from the project +- provar_org_describe — describe Salesforce objects from the Provar workspace .metadata cache Page Object: diff --git a/package.json b/package.json index c780d792..aa4b5cf4 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "@provartesting/provardx-cli", "description": "A plugin for the Salesforce CLI to orchestrate testing activities and report quality metrics to Provar Quality Hub", - "version": "1.5.4", + "version": "1.6.0", "mcpName": "io.github.ProvarTesting/provar", "license": "BSD-3-Clause", "plugins": [ @@ -152,7 +152,7 @@ ] }, "compile": { - "command": "tsc -p . --pretty --incremental && shx mkdir -p lib/mcp/rules && shx cp src/mcp/rules/*.json lib/mcp/rules/ && shx mkdir -p lib/mcp/docs && shx cp docs/PROVAR_TEST_STEP_REFERENCE.md lib/mcp/docs/ && shx cp docs/NITROX_COMPONENT_CATALOG.md lib/mcp/docs/ && shx cp docs/NITROX_CATALOG_SOURCE.json lib/mcp/docs/ && shx cp docs/PROVAR_TOOL_GUIDE.md lib/mcp/docs/", + "command": "tsc -p . --pretty --incremental && shx mkdir -p lib/mcp/rules && shx cp src/mcp/rules/*.json lib/mcp/rules/ && shx mkdir -p lib/mcp/docs && shx cp docs/PROVAR_TEST_STEP_REFERENCE.md lib/mcp/docs/ && shx cp docs/NITROX_COMPONENT_CATALOG.md lib/mcp/docs/ && shx cp docs/NITROX_CATALOG_SOURCE.json lib/mcp/docs/ && shx cp docs/PROVAR_TOOL_GUIDE.md lib/mcp/docs/ && shx cp docs/VALIDATION_RULE_REGISTRY.md lib/mcp/docs/", "files": [ "src/**/*.ts", "src/mcp/rules/*.json", @@ -160,6 +160,7 @@ "docs/NITROX_CATALOG_SOURCE.json", "docs/PROVAR_TOOL_GUIDE.md", "docs/PROVAR_TEST_STEP_REFERENCE.md", + "docs/VALIDATION_RULE_REGISTRY.md", "**/tsconfig.json", "messages/**" ], diff --git a/scripts/build-validation-rule-registry.cjs b/scripts/build-validation-rule-registry.cjs new file mode 100644 index 00000000..afd8a30d --- /dev/null +++ b/scripts/build-validation-rule-registry.cjs @@ -0,0 +1,122 @@ +#!/usr/bin/env node +/* + * Copyright (c) 2024 Provar Limited. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.md file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ + +/* + * Generates the canonical Validation Rule Registry (docs/VALIDATION_RULE_REGISTRY.md) + * from the two rule sources: + * - Layer 1 (structural validity): metadata catalog in + * src/mcp/rules/provar_layer1_rules.json — the single source of truth shared + * with testCaseValidate.ts (detection stays in code; only metadata is data). + * - Layer 2 (best practices): src/mcp/rules/provar_best_practices_rules.json. + * + * "Gates is_valid?" reflects the PDX-509 model: + * - Layer-1 ERROR → YES (error_count gates is_valid) + * - Layer-1 WARNING/INFO → no (advisory / quality only) + * - Layer-2 critical → YES via the validity bridge, UNLESS the concept is already + * owned by a Layer-1 check (then suppressed to avoid double-reporting) + * - Layer-2 major/minor/info → no (quality_score only) + * + * Re-run after changing any rule: node scripts/build-validation-rule-registry.cjs + */ + +const fs = require('node:fs'); +const path = require('node:path'); + +const ROOT = path.resolve(__dirname, '..'); +const OUT = path.join(ROOT, 'docs', 'VALIDATION_RULE_REGISTRY.md'); +const BP_JSON = path.join(ROOT, 'src', 'mcp', 'rules', 'provar_best_practices_rules.json'); +const LAYER1_JSON = path.join(ROOT, 'src', 'mcp', 'rules', 'provar_layer1_rules.json'); + +// Layer-1 structural-validity catalog (id, severity, applies_to, description, +// owns_bp_rules), read from the shared single source of truth. The same JSON is +// consumed by testCaseValidate.ts (owned-set) and validationRuleRegistry.test.ts +// (drift guard). Detection stays in code; only this metadata is centralized. +const LAYER1 = JSON.parse(fs.readFileSync(LAYER1_JSON, 'utf8')).rules; + +// Layer-2 critical rules whose concept a Layer-1 check already owns — derived +// from the catalog's `owns_bp_rules` (mirrors LAYER1_OWNED_BP_RULES in +// testCaseValidate.ts, same source). These criticals are NOT bridged. +const LAYER1_OWNED = new Set(LAYER1.flatMap((r) => r.owns_bp_rules || [])); + +function gatesLayer1(sev) { + return sev === 'ERROR' ? 'Yes' : 'No'; +} +function gatesLayer2(rule) { + return rule.severity === 'critical' && !LAYER1_OWNED.has(rule.id) ? 'Yes' : 'No'; +} + +function esc(s) { + return String(s == null ? '' : s) + .replace(/\|/g, '\\|') + .replace(/\s+/g, ' ') + .trim(); +} + +const bp = JSON.parse(fs.readFileSync(BP_JSON, 'utf8')); +const rules = bp.rules.slice().sort((a, b) => (a.category + a.id).localeCompare(b.category + b.id)); + +const sev = { critical: 0, major: 0, minor: 0, info: 0 }; +for (const r of rules) sev[r.severity] = (sev[r.severity] || 0) + 1; +const bpGating = rules.filter((r) => gatesLayer2(r) === 'Yes').length; + +const lines = []; +lines.push('# Provar Validation Rule Registry'); +lines.push(''); +lines.push( + '> **Generated** by `scripts/build-validation-rule-registry.cjs`. Do not edit by hand — re-run the script after changing a rule.' +); +lines.push(''); +lines.push( + 'Provar test-case validation runs in two layers. This registry is the single canonical list of every rule across both.' +); +lines.push(''); +lines.push( + '- **Layer 1 — structural validity** (hand-coded in `testCaseValidate.ts`): emits `issues[]` with `ERROR`/`WARNING`. `is_valid = error_count === 0`.' +); +lines.push( + '- **Layer 2 — best practices** (`provar_best_practices_rules.json`, same engine/weights as the Quality Hub API): emits `best_practices_violations[]` with `critical`/`major`/`minor`/`info` and a weighted `quality_score`.' +); +lines.push(''); +lines.push( + '**Severity taxonomy:** `critical` = the test will not load/render in Provar; `major` = a runtime ERROR (loads, fails at execution); `minor` = warning; `info` = advisory.' +); +lines.push(''); +lines.push( + '**The validity bridge (PDX-509):** a `critical` best-practice violation is surfaced into `issues[]` as an `ERROR` and therefore gates `is_valid` — EXCEPT where a Layer-1 check already owns the concept (then it is suppressed to avoid double-reporting). `major`/`minor`/`info` affect `quality_score` (and the `needs_improvement` status) only. The `status` field is tri-state: `invalid` (a critical) / `needs_improvement` (loads but `quality_score < quality_threshold`) / `valid`.' +); +lines.push(''); +lines.push( + `**Counts:** Layer 1 — ${LAYER1.length} rules (${ + LAYER1.filter((r) => r.severity === 'ERROR').length + } gating). Layer 2 — ${rules.length} rules (critical ${sev.critical} / major ${sev.major} / minor ${ + sev.minor + } / info ${sev.info}; ${bpGating} bridged to \`is_valid\`).` +); +lines.push(''); +lines.push('## Layer 1 — Structural validity rules'); +lines.push(''); +lines.push('| Rule ID | Severity | Gates is_valid? | Applies to | Checks |'); +lines.push('| ------- | -------- | --------------- | ---------- | ------ |'); +for (const r of LAYER1) { + lines.push(`| \`${r.id}\` | ${r.severity} | ${gatesLayer1(r.severity)} | ${r.applies_to} | ${esc(r.description)} |`); +} +lines.push(''); +lines.push('## Layer 2 — Best-practice rules'); +lines.push(''); +lines.push('| Rule ID | Category | Severity | Weight | Gates is_valid? | Checks |'); +lines.push('| ------- | -------- | -------- | ------ | --------------- | ------ |'); +for (const r of rules) { + const note = r.severity === 'critical' && LAYER1_OWNED.has(r.id) ? ' _(Layer-1 owns this concept; not bridged)_' : ''; + lines.push( + `| \`${r.id}\` | ${esc(r.category)} | ${r.severity} | ${r.weight} | ${gatesLayer2(r)} | ${esc(r.name)}.${note} |` + ); +} +lines.push(''); + +fs.writeFileSync(OUT, lines.join('\n'), 'utf8'); +process.stdout.write(`Wrote ${OUT} (${LAYER1.length} Layer-1 + ${rules.length} Layer-2 rules)\n`); diff --git a/scripts/mcp-smoke.cjs b/scripts/mcp-smoke.cjs index d1ccc833..467df368 100644 --- a/scripts/mcp-smoke.cjs +++ b/scripts/mcp-smoke.cjs @@ -209,6 +209,33 @@ async function runTests() { '', }); + // ── 7c. provar_testcase_validate — UI-INTERACTION-001 (typed Action value) ── + // Drives the UI-INTERACTION-001 ERROR path: a UiDoAction whose `interaction` + // argument is a plain string runs green from the CLI but renders the IDE Action + // widget blank. The smoke framework counts any JSON-RPC response as PASS; this + // keeps the ERROR-tier code path exercised on every run. + if (inGroup('validation')) + await callTool('provar_testcase_validate', { + content: + '' + + '' + + 'ui:interaction?name=click' + + '', + }); + + // ── 7d. provar_testcase_validate — UI-ASSERT-STRUCTURE-001 (nested asserts) ─ + // Drives the UI-ASSERT-STRUCTURE-001 ERROR path: a UiAssert with a flat + // top-level fieldLocator argument runs green from the CLI but renders the IDE + // Result Assertions tab blank. Keeps the ERROR-tier code path exercised. + if (inGroup('validation')) + await callTool('provar_testcase_validate', { + content: + '' + + '' + + 'ui:locator?name=Priority' + + '', + }); + // ── 8. provar_testsuite_validate ────────────────────────────────────────── if (inGroup('validation')) await callTool('provar_testsuite_validate', { suite_name: 'SmokeTestSuite' }); diff --git a/server.json b/server.json index 0ebe3b05..88e771f5 100644 --- a/server.json +++ b/server.json @@ -14,12 +14,12 @@ "url": "https://github.com/ProvarTesting/provardx-cli", "source": "github" }, - "version": "1.5.4", + "version": "1.6.0", "packages": [ { "registryType": "npm", "identifier": "@provartesting/provardx-cli", - "version": "1.5.4", + "version": "1.6.0", "transport": { "type": "stdio" }, diff --git a/src/commands/provar/automation/project/validate.ts b/src/commands/provar/automation/project/validate.ts index 4db13bbf..eb02b173 100644 --- a/src/commands/provar/automation/project/validate.ts +++ b/src/commands/provar/automation/project/validate.ts @@ -8,7 +8,11 @@ /* eslint-disable camelcase */ import { SfCommand, Flags } from '@salesforce/sf-plugins-core'; import { Messages } from '@salesforce/core'; -import { validateProjectFromPath, ProjectValidationError, type ProjectValidationResult } from '../../../../services/projectValidation.js'; +import { + validateProjectFromPath, + ProjectValidationError, + type ProjectValidationResult, +} from '../../../../services/projectValidation.js'; Messages.importMessagesDirectoryFromMetaUrl(import.meta.url); const messages = Messages.loadMessages('@provartesting/provardx-cli', 'sf.provar.automation.project.validate'); @@ -27,7 +31,7 @@ export default class SfProvarAutomationProjectValidate extends SfCommand. Use dotted paths for nested references: . If using provar.testcase.generate, the {VarName} syntax in attributes is converted automatically.", + "check": { + "type": "varStringLiteral" + }, + "source": "Provar Test Step Schema: variable references must use class=\"variable\", not string literals" + }, + { + "id": "CONN-DB-002", + "category": "ConnectionsAndEnvironments", + "name": "DbConnect resultName must match dbConnectionName in DB operations", + "description": "The resultName set on a DbConnect step is the connection handle variable that all subsequent DB operations (SqlQuery, DbRead, DbInsert, DbUpdate, DbDelete) must reference via their dbConnectionName argument. A mismatch between DbConnect resultName and dbConnectionName causes a runtime failure because the operation cannot find the open connection. This is the leading cause of silent Database step failures in Provar test suites.", + "appliesTo": ["Step"], + "severity": "major", + "weight": 5, + "recommendation": "Ensure every DB operation step's dbConnectionName argument exactly matches the resultName set on the preceding DbConnect step. For example, if DbConnect sets resultName='SQLServer', every SqlQuery/DbRead/DbInsert/DbUpdate/DbDelete in the same test must set dbConnectionName='SQLServer'.", + "check": { + "type": "dbConnectResultNameMismatch", + "target": "testCase" + }, + "source": "Provar DB Connection Best Practices - DbConnect resultName / dbConnectionName alignment" + }, + { + "id": "SETVALUES-FUNC-STR-001", + "category": "StructureAndGrouping", + "name": "SetValues must not use string interpolation for function calls", + "description": "SetValues steps must use the correct XML structure for function calls () rather than embedding Provar's curly-brace template syntax as a plain string literal (e.g. {Count(AccountList)}). At runtime Provar reads the string literally and does not evaluate it, so the target variable is set to the text \"{Count(AccountList)}\" instead of the computed count. This pattern is generated by AI models that reproduce the Provar UI tooltip syntax verbatim.", + "appliesTo": ["Step"], + "severity": "major", + "weight": 5, + "recommendation": "Replace the string literal with the proper funcCall XML: . Supported functions include Count, Sum, Avg, Min, Max, Concat, StringLength, etc.", + "check": { + "type": "setValuesFuncCallString", + "target": "step" + }, + "notes": "AI models output {Count(var)} in a valueClass=string element because they mirror Provar's UI tooltip. The correct form is a funcCall value element. This causes silent test failures where the variable receives the literal string rather than the computed value.", + "source": "Field observation: SetValues function-call string interpolation anti-pattern" + }, + { + "id": "SETVALUES-ZERO-IDX-001", + "category": "StructureAndGrouping", + "name": "SetValues string expression must not use [0] index", + "description": "Provar's string-template engine is 1-indexed: {List[1]} accesses the first element. Using {List[0].Field} in a valueClass=string expression causes an out-of-bounds exception at runtime because index 0 is invalid in the template system. Note: the XML variable-path structure (0) correctly uses 0-based indexing via a different code path — this rule targets only the broken string-template form.", + "appliesTo": ["Step"], + "severity": "major", + "weight": 5, + "recommendation": "Either switch to the proper XML variable path structure (0) or, if you must use string interpolation, change [0] to [1] to access the first element.", + "check": { + "type": "setValuesZeroIndexString", + "target": "step" + }, + "notes": "AI models that generate string-template expressions derive index numbers from the literal test intent (e.g. 'first item = index 0') but Provar string templates are 1-indexed. This causes runtime out-of-bounds errors.", + "source": "Field observation: SetValues zero-index string interpolation anti-pattern" + }, + { + "id": "ASSERT-STR-VAR-001", + "category": "StructureAndGrouping", + "name": "AssertValues must not use string literal to reference a variable", + "description": "AssertValues steps must use a proper XML variable reference () when comparing against a stored variable. Using a string literal that wraps a variable name — e.g. {RowCount} — causes Provar to compare the literal text \"{RowCount}\" against the actual value, which always produces a false failure. The same applies to function-call expressions such as {Count(Results)}.", + "appliesTo": ["Step"], + "severity": "major", + "weight": 5, + "recommendation": "Replace the string literal with a proper variable reference: . For function calls, use a preceding SetValues step to compute the value into a variable, then reference that variable in AssertValues.", + "check": { + "type": "assertValuesStringExpr", + "target": "step" + }, + "notes": "AI models generate {VarName} in a valueClass=string element on the expectedValue side because they reproduce the Provar UI display format. The correct form is a class=variable element. This causes every assertion to fail with a string-vs-value mismatch.", + "source": "Field observation: AssertValues string-variable reference anti-pattern" + }, + { + "id": "UI-LOCATOR-BUTTON-CASING-001", + "category": "TestCaseDesign", + "name": "Standard Salesforce flow buttons must use correct locator pattern", + "description": "Standard Salesforce flow buttons must use the correct locator name. Cancel must use lowercase 'name=cancel'. The Continue button on the Record Type selection screen uses a special path-based locator 'name=save&path=selectRecordType' — using 'name=continue' or 'name=Continue' will not resolve and causes Provar to show 'Not Available' at runtime. AI models commonly generate these incorrectly.", + "appliesTo": ["Step"], + "severity": "major", + "weight": 5, + "recommendation": "Cancel button: use name=cancel (lowercase). Continue button on record type selection: use name=save&path=selectRecordType (NOT name=continue or name=Continue). Corpus-validated patterns from 2000+ real test cases.", + "check": { + "type": "uiLocatorButtonCasing", + "target": "step", + "apiId": "com.provar.plugins.forcedotcom.core.ui.UiDoAction" + }, + "notes": "Corpus analysis: Cancel uses name=cancel (10 occurrences) vs name=Cancel (4). Continue on record type selection consistently uses name=save&path=selectRecordType across all corpus examples.", + "source": "Corpus analysis: AllPOCProjects + InternalProjects (2108 test cases)" + }, + { + "id": "UI-LOCATOR-RECORDTYPE-001", + "category": "TestCaseDesign", + "name": "Record Type field locator must use name=RecordType not name=recordTypeId", + "description": "On the Record Type selection screen (action=recordTypeNew), the Record Type picker field must use 'name=RecordType' in the locator URI with 'field=RecordTypeId' in the binding. AI models commonly generate 'name=recordTypeId' (the camelCase API name), which causes Provar to show 'Not Available' and the step fails at runtime.", + "appliesTo": ["Step"], + "severity": "major", + "weight": 5, + "recommendation": "Use name=RecordType (not name=recordTypeId or name=recordType) with field=RecordTypeId in the binding. Correct example: ui:locator?name=RecordType&binding=sf%3Aui%3Abinding%3Aobject%3Fobject%3D%7BtargetUrl%3Aobject%7D%26field%3DRecordTypeId", + "check": { + "type": "uiLocatorRecordTypeField", + "target": "step", + "apiId": "com.provar.plugins.forcedotcom.core.ui.UiDoAction" + }, + "notes": "Corpus analysis: 258 occurrences of name=RecordType+field=RecordTypeId vs 0 occurrences of name=recordTypeId. The {targetUrl:object} binding resolves dynamically to the current screen's object.", + "source": "Corpus analysis: AllPOCProjects + InternalProjects (2108 test cases)" } ] } diff --git a/src/mcp/rules/provar_layer1_rules.json b/src/mcp/rules/provar_layer1_rules.json new file mode 100644 index 00000000..289eb69f --- /dev/null +++ b/src/mcp/rules/provar_layer1_rules.json @@ -0,0 +1,151 @@ +{ + "schemaVersion": "1.0", + "name": "Provar Test Case Layer-1 Structural Validity Rules", + "description": "Single source of truth for the Layer-1 (structural validity) rule catalog. The DETECTION logic stays imperative in src/mcp/tools/testCaseValidate.ts; only the rule metadata (id, severity, applies_to, description) and the best-practice ownership/suppression mapping are centralized here. Consumers: testCaseValidate.ts (derives the bridge-suppression owned-set), scripts/build-validation-rule-registry.cjs (renders the Layer-1 registry rows), and test/unit/mcp/validationRuleRegistry.test.ts (drift guard). The `owns_bp_rules` field lists the Layer-2 `critical` best-practice rule ids whose concept this Layer-1 check already owns — those criticals are NOT bridged into is_valid (avoids double-reporting). Order is display order for the registry; keep it stable. severity is ERROR or WARNING (INFO reserved).", + "rules": [ + { + "id": "TC_001", + "severity": "ERROR", + "applies_to": "document", + "description": "XML declaration present ( first line)." + }, + { + "id": "TC_002", + "severity": "ERROR", + "applies_to": "document", + "description": "XML is well-formed (parses without error)." + }, + { + "id": "TC_003", + "severity": "ERROR", + "applies_to": "document", + "description": "Root element is .", + "owns_bp_rules": ["SCHEMA-ROOT-001"] + }, + { + "id": "TC_010", + "severity": "ERROR", + "applies_to": "testCase", + "description": "testCase id, when present, is a non-negative integer (id is optional; guid is the identifier).", + "owns_bp_rules": ["SCHEMA-ID-001"] + }, + { + "id": "TC_011", + "severity": "ERROR", + "applies_to": "testCase", + "description": "testCase has a guid attribute.", + "owns_bp_rules": ["VALID-GUID-001"] + }, + { + "id": "TC_012", + "severity": "ERROR", + "applies_to": "testCase", + "description": "testCase guid is a valid UUID v4." + }, + { + "id": "TC_020", + "severity": "ERROR", + "applies_to": "testCase", + "description": "testCase has a element.", + "owns_bp_rules": ["SCHEMA-STEPS-001", "VALID-STEPS-001"] + }, + { + "id": "TC_030", + "severity": "ERROR", + "applies_to": "apiCall", + "description": "Each apiCall has a guid attribute." + }, + { + "id": "TC_031", + "severity": "ERROR", + "applies_to": "apiCall", + "description": "Each apiCall guid is a valid UUID v4." + }, + { + "id": "TC_032", + "severity": "ERROR", + "applies_to": "apiCall", + "description": "Each apiCall has an apiId attribute." + }, + { + "id": "TC_033", + "severity": "WARNING", + "applies_to": "apiCall", + "description": "Each apiCall has a descriptive name attribute." + }, + { + "id": "TC_034", + "severity": "ERROR", + "applies_to": "apiCall", + "description": "Each apiCall has a testItemId attribute.", + "owns_bp_rules": ["STEP-ITEMID-001"] + }, + { + "id": "TC_035", + "severity": "ERROR", + "applies_to": "apiCall", + "description": "apiCall testItemId is a whole number." + }, + { + "id": "DATA-001", + "severity": "WARNING", + "applies_to": "testCase", + "description": " only iterates under a test plan; flags direct testCase-mode execution." + }, + { + "id": "VAR-REF-001", + "severity": "WARNING", + "applies_to": "argument", + "description": "A whole-token {Var} stored as valueClass=\"string\" (use class=\"variable\")." + }, + { + "id": "VAR-REF-002", + "severity": "WARNING", + "applies_to": "argument", + "description": "{Var} tokens embedded in a plain string (use class=\"compound\")." + }, + { + "id": "UI-TARGET-001", + "severity": "ERROR", + "applies_to": "apiCall", + "description": "UiWithScreen/UiWithRow target uses class=\"uiTarget\"." + }, + { + "id": "UI-LOCATOR-001", + "severity": "ERROR", + "applies_to": "apiCall", + "description": "UI action locator uses class=\"uiLocator\"." + }, + { + "id": "UI-INTERACTION-001", + "severity": "ERROR", + "applies_to": "apiCall", + "description": "UiDoAction interaction uses class=\"uiInteraction\"." + }, + { + "id": "UI-ASSERT-STRUCTURE-001", + "severity": "ERROR", + "applies_to": "apiCall", + "description": "UiAssert uses nested field/column/page assertion containers, not a flat argument." + }, + { + "id": "SETVALUES-STRUCTURE-001", + "severity": "ERROR", + "applies_to": "apiCall", + "description": "SetValues values argument uses class=\"valueList\" with ." + }, + { + "id": "ASSERT-001", + "severity": "WARNING", + "applies_to": "apiCall", + "description": "AssertValues namedValues format flagged for variable/Apex comparisons." + }, + { + "id": "COMPARISON-TYPE-001", + "severity": "ERROR", + "applies_to": "apiCall", + "description": "comparisonType is within the step-scoped enum subset (load-blocking otherwise).", + "owns_bp_rules": ["COMPARISON-TYPE-ENUM-001"] + } + ] +} diff --git a/src/mcp/rules/provar_test_step_schema.json b/src/mcp/rules/provar_test_step_schema.json new file mode 100644 index 00000000..fdcc9184 --- /dev/null +++ b/src/mcp/rules/provar_test_step_schema.json @@ -0,0 +1,3005 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "$comment": "Provar-specific structural reference for test case XML (top-level keys are Provar domain entities such as testCase/apiCalls, not JSON-Schema keywords). Despite the $schema declaration this is NOT a standards-compliant constraint JSON Schema; it is consumed as the provar://schema/test-step MCP resource and as the source for the validator's API-ID / value-class sets.", + "title": "Provar Test Case XML Structure Schema", + "description": "Complete mapping of Provar test case XML elements, apiCall types, and their required/optional arguments - organized by Test Palette categories", + "version": "2.0.0", + + "testCase": { + "description": "Root element of a Provar test case", + "required_attributes": ["id"], + "optional_attributes": ["guid", "registryId", "failureBehaviour", "visibility", "isTestTemplate"], + "child_elements": { + "summary": { + "description": "Human-readable description of test case", + "type": "text", + "required": false + }, + "params": { + "description": "Test case parameters for data-driven testing", + "required": false, + "children": { + "param": { + "required_attributes": ["name", "linkedToUrl"], + "optional_attributes": ["type", "passwordVariableAllowed", "title"] + } + } + }, + "args": { + "description": "Default argument values for test case parameters", + "required": false + }, + "steps": { + "description": "Container for all test steps", + "required": true, + "children": ["apiCall"] + } + } + }, + + "apiCall": { + "description": "Generic apiCall element representing a test step", + "required_attributes": ["guid", "apiId", "name", "testItemId"], + "optional_attributes": ["title", "isTitleTemplate", "disabled", "continueOnFailure"], + "guid_format": "UUID v4 format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", + "validation_rules": [ + "guid must be unique across all steps in test case", + "guid must be valid UUID v4 format", + "apiId must match a valid Provar API identifier", + "testItemId must be unique sequential integer starting from 1" + ], + "note": "Every element MUST have a guid attribute - this is mandatory for Provar test case validation" + }, + + "apiCalls": { + "description": "All supported Provar test step types organized by Test Palette categories", + + "Control": { + "description": "Control flow steps for loops, conditionals, assertions, and test flow management", + + "AssertValues": { + "apiId": "com.provar.plugins.bundled.apis.AssertValues", + "description": "Validate values using various comparison operators. IMPORTANT: actualValue can be empty for NotEqualTo blank string checks.", + "category": "Control", + "required_arguments": [ + { + "id": "expectedValue", + "type": "value", + "description": "Expected value for comparison" + }, + { + "id": "actualValue", + "type": "value", + "description": "Actual value to compare. Can be empty for NotEqualTo blank string checks." + }, + { + "id": "comparisonType", + "type": "string", + "description": "Type of comparison (EqualTo, NotEqualTo, Contains, GreaterThan, LessThan, etc.)" + } + ], + "optional_arguments": [ + { + "id": "caseSensitive", + "type": "boolean", + "default": "true", + "description": "Whether comparison is case-sensitive" + }, + { + "id": "numeric", + "type": "boolean", + "default": "true", + "description": "Whether to treat values as numbers" + }, + { + "id": "retainDecimals", + "type": "boolean", + "default": "false", + "description": "Whether to retain decimal precision" + }, + { + "id": "nullGreater", + "type": "boolean", + "default": "false", + "description": "Whether null is greater than non-null" + }, + { + "id": "matchMultiLine", + "type": "boolean", + "default": "false", + "description": "Whether to match across multiple lines (for regex)" + }, + { + "id": "matchDotAll", + "type": "boolean", + "default": "false", + "description": "Whether dot matches all characters (for regex)" + }, + { + "id": "failureMessage", + "type": "string", + "description": "Custom failure message" + }, + { + "id": "continueOnFailure", + "type": "boolean", + "default": "false" + } + ], + "validation_rules": [ + "expectedValue and actualValue types should be compatible", + "comparisonType must be valid operator (EqualTo, NotEqualTo, Contains, NotContain, StartsWith, EndsWith, GreaterThan, LessThan, GreaterOrEqual, LessOrEqual, Matches, NotMatches)", + "For NotEqualTo blank string checks, actualValue can be empty" + ], + "best_practices": [ + "Use specific comparison types rather than generic EqualTo", + "Add meaningful failure messages", + "For blank string checks, use expectedValue={variable}, comparisonType='NotEqualTo', actualValue=" + ] + }, + + "Break": { + "apiId": "com.provar.plugins.bundled.apis.control.Break", + "description": "Exit from a loop (ForEach, While) prematurely", + "category": "Control", + "required_arguments": [], + "optional_arguments": [], + "validation_rules": ["Must be inside a ForEach or While loop"], + "best_practices": [ + "Use Break sparingly - consider loop conditions instead", + "Document reason for breaking loop" + ] + }, + + "Fail": { + "apiId": "com.provar.plugins.bundled.apis.control.Fail", + "description": "Immediately fail the test with a message", + "category": "Control", + "required_arguments": [ + { + "id": "message", + "type": "string", + "description": "Failure message to display" + } + ], + "optional_arguments": [], + "validation_rules": ["message should be descriptive"], + "best_practices": ["Use for critical validation failures", "Provide clear, actionable failure messages"] + }, + + "Finally": { + "apiId": "com.provar.plugins.bundled.apis.control.Finally", + "description": "Execute cleanup steps that always run regardless of test outcome", + "category": "Control", + "required_arguments": [], + "optional_arguments": [], + "substeps": true, + "validation_rules": ["Should be at end of test", "Typically contains cleanup operations"], + "best_practices": [ + "Use for critical cleanup that must always execute", + "Keep Finally blocks simple and fast", + "Don't perform assertions in Finally" + ] + }, + + "ForEach": { + "apiId": "com.provar.plugins.bundled.apis.control.ForEach", + "description": "Loop through collection of items", + "category": "Control", + "required_arguments": [ + { + "id": "valuePath", + "type": "variable", + "description": "Collection to iterate over" + }, + { + "id": "iterationPathName", + "type": "string", + "description": "Variable name for current item" + } + ], + "optional_arguments": [], + "substeps": true, + "validation_rules": [ + "valuePath must reference a valid list/collection", + "iterationPathName should be meaningful", + "Must contain substeps" + ], + "best_practices": [ + "Use descriptive iteration variable names", + "Consider performance with large collections", + "Avoid nested ForEach when possible" + ] + }, + + "GroupSteps": { + "apiId": "com.provar.plugins.bundled.apis.control.StepGroup", + "description": "Organize related steps into logical groups", + "category": "Control", + "required_arguments": [ + { + "id": "name", + "type": "string", + "description": "Group name" + } + ], + "optional_arguments": [ + { + "id": "description", + "type": "string", + "description": "Detailed group description" + } + ], + "substeps": true, + "validation_rules": ["name should be descriptive", "Should contain multiple related steps"], + "best_practices": ["Use for tests with 10+ steps", "Group by logical functionality", "Add clear descriptions"] + }, + + "If": { + "apiId": "com.provar.plugins.bundled.apis.If", + "description": "Conditional execution based on boolean expression", + "category": "Control", + "required_arguments": [ + { + "id": "condition", + "type": "boolean", + "description": "Boolean expression to evaluate" + } + ], + "optional_arguments": [], + "substeps": true, + "clauses": ["then", "else"], + "validation_rules": [ + "condition must be boolean expression", + "Must have then clause", + "else clause is optional" + ], + "best_practices": [ + "Keep conditions simple and readable", + "Consider using Switch for multiple conditions", + "Avoid deeply nested If statements" + ] + }, + + "SetValues": { + "apiId": "com.provar.plugins.bundled.apis.control.SetValues", + "description": "Set variable values for use in test", + "category": "Control", + "required_arguments": [], + "optional_arguments": [], + "child_elements": { + "namedValues": { + "description": "Container for namedValue elements", + "children": ["namedValue"] + }, + "namedValue": { + "required_attributes": ["name"], + "optional_attributes": ["readOnly", "value"], + "description": "Individual variable assignment" + } + }, + "validation_rules": [ + "Must contain namedValues container", + "Each namedValue must have name attribute", + "Variable names should follow naming conventions" + ], + "best_practices": [ + "Use at beginning of test for test data", + "Use descriptive variable names", + "Group related variables together" + ] + }, + + "Sleep": { + "apiId": "com.provar.plugins.bundled.apis.control.Sleep", + "description": "Pause test execution for specified duration", + "category": "Control", + "required_arguments": [ + { + "id": "sleepSecs", + "type": "decimal", + "description": "Sleep duration" + } + ], + "optional_arguments": [ + { + "id": "unit", + "type": "string", + "default": "seconds", + "description": "Time unit (seconds)" + } + ], + "validation_rules": ["duration must be positive number", "Avoid long sleep durations"], + "best_practices": [ + "Prefer WaitFor over fixed Sleep", + "Keep sleepSecs under 10 seconds", + "Document reason for sleep" + ] + }, + + "Switch": { + "apiId": "com.provar.plugins.bundled.apis.Switch", + "description": "Multi-way conditional branching (similar to switch/case in programming languages)", + "category": "Control", + "required_arguments": [ + { + "id": "value", + "type": "value", + "description": "Expression to evaluate against case values" + } + ], + "optional_arguments": [], + "substeps": false, + "clauses": ["case"], + "structure": "Switch contains clause name='case' with steps containing multiple SwitchCase apiCalls", + "validation_rules": [ + "Must have at least one case", + "Uses clause name='case' (not 'substeps')", + "Individual cases are SwitchCase apiCalls within the case clause" + ], + "best_practices": [ + "Use Switch instead of multiple If statements", + "Always include default case", + "Keep cases simple" + ], + "note": "Individual case/default clauses use apiId='com.provar.plugins.bundled.apis.SwitchCase' as separate apiCall elements within the case clause" + }, + + "SwitchCase": { + "apiId": "com.provar.plugins.bundled.apis.SwitchCase", + "description": "Individual case or default clause within a Switch statement (similar to 'case' keyword in programming)", + "category": "Control", + "required_arguments": [ + { + "id": "value", + "type": "value", + "description": "Value to match against Switch value (omitted or empty for default case)" + }, + { + "id": "caseSensitive", + "type": "value", + "description": "Whether comparison is case-sensitive (Yes/No)" + } + ], + "optional_arguments": [ + { + "id": "alreadyMatched", + "type": "value", + "description": "Internal flag indicating if a previous case matched" + }, + { + "id": "switchValue", + "type": "value", + "description": "Internal reference to parent Switch value" + } + ], + "substeps": true, + "clauses": ["steps"], + "validation_rules": [ + "Must be child of Switch step's case clause", + "Default case has empty or no value argument", + "Multiple SwitchCase elements can exist within same Switch" + ], + "best_practices": [ + "Keep case logic simple", + "Use default for catch-all behavior", + "Order cases from most to least likely" + ], + "structure_example": "Switch > clause name='case' > steps > SwitchCase (with clause name='steps' containing substeps)" + }, + + "WaitFor": { + "apiId": "com.provar.plugins.bundled.apis.control.WaitFor", + "description": "Wait for condition or duration", + "category": "Control", + "required_arguments": [], + "optional_arguments": [ + { + "id": "duration", + "type": "decimal", + "description": "Maximum wait duration" + }, + { + "id": "condition", + "type": "boolean", + "description": "Condition to wait for" + }, + { + "id": "pollInterval", + "type": "decimal", + "default": "1", + "description": "How often to check condition (seconds)" + } + ], + "validation_rules": ["Must specify duration OR condition", "duration should be reasonable (< 60 seconds)"], + "best_practices": [ + "Prefer condition-based over fixed duration", + "Set reasonable timeout values", + "Use for async operations" + ] + }, + + "While": { + "apiId": "com.provar.plugins.bundled.apis.control.DoWhile", + "description": "Loop while condition is true", + "category": "Control", + "required_arguments": [ + { + "id": "condition", + "type": "boolean", + "description": "Loop condition" + } + ], + "optional_arguments": [ + { + "id": "maxIterations", + "type": "decimal", + "description": "Maximum loop iterations to prevent infinite loops" + } + ], + "substeps": true, + "validation_rules": [ + "condition must be boolean", + "Should have maxIterations to prevent infinite loops", + "Must contain substeps" + ], + "best_practices": [ + "Always set maxIterations", + "Ensure condition will eventually become false", + "Prefer ForEach for collections" + ] + } + }, + + "Data": { + "description": "Database, REST/Web, and messaging operations", + + "DbConnect": { + "apiId": "com.provar.plugins.bundled.apis.db.DbConnect", + "description": "Connect to external database (Oracle, SQL Server, DB2, MySQL, PostgreSQL)", + "category": "Data", + "required_arguments": [ + { + "id": "connectionName", + "type": "string", + "description": "Name of database connection" + } + ], + "optional_arguments": [ + { + "id": "connectionId", + "type": "id" + }, + { + "id": "autoCommit", + "type": "boolean", + "default": "true" + }, + { + "id": "commitBehavior", + "type": "string", + "default": "AfterTest" + }, + { + "id": "ifAlreadyOpen", + "type": "string", + "default": "Fail" + } + ], + "validation_rules": ["Should be first DB operation", "connectionName must be unique"], + "best_practices": ["Use connection pooling when possible", "Set appropriate commit behavior"] + }, + + "DbDelete": { + "apiId": "com.provar.plugins.bundled.apis.db.DbDelete", + "description": "Delete records from database table", + "category": "Data", + "required_arguments": [ + { + "id": "dbConnectionName", + "type": "string" + }, + { + "id": "tableName", + "type": "string" + } + ], + "optional_arguments": [ + { + "id": "whereClause", + "type": "string" + } + ], + "validation_rules": [ + "Must have DbConnect first", + "tableName must be valid", + "Should have whereClause to avoid deleting all records" + ], + "best_practices": ["Always use WHERE clause", "Test with SELECT first"] + }, + + "DbInsert": { + "apiId": "com.provar.plugins.bundled.apis.db.DbInsert", + "description": "Insert records into database table", + "category": "Data", + "required_arguments": [ + { + "id": "dbConnectionName", + "type": "string" + }, + { + "id": "tableName", + "type": "string" + } + ], + "optional_arguments": [ + { + "id": "columnValues", + "type": "object", + "description": "Column name/value pairs" + } + ], + "validation_rules": [ + "Must have DbConnect first", + "tableName must be valid", + "columnValues should match table schema" + ], + "best_practices": ["Validate data before insert", "Handle insert failures gracefully"] + }, + + "DbRead": { + "apiId": "com.provar.plugins.bundled.apis.db.DbRead", + "description": "Read records from database table", + "category": "Data", + "required_arguments": [ + { + "id": "dbConnectionName", + "type": "string" + }, + { + "id": "tableName", + "type": "string" + } + ], + "optional_arguments": [ + { + "id": "whereClause", + "type": "string" + }, + { + "id": "resultName", + "type": "string" + } + ], + "validation_rules": ["Must have DbConnect first", "Should store results in variable"], + "best_practices": ["Use WHERE clause to limit results", "Store in meaningful variable name"] + }, + + "DbUpdate": { + "apiId": "com.provar.plugins.bundled.apis.db.DbUpdate", + "description": "Update records in database table", + "category": "Data", + "required_arguments": [ + { + "id": "dbConnectionName", + "type": "string" + }, + { + "id": "tableName", + "type": "string" + } + ], + "optional_arguments": [ + { + "id": "columnValues", + "type": "object" + }, + { + "id": "whereClause", + "type": "string" + } + ], + "validation_rules": ["Must have DbConnect first", "Should have WHERE clause"], + "best_practices": ["Always use WHERE clause", "Verify update with SELECT"] + }, + + "SqlQuery": { + "apiId": "com.provar.plugins.bundled.apis.db.SqlQuery", + "description": "Execute custom SQL query", + "category": "Data", + "required_arguments": [ + { + "id": "dbConnectionName", + "type": "string" + }, + { + "id": "query", + "type": "string", + "description": "SQL query to execute" + } + ], + "optional_arguments": [ + { + "id": "resultName", + "type": "string" + } + ], + "validation_rules": ["Must have DbConnect first", "query must be valid SQL"], + "best_practices": ["Use parameterized queries", "Store results for verification"] + }, + + "WebConnect": { + "apiId": "com.provar.plugins.bundled.apis.restservice.WebConnect", + "description": "Connect to REST API or web service", + "category": "Data", + "required_arguments": [ + { + "id": "connectionName", + "type": "string" + } + ], + "optional_arguments": [ + { + "id": "connectionId", + "type": "id" + }, + { + "id": "resultName", + "type": "string" + } + ], + "validation_rules": ["Required before RestRequest", "Required for Agentforce connections"], + "best_practices": ["Reuse connections when possible", "Configure authentication properly"] + }, + + "RestRequest": { + "apiId": "com.provar.plugins.bundled.apis.restservice.RestRequest", + "description": "Make REST API call", + "category": "Data", + "required_arguments": [ + { + "id": "webConnectionName", + "type": "string" + }, + { + "id": "endpoint", + "type": "string" + }, + { + "id": "method", + "type": "string", + "description": "HTTP method (GET, POST, PUT, DELETE, PATCH)" + } + ], + "optional_arguments": [ + { + "id": "body", + "type": "string" + }, + { + "id": "headers", + "type": "object" + }, + { + "id": "resultName", + "type": "string" + } + ], + "validation_rules": [ + "Must have WebConnect first", + "method must be valid HTTP method", + "Body required for POST/PUT/PATCH" + ], + "best_practices": ["Validate response status", "Store response for assertions"] + }, + + "SoapRequest": { + "apiId": "com.provar.plugins.bundled.apis.restservice.SoapRequest", + "description": "Make SOAP web service request", + "category": "Data", + "required_arguments": [ + { + "id": "connectionName", + "type": "string" + }, + { + "id": "operation", + "type": "string", + "description": "SOAP operation name" + } + ], + "optional_arguments": [ + { + "id": "requestBody", + "type": "string", + "description": "SOAP request XML" + }, + { + "id": "headers", + "type": "object" + }, + { + "id": "resultName", + "type": "string" + } + ], + "validation_rules": [ + "Must have WebConnect first", + "operation must be valid SOAP operation", + "requestBody should be well-formed XML" + ], + "best_practices": ["Use REST instead of SOAP when possible", "Validate SOAP response"] + }, + + "PublishMessage": { + "apiId": "com.provar.plugins.bundled.apis.messaging.PublishMessage", + "description": "Publish message to messaging service", + "category": "Data", + "required_arguments": [ + { + "id": "connectionName", + "type": "string" + }, + { + "id": "message", + "type": "string" + } + ], + "optional_arguments": [ + { + "id": "topic", + "type": "string" + }, + { + "id": "properties", + "type": "object" + } + ], + "validation_rules": ["connectionName must be messaging connection", "message should be valid format"], + "best_practices": ["Use for async testing", "Validate message format"] + }, + + "ReceiveMessage": { + "apiId": "com.provar.plugins.bundled.apis.messaging.ReceiveMessage", + "description": "Receive message from messaging service", + "category": "Data", + "required_arguments": [ + { + "id": "connectionName", + "type": "string" + } + ], + "optional_arguments": [ + { + "id": "timeout", + "type": "decimal", + "default": "30", + "description": "Timeout in seconds" + }, + { + "id": "selector", + "type": "string", + "description": "Message selector/filter" + }, + { + "id": "resultName", + "type": "string" + } + ], + "validation_rules": ["timeout should be reasonable", "Store result for verification"], + "best_practices": ["Set appropriate timeout", "Use selector to filter messages"] + }, + + "SendMessage": { + "apiId": "com.provar.plugins.bundled.apis.messaging.SendMessage", + "description": "Send message to specific recipient", + "category": "Data", + "required_arguments": [ + { + "id": "connectionName", + "type": "string" + }, + { + "id": "message", + "type": "string" + }, + { + "id": "recipient", + "type": "string" + } + ], + "optional_arguments": [ + { + "id": "properties", + "type": "object" + } + ], + "validation_rules": ["recipient must be valid", "message format should be correct"], + "best_practices": ["Validate recipient exists", "Handle send failures"] + }, + + "Subscribe": { + "apiId": "com.provar.plugins.bundled.apis.messaging.Subscribe", + "description": "Subscribe to messaging topic/channel", + "category": "Data", + "required_arguments": [ + { + "id": "connectionName", + "type": "string" + }, + { + "id": "topic", + "type": "string" + } + ], + "optional_arguments": [ + { + "id": "resultName", + "type": "string" + } + ], + "validation_rules": ["topic must exist", "Subscribe before ReceiveMessage"], + "best_practices": ["Unsubscribe in cleanup", "Handle subscription errors"] + } + }, + + "Design": { + "description": "BDD (Behavior-Driven Development) steps for Gherkin-style test organization", + + "Given": { + "apiId": "com.provar.plugins.bundled.apis.bdd.Given", + "description": "BDD Given step - set up preconditions", + "category": "Design", + "required_arguments": [ + { + "id": "description", + "type": "string", + "description": "Given statement description" + } + ], + "optional_arguments": [], + "substeps": true, + "validation_rules": [ + "Should be first BDD step in scenario", + "Description should start with present tense", + "Must contain substeps" + ], + "best_practices": [ + "Use for test setup and preconditions", + "Keep description clear and concise", + "Focus on system state, not actions" + ] + }, + + "When": { + "apiId": "com.provar.plugins.bundled.apis.bdd.When", + "description": "BDD When step - perform actions", + "category": "Design", + "required_arguments": [ + { + "id": "description", + "type": "string", + "description": "When statement description" + } + ], + "optional_arguments": [], + "substeps": true, + "validation_rules": ["Should follow Given step", "Description should describe action", "Must contain substeps"], + "best_practices": [ + "Use for test actions/operations", + "One When per scenario when possible", + "Describe user actions clearly" + ] + }, + + "Then": { + "apiId": "com.provar.plugins.bundled.apis.bdd.Then", + "description": "BDD Then step - verify outcomes", + "category": "Design", + "required_arguments": [ + { + "id": "description", + "type": "string", + "description": "Then statement description" + } + ], + "optional_arguments": [], + "substeps": true, + "validation_rules": [ + "Should follow When step", + "Description should describe expected outcome", + "Must contain substeps with assertions" + ], + "best_practices": [ + "Use for verification and assertions", + "Focus on observable outcomes", + "Each Then should verify one concept" + ] + }, + + "And": { + "apiId": "com.provar.plugins.bundled.apis.bdd.And", + "description": "BDD And connector - additional steps of same type", + "category": "Design", + "required_arguments": [ + { + "id": "description", + "type": "string", + "description": "And statement description" + } + ], + "optional_arguments": [], + "substeps": true, + "validation_rules": [ + "Should follow Given, When, or Then", + "Inherits context from previous step", + "Must contain substeps" + ], + "best_practices": [ + "Use to add additional steps of same type", + "Keep And chains reasonable (< 5)", + "Consider breaking into separate scenarios if too many Ands" + ] + }, + + "But": { + "apiId": "com.provar.plugins.bundled.apis.bdd.But", + "description": "BDD But connector - contrasting additional steps", + "category": "Design", + "required_arguments": [ + { + "id": "description", + "type": "string", + "description": "But statement description" + } + ], + "optional_arguments": [], + "substeps": true, + "validation_rules": [ + "Should follow Given, When, or Then", + "Used for contrasting conditions", + "Must contain substeps" + ], + "best_practices": [ + "Use for negative assertions or exceptions", + "Clarifies contrast with previous steps", + "Less common than And" + ] + }, + + "ActualResult": { + "apiId": "com.provar.plugins.bundled.apis.control.ActualResult", + "description": "Document actual test result for manual review (Design phase step)", + "category": "Design", + "required_arguments": [ + { + "id": "description", + "type": "string", + "description": "Actual result description" + } + ], + "optional_arguments": [], + "substeps": true, + "validation_rules": ["description should be descriptive"], + "best_practices": [ + "Use for exploratory testing", + "Document unexpected behavior", + "Helpful for test case design phase" + ] + }, + + "DesignStep": { + "apiId": "com.provar.plugins.bundled.apis.control.DesignStep", + "description": "Placeholder step for test case design phase", + "category": "Design", + "required_arguments": [ + { + "id": "name", + "type": "string", + "description": "Step name" + }, + { + "id": "description", + "type": "string", + "description": "Step description" + }, + { + "id": "expectedResult", + "type": "string", + "description": "Step expected result" + } + ], + "optional_arguments": [], + "validation_rules": [ + "Should be replaced with actual implementation", + "description should describe intended action" + ], + "best_practices": [ + "Use during test design phase", + "Replace with actual steps before execution", + "Helps plan test structure" + ] + } + }, + + "ProvarAI": { + "description": "AI-powered testing capabilities including Agentforce and test data generation", + + "AIAgentSession": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ai.AIAgentSession", + "description": "Create Agentforce AI agent session", + "category": "ProvarAI", + "required_arguments": [ + { + "id": "agentConnectionName", + "type": "string", + "description": "WebConnect connection name" + }, + { + "id": "agentId", + "type": "string", + "description": "Agent ID from Salesforce" + }, + { + "id": "resultName", + "type": "string", + "description": "Variable to store session ID" + } + ], + "optional_arguments": [ + { + "id": "endSession", + "type": "boolean", + "default": "false", + "description": "End session after creation" + } + ], + "validation_rules": [ + "Must have WebConnect step first", + "agentId must be valid GUID format", + "resultName should be unique" + ], + "best_practices": [ + "Create session at start of agent testing", + "Store session ID for subsequent conversations", + "End session explicitly in cleanup" + ] + }, + + "AIAgentConversation": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ai.AIAgentConversation", + "description": "Send message to AI agent and receive response", + "category": "ProvarAI", + "required_arguments": [ + { + "id": "agentConnectionName", + "type": "string" + }, + { + "id": "sessionID", + "type": "variable", + "description": "Session ID from AIAgentSession" + }, + { + "id": "message", + "type": "string", + "description": "Message to send to agent" + }, + { + "id": "resultName", + "type": "string", + "description": "Variable to store agent response" + } + ], + "optional_arguments": [ + { + "id": "endSession", + "type": "boolean", + "default": "false" + } + ], + "validation_rules": [ + "Must have AIAgentSession first", + "sessionID must reference valid session", + "message should be meaningful test input" + ], + "best_practices": [ + "Test various conversation scenarios", + "Validate agent responses", + "Use in ForEach for multiple messages" + ] + }, + + "GenerateUtterance": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ai.GenerateUtterance", + "description": "Generate AI-powered test utterances for intent testing", + "category": "ProvarAI", + "required_arguments": [ + { + "id": "intent", + "type": "string", + "description": "Intent to generate utterances for" + }, + { + "id": "resultName", + "type": "string", + "description": "Variable to store generated utterances" + } + ], + "optional_arguments": [ + { + "id": "count", + "type": "decimal", + "default": "10", + "description": "Number of utterances to generate" + }, + { + "id": "generateCsvFile", + "type": "boolean", + "default": "false" + } + ], + "validation_rules": ["intent should be descriptive", "count should be reasonable (1-100)"], + "best_practices": [ + "Use for comprehensive intent testing", + "Generate variety of phrasings", + "Validate with IntentValidator" + ] + }, + + "IntentValidator": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ai.IntentValidator", + "description": "Validate that utterance matches expected intent", + "category": "ProvarAI", + "required_arguments": [ + { + "id": "intent", + "type": "string", + "description": "Expected intent" + }, + { + "id": "utterance", + "type": "string", + "description": "Utterance to validate" + }, + { + "id": "resultName", + "type": "string", + "description": "Variable to store validation result" + } + ], + "optional_arguments": [ + { + "id": "continueOnFailure", + "type": "boolean", + "default": "false" + } + ], + "validation_rules": ["intent must be valid", "utterance should be test input"], + "best_practices": [ + "Use with GenerateUtterance", + "Test edge cases and variations", + "Document validation failures" + ] + }, + + "GenerateTestData": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.generate.GenerateTestData", + "description": "AI-powered test data generation", + "category": "ProvarAI", + "required_arguments": [ + { + "id": "dataType", + "type": "string", + "description": "Type of data to generate (email, name, address, etc.)" + }, + { + "id": "resultName", + "type": "string", + "description": "Variable to store generated data" + } + ], + "optional_arguments": [ + { + "id": "count", + "type": "decimal", + "default": "1", + "description": "Number of data items to generate" + } + ], + "validation_rules": ["dataType must be supported type", "count should be reasonable"], + "best_practices": [ + "Use for realistic test data", + "Validate generated data format", + "Consider data privacy requirements" + ] + }, + + "ImageValidator": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ai.ImageValidator", + "description": "AI-powered image validation", + "category": "ProvarAI", + "required_arguments": [ + { + "id": "imagePath", + "type": "string", + "description": "Path to image file" + }, + { + "id": "validationCriteria", + "type": "string", + "description": "What to validate in image" + }, + { + "id": "resultName", + "type": "string" + } + ], + "optional_arguments": [ + { + "id": "confidenceThreshold", + "type": "decimal", + "default": "0.8", + "description": "Minimum confidence score (0.0-1.0)" + } + ], + "validation_rules": [ + "imagePath must be valid file path", + "validationCriteria should be specific", + "confidenceThreshold should be 0.0-1.0" + ], + "best_practices": [ + "Use for visual regression testing", + "Set appropriate confidence threshold", + "Test with various image conditions" + ] + } + }, + + "ProvarLabs": { + "description": "Experimental and advanced Provar features", + + "GenerateTestCase": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.GenerateTestCase", + "description": "AI-powered test case generation from scenario", + "category": "ProvarLabs", + "required_arguments": [ + { + "id": "scenario", + "type": "string", + "description": "Test scenario description" + }, + { + "id": "resultName", + "type": "string", + "description": "Variable to store generated test case" + } + ], + "optional_arguments": [ + { + "id": "template", + "type": "string", + "description": "Template to use for generation" + }, + { + "id": "framework", + "type": "string", + "description": "Test framework (BDD, standard, etc.)" + } + ], + "validation_rules": ["scenario should be clear and detailed", "Review generated test before use"], + "best_practices": [ + "Use for rapid test creation", + "Review and refine generated tests", + "Provide detailed scenarios for better results" + ] + }, + + "PageObjectCleaner": { + "apiId": "com.provar.plugins.bundled.apis.provarlabs.PageObjectCleaner", + "description": "Clean up and optimize page object models", + "category": "ProvarLabs", + "required_arguments": [ + { + "id": "pageObjectPath", + "type": "string", + "description": "Path to page object file" + } + ], + "optional_arguments": [ + { + "id": "cleanupStrategy", + "type": "string", + "default": "RemoveUnused", + "description": "Strategy for cleanup" + }, + { + "id": "resultName", + "type": "string" + } + ], + "validation_rules": ["pageObjectPath must be valid", "Backup page objects before cleaning"], + "best_practices": [ + "Run periodically to maintain page objects", + "Review changes before committing", + "Use in maintenance scripts" + ] + } + }, + + "Salesforce": { + "description": "Salesforce-specific operations including Apex API, SOQL, and SF-specific features", + + "ApexConnect": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ApexConnect", + "description": "Connect to Salesforce Apex API", + "category": "Salesforce", + "required_arguments": [ + { + "id": "connectionName", + "type": "string", + "description": "Name of the Salesforce connection" + } + ], + "optional_arguments": [ + { + "id": "connectionId", + "type": "string" + }, + { + "id": "autoCleanup", + "type": "boolean", + "default": "true", + "description": "Automatically delete created objects after test" + }, + { + "id": "enableObjectIdLogging", + "type": "boolean", + "default": "false" + }, + { + "id": "resultName", + "type": "string" + }, + { + "id": "resultScope", + "type": "string", + "default": "Test" + } + ], + "validation_rules": [ + "Should be first step when using Apex API", + "connectionName must be unique", + "If autoCleanup=true, enableObjectIdLogging must be true" + ], + "best_practices": [ + "Use meaningful connection names", + "Set autoCleanup=true unless manual cleanup required", + "Reuse connections when possible" + ] + }, + + "ApexCreateObject": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ApexCreateObject", + "description": "Create Salesforce object via Apex", + "category": "Salesforce", + "required_arguments": [ + { + "id": "apexConnectionName", + "type": "string" + }, + { + "id": "resultIdName", + "type": "string", + "description": "Variable to store created object ID" + }, + { + "id": "objectType", + "type": "string", + "description": "Salesforce object type (Account, Contact, etc.)" + } + ], + "optional_arguments": [ + { + "id": "fieldValues", + "type": "object", + "description": "Field values for the object" + } + ], + "validation_rules": [ + "Must have ApexConnect first", + "objectType must be valid SF object", + "Required fields must be provided" + ], + "best_practices": ["Store result ID for later use", "Provide all required fields", "Use for test data setup"] + }, + + "ApexReadObject": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ApexReadObject", + "description": "Read Salesforce object by ID", + "category": "Salesforce", + "required_arguments": [ + { + "id": "apexConnectionName", + "type": "string" + }, + { + "id": "objectType", + "type": "string" + }, + { + "id": "resultName", + "type": "string" + }, + { + "id": "objectId", + "type": "variable", + "description": "ID of object to read" + } + ], + "optional_arguments": [ + { + "id": "fields", + "type": "array", + "description": "Fields to retrieve" + } + ], + "validation_rules": ["Must have ApexConnect first", "objectId must be valid 15 or 18 character SF ID"], + "best_practices": ["Specify fields to retrieve", "Use for verification steps"] + }, + + "ApexUpdateObject": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ApexUpdateObject", + "description": "Update Salesforce object", + "category": "Salesforce", + "required_arguments": [ + { + "id": "apexConnectionName", + "type": "string" + }, + { + "id": "objectType", + "type": "string" + }, + { + "id": "objectId", + "type": "variable" + } + ], + "optional_arguments": [ + { + "id": "fieldValues", + "type": "object" + } + ], + "validation_rules": ["Must have ApexConnect first", "objectId must be valid"], + "best_practices": ["Only update fields that changed", "Verify update with ApexReadObject"] + }, + + "ApexDeleteObject": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ApexDeleteObject", + "description": "Delete Salesforce object", + "category": "Salesforce", + "required_arguments": [ + { + "id": "apexConnectionName", + "type": "string" + }, + { + "id": "objectId", + "type": "variable" + } + ], + "optional_arguments": [], + "validation_rules": ["Must have ApexConnect first", "Use only if autoCleanup=false"], + "best_practices": [ + "Prefer autoCleanup over manual delete", + "Delete in Finally block for reliability", + "Use ApexLogForCleanup for SOQL-based cleanup" + ] + }, + + "ApexSoqlQuery": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ApexSoqlQuery", + "description": "Execute SOQL query", + "category": "Salesforce", + "required_arguments": [ + { + "id": "apexConnectionName", + "type": "string" + }, + { + "id": "soqlQuery", + "type": "string", + "description": "SOQL query string" + } + ], + "optional_arguments": [ + { + "id": "resultListName", + "type": "string", + "description": "Variable to store query results" + }, + { + "id": "resultScope", + "type": "string", + "default": "Test" + } + ], + "validation_rules": [ + "Must have ApexConnect first", + "SOQL must have SELECT and FROM clauses", + "Always retrieve Id field", + "Use {VariableName[1].FieldName} syntax (1-indexed)" + ], + "best_practices": [ + "Always select Id and Name fields", + "Use WHERE clause to limit results", + "Store results for later use", + "Use ORDER BY for predictable results" + ] + }, + + "ApexExecute": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ApexExecute", + "description": "Execute anonymous Apex code", + "category": "Salesforce", + "required_arguments": [ + { + "id": "apexConnectionName", + "type": "string" + }, + { + "id": "apexBlock", + "type": "string", + "description": "Apex code to execute" + } + ], + "optional_arguments": [ + { + "id": "results", + "type": "string", + "description": "Variable to store execution results" + } + ], + "validation_rules": ["Must have ApexConnect first", "apexBlock must be valid Apex syntax"], + "best_practices": [ + "Use for complex operations not available via API", + "Keep Apex code simple", + "Test Apex code in Developer Console first" + ] + }, + + "ApexBulk": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ApexBulk", + "description": "Execute bulk Apex operations", + "category": "Salesforce", + "required_arguments": [ + { + "id": "apexConnectionName", + "type": "string" + }, + { + "id": "objectType", + "type": "string" + }, + { + "id": "operation", + "type": "string", + "description": "Bulk operation (insert, update, delete, etc.)" + } + ], + "optional_arguments": [ + { + "id": "records", + "type": "array", + "description": "Records to process" + }, + { + "id": "resultName", + "type": "string" + } + ], + "validation_rules": [ + "Must have ApexConnect first", + "operation must be valid bulk operation", + "records must be valid for operation" + ], + "best_practices": [ + "Use for large data volumes (> 200 records)", + "Handle bulk API limits", + "Monitor bulk job status" + ] + }, + + "ApexApproveWorkItem": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ApexApproveWorkItem", + "description": "Approve Salesforce approval work item", + "category": "Salesforce", + "required_arguments": [ + { + "id": "apexConnectionName", + "type": "string" + }, + { + "id": "workItemId", + "type": "string" + } + ], + "optional_arguments": [ + { + "id": "comments", + "type": "string" + } + ], + "validation_rules": ["Must have ApexConnect first", "workItemId must be valid"], + "best_practices": ["Add approval comments", "Verify approval status after"] + }, + + "SubmitForApproval": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ApexSubmitForApproval", + "description": "Submit record for approval", + "category": "Salesforce", + "required_arguments": [ + { + "id": "apexConnectionName", + "type": "string" + }, + { + "id": "objectId", + "type": "variable" + } + ], + "optional_arguments": [ + { + "id": "comments", + "type": "string" + }, + { + "id": "processName", + "type": "string" + } + ], + "validation_rules": ["Must have ApexConnect first", "Object must have approval process"], + "best_practices": ["Verify record meets approval criteria", "Track approval process status"] + }, + + "ConvertLead": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ApexConvertLead", + "description": "Convert Salesforce lead", + "category": "Salesforce", + "required_arguments": [ + { + "id": "apexConnectionName", + "type": "string" + }, + { + "id": "leadId", + "type": "variable" + }, + { + "id": "convertedStatus", + "type": "string" + } + ], + "optional_arguments": [ + { + "id": "contactId", + "type": "variable" + }, + { + "id": "accountId", + "type": "variable" + }, + { + "id": "ownerId", + "type": "variable" + }, + { + "id": "createOpportunity", + "type": "boolean", + "default": "false" + }, + { + "id": "opportunityName", + "type": "string" + } + ], + "validation_rules": ["Must have ApexConnect first", "convertedStatus must be valid lead status"], + "best_practices": ["Verify lead meets conversion criteria", "Store converted IDs for verification"] + }, + + "ExtractSalesforceLayout": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ApexExtractLayout", + "description": "Extract Salesforce page layout for comparison", + "category": "Salesforce", + "required_arguments": [ + { + "id": "apexConnectionName", + "type": "string" + }, + { + "id": "objectName", + "type": "string" + } + ], + "optional_arguments": [ + { + "id": "extractRecordPageLayout", + "type": "boolean", + "default": "true" + }, + { + "id": "extractDynamicFormLayout", + "type": "boolean", + "default": "false" + }, + { + "id": "dataUrl", + "type": "string", + "description": "File path to save layout" + } + ], + "validation_rules": ["Must have ApexConnect first", "objectName must be valid"], + "best_practices": ["Use for layout validation tests", "Save layouts for regression testing"] + }, + + "AssertSalesforceLayout": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ApexAssertLayout", + "description": "Assert Salesforce page layout matches expected", + "category": "Salesforce", + "required_arguments": [ + { + "id": "apexConnectionName", + "type": "string" + }, + { + "id": "objectName", + "type": "string" + }, + { + "id": "dataUrl", + "type": "string", + "description": "Path to expected layout file" + } + ], + "optional_arguments": [ + { + "id": "extractRecordPageLayout", + "type": "boolean", + "default": "true" + }, + { + "id": "extractDynamicFormLayout", + "type": "boolean", + "default": "false" + } + ], + "validation_rules": [ + "Must have ApexConnect first", + "Must have ExtractSalesforceLayout first", + "dataUrl must point to valid layout file" + ], + "best_practices": ["Use for regression testing", "Update baseline after intentional changes"] + }, + + "LogForCleanup": { + "apiId": "com.provar.plugins.forcedotcom.core.testapis.ApexLogForCleanup", + "description": "Log SOQL query results for cleanup", + "category": "Salesforce", + "required_arguments": [ + { + "id": "apexConnectionName", + "type": "string" + }, + { + "id": "selector", + "type": "string", + "description": "SOQL query to select records for cleanup" + } + ], + "optional_arguments": [ + { + "id": "maxMatches", + "type": "decimal", + "description": "Maximum records to log" + } + ], + "validation_rules": [ + "Must have ApexConnect first with enableObjectIdLogging=true", + "selector must be valid SOQL", + "Use ORDER BY CreatedDate DESC LIMIT 1 for latest record" + ], + "best_practices": [ + "Use for dynamic cleanup", + "Add after create operations", + "Use with ORDER BY for predictable cleanup" + ] + } + }, + + "UI": { + "description": "User interface automation steps for Salesforce and web applications", + + "UiConnect": { + "apiId": "com.provar.plugins.forcedotcom.core.ui.UiConnect", + "description": "Connect to Salesforce UI or web application", + "category": "UI", + "required_arguments": [ + { + "id": "connectionName", + "type": "string" + } + ], + "optional_arguments": [ + { + "id": "connectionId", + "type": "string" + }, + { + "id": "uiApplicationName", + "type": "string", + "default": "LightningSales" + }, + { + "id": "reuseConnectionName", + "type": "string" + }, + { + "id": "closeAllOtherTabs", + "type": "boolean", + "default": "true" + }, + { + "id": "privateBrowsingMode", + "type": "boolean", + "default": "false" + }, + { + "id": "webBrowser", + "type": "string", + "default": "Chrome_Headless" + } + ], + "validation_rules": ["Should be first UI step", "connectionName must be unique"], + "best_practices": [ + "Use headless mode for CI/CD", + "Reuse connections when possible", + "Close tabs for clean state" + ] + }, + + "MSDynamics365Connect": { + "apiId": "com.provar.plugins.forcedotcom.core.ui.NitroXConnect:ms-dynamics365", + "description": "Connect to a Microsoft Dynamics 365 application via the NitroX connector. Use INSTEAD of UiConnect when targeting Microsoft Dynamics 365.", + "category": "UI", + "required_arguments": [ + { + "id": "connectionName", + "type": "string", + "description": "Provar connection profile name configured for the Dynamics 365 environment" + }, + { + "id": "resultName", + "type": "string", + "description": "Variable name that downstream UI steps reference via uiConnectionName" + }, + { + "id": "resultScope", + "type": "string", + "default": "Test", + "description": "Scope of the connection result (typically 'Test')" + }, + { + "id": "appName", + "type": "string", + "description": "Dynamics 365 app to open after sign-in (e.g., 'Sales Hub', 'Customer Service Hub'). May be left empty if declared as a runtime-bound in ." + } + ], + "optional_arguments": [ + { "id": "reuseConnectionName", "type": "string" }, + { "id": "privateBrowsingMode", "type": "boolean", "default": "false" }, + { "id": "webBrowser", "type": "string", "default": "Chrome_Headless" } + ], + "validation_rules": [ + "Use INSTEAD of UiConnect/ApexConnect for Microsoft Dynamics 365 targets", + "resultName is required so downstream UI steps can set uiConnectionName", + "appName must be populated OR declared as a runtime parameter in (data-driven pattern)" + ], + "best_practices": [ + "Set appName to the target model-driven app to land directly on the right surface", + "Declare appName in when the value is supplied at runtime", + "Use headless mode for CI/CD" + ] + }, + + "MSDataverseConnect": { + "apiId": "com.provar.plugins.forcedotcom.core.ui.NitroXConnect:ms-dataverse", + "description": "Connect to a Microsoft Dataverse environment via the NitroX connector. Use INSTEAD of UiConnect when targeting Microsoft Dataverse.", + "category": "UI", + "required_arguments": [ + { + "id": "connectionName", + "type": "string", + "description": "Provar connection profile name configured for the Dataverse environment" + }, + { + "id": "resultName", + "type": "string", + "description": "Variable name that downstream UI steps reference via uiConnectionName" + }, + { + "id": "resultScope", + "type": "string", + "default": "Test", + "description": "Scope of the connection result (typically 'Test')" + } + ], + "optional_arguments": [ + { "id": "reuseConnectionName", "type": "string" }, + { "id": "privateBrowsingMode", "type": "boolean", "default": "false" }, + { "id": "webBrowser", "type": "string", "default": "Chrome_Headless" } + ], + "validation_rules": [ + "Use INSTEAD of UiConnect/ApexConnect for Microsoft Dataverse targets", + "resultName is required so downstream UI steps can set uiConnectionName", + "Has no variant-specific arguments; stays empty" + ], + "best_practices": [ + "Reuse the connection across the test rather than reconnecting", + "Use headless mode for CI/CD" + ] + }, + + "MSPowerAppConnect": { + "apiId": "com.provar.plugins.forcedotcom.core.ui.NitroXConnect:ms-powerapp", + "description": "Connect to a Microsoft Power App via the NitroX connector. Use INSTEAD of UiConnect when targeting Microsoft Power Apps.", + "category": "UI", + "required_arguments": [ + { + "id": "connectionName", + "type": "string", + "description": "Provar connection profile name configured for the Power Apps tenant" + }, + { + "id": "resultName", + "type": "string", + "description": "Variable name that downstream UI steps reference via uiConnectionName" + }, + { + "id": "resultScope", + "type": "string", + "default": "Test", + "description": "Scope of the connection result (typically 'Test')" + }, + { + "id": "powerAppName", + "type": "string", + "description": "Display name of the target Power App. May be left empty if declared as a runtime-bound in ." + } + ], + "optional_arguments": [ + { "id": "reuseConnectionName", "type": "string" }, + { "id": "privateBrowsingMode", "type": "boolean", "default": "false" }, + { "id": "webBrowser", "type": "string", "default": "Chrome_Headless" } + ], + "validation_rules": [ + "Use INSTEAD of UiConnect/ApexConnect for Microsoft Power Apps targets", + "resultName is required so downstream UI steps can set uiConnectionName", + "powerAppName must be populated OR declared as a runtime parameter in (data-driven pattern)" + ], + "best_practices": [ + "Set powerAppName to the exact display name from the Power Apps maker portal", + "Declare powerAppName in when the value is supplied at runtime", + "Use headless mode for CI/CD" + ] + }, + + "MSPowerPageConnect": { + "apiId": "com.provar.plugins.forcedotcom.core.ui.NitroXConnect:ms-powerpage", + "description": "Connect to a Microsoft Power Page via the NitroX connector. Use INSTEAD of UiConnect when targeting Microsoft Power Pages.", + "category": "UI", + "required_arguments": [ + { + "id": "connectionName", + "type": "string", + "description": "Provar connection profile name configured for the Power Pages tenant" + }, + { + "id": "resultName", + "type": "string", + "description": "Variable name that downstream UI steps reference via uiConnectionName" + }, + { + "id": "resultScope", + "type": "string", + "default": "Test", + "description": "Scope of the connection result (typically 'Test')" + }, + { + "id": "environment", + "type": "string", + "description": "Power Platform environment that hosts the page. May be left empty if declared as a runtime-bound in ." + }, + { + "id": "powerPageName", + "type": "string", + "description": "Display name of the target Power Page site. May be left empty if declared as a runtime-bound in ." + } + ], + "optional_arguments": [ + { "id": "reuseConnectionName", "type": "string" }, + { "id": "privateBrowsingMode", "type": "boolean", "default": "false" }, + { "id": "webBrowser", "type": "string", "default": "Chrome_Headless" } + ], + "validation_rules": [ + "Use INSTEAD of UiConnect/ApexConnect for Microsoft Power Pages targets", + "resultName is required so downstream UI steps can set uiConnectionName", + "environment and powerPageName must each be populated OR declared as runtime parameters in " + ], + "best_practices": [ + "Set environment and powerPageName so the test lands on the right Power Page site", + "Declare environment / powerPageName in when supplied at runtime", + "Use headless mode for CI/CD" + ] + }, + + "UiWithScreen": { + "apiId": "com.provar.plugins.forcedotcom.core.ui.UiWithScreen", + "description": "Navigate to and interact with Salesforce screen", + "category": "UI", + "required_arguments": [ + { + "id": "screenName", + "type": "string", + "description": "Screen name (e.g., 'Account Home', 'Account New')" + }, + { + "id": "uiConnectionName", + "type": "string" + }, + { + "id": "target", + "type": "uiTarget", + "description": "Screen target (sf:ui:target?object=X&action=Y)" + } + ], + "optional_arguments": [ + { + "id": "navigate", + "type": "string", + "default": "Always", + "description": "Always for first screen, Dont for subsequent" + }, + { + "id": "targetDescription", + "type": "string" + }, + { + "id": "windowSelection", + "type": "string", + "default": "Default" + }, + { + "id": "windowSize", + "type": "string", + "default": "Default" + }, + { + "id": "sfUiTargetResultName", + "type": "string", + "description": "Store record ID (required for New screens)" + }, + { + "id": "sfUiTargetResultScope", + "type": "string", + "default": "Test" + }, + { + "id": "sfUiTargetObjectId", + "type": "variable", + "description": "Record ID to edit/view (required for Edit/View screens when navigate=Always)" + } + ], + "substeps": true, + "naming_convention": "On {ObjectName} {ScreenType} screen", + "validation_rules": [ + "Must have UiConnect first", + "First screen: navigate=Always", + "Subsequent screens: navigate=Dont", + "New screens: must store sfUiTargetResultName", + "Edit/View screens with navigate=Always: must have sfUiTargetObjectId" + ], + "best_practices": [ + "Follow naming convention", + "Store record IDs from New screens", + "Group related UI actions in substeps" + ] + }, + + "UiDoAction": { + "apiId": "com.provar.plugins.forcedotcom.core.ui.UiDoAction", + "description": "Perform UI action (click, set, select, etc.)", + "category": "UI", + "required_arguments": [ + { + "id": "locator", + "type": "uiLocator", + "description": "UI element locator" + }, + { + "id": "interaction", + "type": "uiInteraction", + "description": "Interaction type (click, set, check, etc.)" + } + ], + "optional_arguments": [ + { + "id": "value", + "type": "value", + "description": "Value for set/select interactions" + }, + { + "id": "captureBefore", + "type": "boolean", + "default": "false" + }, + { + "id": "captureAfter", + "type": "boolean", + "default": "false" + } + ], + "parent_required": "UiWithScreen", + "validation_rules": [ + "Must be inside UiWithScreen substeps", + "value required for set/select interactions", + "Picklist: use ui:interaction?name=set", + "Lead Conversion buttons: MUST use name=submitConvert and action=submitConvert (NOT Convert, ConvertButton, or action=convert)", + "Dialog fields (e.g., OpportunityName): MUST bind to the TARGET object (object=Opportunity with field=name, NOT object=Lead with field=OpportunityName)" + ], + "locator_format": { + "description": "uiLocator URI format for UiDoAction steps", + "action_buttons": { + "format": "ui:locator?name=&binding=sf%3Aui%3Abinding%3Aobject%3Faction%3D", + "examples": { + "lead_convert_button": "ui:locator?name=submitConvert&binding=sf%3Aui%3Abinding%3Aobject%3Faction%3DsubmitConvert", + "save_button": "ui:locator?name=save&binding=sf%3Aui%3Abinding%3Aobject%3Fobject%3DAccount%26action%3Dsave", + "new_button": "ui:locator?name=New&binding=sf%3Aui%3Abinding%3Aobject%3Fobject%3DAccount%26action%3DNew", + "continue_button_record_type_selection": "ui:locator?name=save&path=selectRecordType", + "cancel_button": "ui:locator?name=cancel&binding=sf%3Aui%3Abinding%3Aobject%3Faction%3Dcancel", + "record_type_field": "ui:locator?name=RecordType&binding=sf%3Aui%3Abinding%3Aobject%3Fobject%3D%7BtargetUrl%3Aobject%7D%26field%3DRecordTypeId" + } + }, + "dialog_fields": { + "format": "ui:locator?name=&binding=sf%3Aui%3Abinding%3Aobject%3Ffield%3D%26object%3D", + "examples": { + "opportunity_name_in_lead_convert": "ui:locator?name=name&binding=sf%3Aui%3Abinding%3Aobject%3Ffield%3Dname%26object%3DOpportunity" + }, + "note": "Dialog fields reference the TARGET object being created, not the source object" + } + }, + "best_practices": [ + "Use descriptive locator names", + "Validate values before setting", + "Add screenshots for failures" + ] + }, + + "UiAssert": { + "apiId": "com.provar.plugins.forcedotcom.core.ui.UiAssert", + "description": "Assert UI element state or value", + "category": "UI", + "required_arguments": [ + { + "id": "locator", + "type": "uiLocator" + }, + { + "id": "assertionType", + "type": "string", + "description": "Type of assertion (EqualTo, Contains, etc.)" + } + ], + "optional_arguments": [ + { + "id": "expectedValue", + "type": "value" + }, + { + "id": "continueOnFailure", + "type": "boolean", + "default": "false" + } + ], + "parent_required": "UiWithScreen", + "validation_rules": [ + "Must be inside UiWithScreen substeps", + "Currency fields: format as $1,000.00", + "Phone fields: format as (123) 456-7890", + "Date/time: use Contains or EqualTo based on format" + ], + "best_practices": [ + "Format values to match UI display", + "Use appropriate assertion type", + "Assert critical values" + ] + }, + + "UiNavigate": { + "apiId": "com.provar.plugins.forcedotcom.core.ui.UiNavigate", + "description": "Navigate to URL", + "category": "UI", + "required_arguments": [ + { + "id": "url", + "type": "string", + "description": "URL to navigate to" + } + ], + "optional_arguments": [ + { + "id": "uiConnectionName", + "type": "string" + } + ], + "validation_rules": ["url must be valid", "Use for non-Salesforce sites or direct URLs"], + "best_practices": [ + "Prefer UiWithScreen for Salesforce navigation", + "Use for external sites", + "Wait for page load after navigation" + ] + }, + + "UiFill": { + "apiId": "com.provar.plugins.forcedotcom.core.ui.UiFill", + "description": "Auto-fill form fields", + "category": "UI", + "required_arguments": [ + { + "id": "formLocator", + "type": "uiLocator", + "description": "Form container locator" + } + ], + "optional_arguments": [ + { + "id": "fieldValues", + "type": "object", + "description": "Field name/value pairs" + } + ], + "parent_required": "UiWithScreen", + "validation_rules": ["Must be inside UiWithScreen substeps", "Field names must match form fields"], + "best_practices": ["Use for multi-field forms", "Verify all fields filled", "Handle required fields first"] + }, + + "UiWithRow": { + "apiId": "com.provar.plugins.forcedotcom.core.ui.UiWithRow", + "description": "Work with specific table/list row", + "category": "UI", + "required_arguments": [ + { + "id": "tableLocator", + "type": "uiLocator", + "description": "Table/list locator" + }, + { + "id": "rowSelector", + "type": "string", + "description": "Row selection criteria" + } + ], + "optional_arguments": [], + "substeps": true, + "parent_required": "UiWithScreen", + "validation_rules": [ + "Must be inside UiWithScreen substeps", + "rowSelector must be valid", + "Must contain substeps" + ], + "best_practices": [ + "Use for table interactions", + "Specify unique row selector", + "Handle multiple rows with ForEach" + ] + }, + + "UiHandleAlert": { + "apiId": "com.provar.plugins.forcedotcom.core.ui.UiHandleAlert", + "description": "Handle browser alerts and pop-ups", + "category": "UI", + "required_arguments": [ + { + "id": "action", + "type": "string", + "description": "Action to take (Accept, Dismiss, GetText)" + } + ], + "optional_arguments": [ + { + "id": "resultName", + "type": "string", + "description": "Variable to store alert text" + } + ], + "validation_rules": ["action must be valid (Accept, Dismiss, GetText)", "Store text when using GetText"], + "best_practices": ["Handle alerts promptly", "Validate alert text", "Use in Try/Catch for optional alerts"] + } + }, + + "Utility": { + "description": "String manipulation, list operations, and file I/O utilities", + + "Read": { + "apiId": "com.provar.plugins.bundled.apis.io.Read", + "description": "Read data from Excel/CSV file", + "category": "Utility", + "required_arguments": [ + { + "id": "dataUrl", + "type": "string", + "description": "Path to Excel/CSV file" + }, + { + "id": "resultName", + "type": "string", + "description": "Variable to store read data" + } + ], + "optional_arguments": [ + { + "id": "worksheetName", + "type": "string", + "description": "Excel worksheet name" + }, + { + "id": "valuesRange", + "type": "string", + "description": "Cell range to read (e.g., A1:D10)" + }, + { + "id": "namesLocation", + "type": "string", + "default": "FirstRowOfRange", + "description": "Where column names are located" + } + ], + "validation_rules": [ + "dataUrl must be valid file path", + "File must exist and be readable", + "valuesRange must be valid Excel range" + ], + "best_practices": ["Use for data-driven testing", "Store in meaningful variable", "Validate data after reading"] + }, + + "Write": { + "apiId": "com.provar.plugins.bundled.apis.io.Write", + "description": "Write data to file", + "category": "Utility", + "required_arguments": [ + { + "id": "data", + "type": "variable", + "description": "Data to write" + }, + { + "id": "dataUrl", + "type": "string", + "description": "Output file path" + } + ], + "optional_arguments": [ + { + "id": "updateValueRange", + "type": "string" + }, + { + "id": "updateMatchType", + "type": "string" + }, + { + "id": "updateMatchLocator", + "type": "string" + } + ], + "validation_rules": ["data must be valid variable", "dataUrl must be writable path"], + "best_practices": ["Use for test result recording", "Ensure output directory exists", "Handle write failures"] + }, + + "Replace": { + "apiId": "com.provar.plugins.bundled.apis.string.Replace", + "description": "Replace text in string", + "category": "Utility", + "required_arguments": [ + { + "id": "input", + "type": "string", + "description": "Input string" + }, + { + "id": "searchString", + "type": "string", + "description": "Text to find" + }, + { + "id": "replacement", + "type": "string", + "description": "Replacement text" + }, + { + "id": "resultName", + "type": "string" + } + ], + "optional_arguments": [ + { + "id": "caseSensitive", + "type": "boolean", + "default": "true" + }, + { + "id": "replaceAll", + "type": "boolean", + "default": "true", + "description": "Replace all occurrences vs first only" + } + ], + "validation_rules": ["searchString should not be empty", "Store result in variable"], + "best_practices": ["Use for string formatting", "Consider regex for complex patterns", "Test with edge cases"] + }, + + "Split": { + "apiId": "com.provar.plugins.bundled.apis.string.Split", + "description": "Split string into array", + "category": "Utility", + "required_arguments": [ + { + "id": "input", + "type": "string", + "description": "String to split" + }, + { + "id": "delimiter", + "type": "string", + "description": "Delimiter character/string" + }, + { + "id": "resultName", + "type": "string", + "description": "Variable to store array" + } + ], + "optional_arguments": [ + { + "id": "maxSplits", + "type": "decimal", + "description": "Maximum number of splits" + } + ], + "validation_rules": ["delimiter should not be empty", "Store result as list"], + "best_practices": ["Use for parsing delimited data", "Iterate results with ForEach", "Handle empty splits"] + }, + + "Match": { + "apiId": "com.provar.plugins.bundled.apis.string.Match", + "description": "Pattern matching and regex operations", + "category": "Utility", + "required_arguments": [ + { + "id": "input", + "type": "string", + "description": "Input string to match against" + }, + { + "id": "pattern", + "type": "string", + "description": "Regex pattern or match string" + }, + { + "id": "resultName", + "type": "string" + } + ], + "optional_arguments": [ + { + "id": "isRegex", + "type": "boolean", + "default": "false", + "description": "Treat pattern as regex" + }, + { + "id": "caseSensitive", + "type": "boolean", + "default": "true" + } + ], + "validation_rules": ["pattern must be valid regex if isRegex=true", "Store match results"], + "best_practices": [ + "Use for validation and extraction", + "Test regex patterns separately", + "Handle no-match scenarios" + ] + }, + + "ListCompare": { + "apiId": "com.provar.plugins.bundled.apis.list.ListCompareListCompare", + "description": "Compare two lists", + "category": "Utility", + "required_arguments": [ + { + "id": "list1", + "type": "variable", + "description": "First list" + }, + { + "id": "list2", + "type": "variable", + "description": "Second list" + }, + { + "id": "resultName", + "type": "string", + "description": "Variable to store comparison results" + } + ], + "optional_arguments": [ + { + "id": "compareType", + "type": "string", + "default": "Equal", + "description": "Comparison type (Equal, Contains, etc.)" + } + ], + "validation_rules": ["Both lists must be valid", "Store comparison result"], + "best_practices": [ + "Use for data validation", + "Compare sorted lists for order-independent comparison", + "Handle different list sizes" + ] + } + } + }, + + "value_types": { + "description": "Standard value class types used in Provar test steps. Based on corpus analysis of 329,424 value elements across 1,451 test files.", + + "_valid_class_values": { + "description": "Complete list of valid 'class' attribute values for elements", + "literal_and_container": ["value", "variable", "compound", "funcCall", "valueList", "namedValues"], + "ui_types": ["uiWait", "uiLocator", "uiTarget", "uiInteraction"], + "data_sources": ["restTarget", "excelTarget", "csvTarget", "url", "template"], + "expression_operators": ["add", "sub", "mult", "div", "eq", "ne", "gt", "lt", "ge", "le", "and", "or", "match"], + "invalid_never_use": ["null"] + }, + + "_valid_valueClass_values": { + "description": "Complete list of valid 'valueClass' attribute values when class='value'", + "values": ["string", "boolean", "decimal", "id", "date", "dateTime"], + "invalid_never_use": ["null", "integer", "number", "text"] + }, + + "string": { + "class": "value", + "valueClass": "string", + "description": "String literal value" + }, + "boolean": { + "class": "value", + "valueClass": "boolean", + "valid_values": ["true", "false"] + }, + "decimal": { + "class": "value", + "valueClass": "decimal", + "description": "Numeric value" + }, + "id": { + "class": "value", + "valueClass": "id", + "description": "Salesforce ID or GUID" + }, + "date": { + "class": "value", + "valueClass": "date", + "description": "Date value" + }, + "dateTime": { + "class": "value", + "valueClass": "dateTime", + "description": "DateTime value" + }, + "variable": { + "class": "variable", + "description": "Reference to test variable", + "syntax": "{VariableName} or {VariableName.FieldName} or {VariableName[index].FieldName}", + "note": "Array indices are 1-based, not 0-based", + "child_element": { + "path": { + "required_attribute": "element", + "description": "Variable path (e.g., VariableName, VariableName.Field)" + } + } + }, + "expression": { + "class": "expression", + "description": "Dynamic expression evaluation" + }, + "uiLocator": { + "class": "uiLocator", + "description": "UI element locator", + "uri_format": "ui:locator?..." + }, + "uiTarget": { + "class": "uiTarget", + "description": "UI navigation target", + "uri_format": "sf:ui:target?object=X&action=Y" + }, + "uiInteraction": { + "class": "uiInteraction", + "description": "UI interaction type", + "uri_format": "ui:interaction?name=set|click|check|..." + } + }, + + "common_patterns": { + "description": "Common test case patterns and best practices", + + "apex_crud_pattern": { + "description": "Standard Salesforce Apex CRUD operation pattern", + "category": "Salesforce", + "steps": [ + { + "step": "ApexConnect", + "purpose": "Establish connection with autoCleanup=true" + }, + { + "step": "ApexCreateObject or ApexUpdateObject", + "purpose": "Perform CRUD operation, store result ID" + }, + { + "step": "ApexSoqlQuery (optional)", + "purpose": "Verify operation success" + }, + { + "step": "Automatic cleanup", + "purpose": "autoCleanup deletes created objects" + } + ], + "best_practices": [ + "Enable autoCleanup for automatic cleanup", + "Store object IDs for verification", + "Use SOQL to verify operations" + ] + }, + + "ui_interaction_pattern": { + "description": "Standard UI interaction and verification pattern", + "category": "UI", + "steps": [ + { + "step": "UiConnect", + "purpose": "Establish UI connection" + }, + { + "step": "UiWithScreen (navigate=Always)", + "purpose": "Navigate to target screen", + "substeps": [ + { + "step": "UiDoAction", + "purpose": "Interact with UI elements" + }, + { + "step": "UiAssert", + "purpose": "Verify UI state" + } + ] + }, + { + "step": "UiWithScreen (navigate=Dont)", + "purpose": "Subsequent screens", + "substeps": ["More UI interactions"] + } + ], + "best_practices": [ + "First screen: navigate=Always", + "Subsequent screens: navigate=Dont", + "Store record IDs from New screens", + "Group related actions in substeps" + ] + }, + + "data_driven_pattern": { + "description": "Data-driven testing with external data files", + "category": "Control + Utility", + "steps": [ + { + "step": "Read", + "purpose": "Read test data from Excel/CSV" + }, + { + "step": "ForEach", + "purpose": "Loop through data rows", + "substeps": [ + { + "step": "SetValues", + "purpose": "Extract current row data" + }, + { + "step": "Connection + Operations", + "purpose": "Execute test with data" + }, + { + "step": "Assert", + "purpose": "Verify results" + }, + { + "step": "Write", + "purpose": "Record results back to file" + } + ] + } + ], + "best_practices": ["Use Read to load test data", "Use ForEach to iterate rows", "Write results for reporting"] + }, + + "agentforce_conversation_pattern": { + "description": "Agentforce AI agent conversation testing pattern", + "category": "ProvarAI + Data", + "steps": [ + { + "step": "Read", + "purpose": "Load conversation test data" + }, + { + "step": "WebConnect", + "purpose": "Establish agent connection" + }, + { + "step": "ApexConnect (optional)", + "purpose": "Connect to Salesforce for record validation" + }, + { + "step": "AIAgentSession", + "purpose": "Create agent session, store session ID" + }, + { + "step": "ForEach (conversation data)", + "purpose": "Iterate through test scenarios", + "substeps": [ + { + "step": "GenerateUtterance (optional)", + "purpose": "Generate test utterances for intent" + }, + { + "step": "AIAgentConversation", + "purpose": "Send message, receive response" + }, + { + "step": "IntentValidator (optional)", + "purpose": "Validate agent understood intent" + }, + { + "step": "ApexSoqlQuery (optional)", + "purpose": "Verify agent created/updated records" + }, + { + "step": "LogForCleanup (optional)", + "purpose": "Log created records for cleanup" + }, + { + "step": "Write", + "purpose": "Record conversation results" + } + ] + }, + { + "step": "AIAgentConversation (endSession=true)", + "purpose": "End agent session" + } + ], + "best_practices": [ + "Test multiple conversation paths", + "Validate agent responses", + "Log records for cleanup", + "End session explicitly" + ] + }, + + "callable_test_case_pattern": { + "description": "Calling reusable test cases with caseCall element", + "category": "Design + Control", + "usage": { + "caseCall_element": { + "description": "Reference to another test case to execute", + "required_attributes": [ + { + "name": "testCaseId", + "description": "GUID of the callable test case" + }, + { + "name": "testCasePath", + "description": "Relative path to the .testcase file" + }, + { + "name": "testItemId", + "description": "Unique test item ID" + } + ], + "optional_attributes": [ + { + "name": "guid", + "description": "Unique GUID for this call instance" + } + ] + }, + "callable_test_requirements": { + "visibility": "Internal or Callable", + "description": "Test case being called must have visibility='Internal' or visibility='Callable'", + "note": "Public tests cannot be called from other tests" + } + }, + "example": "", + "best_practices": [ + "Use for reusable test logic (setup, cleanup, common flows)", + "Ensure called test has visibility='Internal'", + "Pass parameters using elements if test has ", + "Use for modular test design" + ], + "validation_rules": [ + "testCaseId must match an existing test case GUID", + "testCasePath must be valid relative path", + "Called test must have visibility='Internal' or visibility='Callable'", + "Parameter types must match called test signature" + ] + }, + + "parameterized_test_pattern": { + "description": "Test case with input parameters for reusability", + "category": "Design", + "structure": { + "params_section": { + "description": "Define test case parameters in element", + "param_attributes": [ + { + "name": "name", + "required": true, + "description": "Parameter name" + }, + { + "name": "passwordVariableAllowed", + "required": false, + "description": "Allow password variables" + }, + { + "name": "title", + "required": false, + "description": "Display title" + } + ], + "param_child_elements": [ + { + "name": "summary", + "description": "Parameter description" + }, + { + "name": "type", + "description": "Parameter data type (textType, booleanType, etc.)" + } + ] + }, + "args_section": { + "description": "Default parameter values in element", + "note": "Arguments can be overridden when calling the test via caseCall" + } + }, + "example_usage": { + "define_params": "...", + "provide_defaults": "123", + "call_with_params": "456" + }, + "best_practices": [ + "Define clear parameter names", + "Provide default values in ", + "Document parameter purpose in summary", + "Use for reusable callable tests" + ] + }, + + "bdd_scenario_pattern": { + "description": "BDD (Behavior-Driven Development) test scenario structure", + "category": "Design", + "steps": [ + { + "step": "Given", + "purpose": "Set up preconditions", + "substeps": ["Setup steps, data creation"] + }, + { + "step": "And (optional)", + "purpose": "Additional Given conditions" + }, + { + "step": "When", + "purpose": "Perform test action", + "substeps": ["Action steps"] + }, + { + "step": "And (optional)", + "purpose": "Additional When actions" + }, + { + "step": "Then", + "purpose": "Verify outcomes", + "substeps": ["Assertion steps"] + }, + { + "step": "And/But (optional)", + "purpose": "Additional Then verifications" + } + ], + "best_practices": [ + "One When per scenario when possible", + "Focus Given on state, not actions", + "Use Then for observable outcomes", + "Limit And/But chains to 3-5" + ] + }, + + "database_testing_pattern": { + "description": "External database testing pattern", + "category": "Data", + "steps": [ + { + "step": "DbConnect", + "purpose": "Establish database connection" + }, + { + "step": "DbInsert or DbUpdate", + "purpose": "Set up test data" + }, + { + "step": "SqlQuery or DbRead", + "purpose": "Verify data state" + }, + { + "step": "DbDelete", + "purpose": "Clean up test data" + } + ], + "best_practices": ["Always use WHERE clauses", "Test queries with SELECT first", "Clean up in Finally block"] + }, + + "rest_api_testing_pattern": { + "description": "REST API testing pattern", + "category": "Data", + "steps": [ + { + "step": "WebConnect", + "purpose": "Establish API connection" + }, + { + "step": "RestRequest (GET)", + "purpose": "Retrieve initial state" + }, + { + "step": "RestRequest (POST/PUT)", + "purpose": "Modify data" + }, + { + "step": "RestRequest (GET)", + "purpose": "Verify changes" + }, + { + "step": "Assert", + "purpose": "Validate response data" + } + ], + "best_practices": [ + "Validate response status codes", + "Store responses for assertions", + "Handle authentication properly" + ] + } + } +} diff --git a/src/mcp/server.ts b/src/mcp/server.ts index 1f9de722..309d3e04 100644 --- a/src/mcp/server.ts +++ b/src/mcp/server.ts @@ -177,7 +177,9 @@ export function createProvarMcpServer(config: ServerConfig): McpServer { registerAllPrompts(server); // ── Documentation resources ────────────────────────────────────────────────── - const docsDir = resolveDocsDir(dirname(fileURLToPath(import.meta.url))); + const moduleDir = dirname(fileURLToPath(import.meta.url)); + const docsDir = resolveDocsDir(moduleDir); + const rulesDir = resolveRulesDir(moduleDir); server.resource( 'provar-nitrox-component-catalog', @@ -257,6 +259,50 @@ export function createProvarMcpServer(config: ServerConfig): McpServer { } ); + server.resource( + 'provar-validation-rules', + 'provar://docs/validation-rules', + { + description: + 'Canonical registry of every Provar test-case validation rule across both layers: the structural validity rules (Layer 1, gate is_valid) and the best-practice rules (Layer 2, weighted quality_score). For each rule it lists the id, severity, weight, what it checks, and whether it gates is_valid (a critical best-practice violation does, via the validity bridge). Read this to understand why provar_testcase_validate returned a given issue or marked a test invalid vs needs_improvement.', + mimeType: 'text/markdown', + }, + () => { + try { + const text = readFileSync(join(docsDir, 'VALIDATION_RULE_REGISTRY.md'), 'utf-8'); + return { + contents: [{ uri: 'provar://docs/validation-rules', mimeType: 'text/markdown', text }], + }; + } catch { + return { + contents: [ + { + uri: 'provar://docs/validation-rules', + mimeType: 'text/markdown', + text: '# Provar Validation Rule Registry\n\nRegistry not found. If you are developing from source, run `node scripts/build-validation-rule-registry.cjs` then rebuild. Otherwise, reinstall or upgrade the plugin/package and try again.', + }, + ], + }; + } + } + ); + + server.resource( + 'provar-test-step-schema', + 'provar://schema/test-step', + { + description: + 'Structured JSON reference describing the full Provar test case XML structure: the root, the generic shape, every supported step type organised by category (Control, Data, Design, ProvarAI, ProvarLabs, Salesforce, UI, Utility) with required/optional arguments and validation rules, plus value-class types and common patterns. This is a Provar-specific schema reference (domain-keyed: testCase / apiCalls / value_types), NOT a standards-compliant constraint JSON Schema — read it to author or validate test-step XML with exact argument names and structures, not to drive a JSON-Schema validator.', + mimeType: 'application/json', + }, + () => { + const text = readTestStepSchema(rulesDir); + return { + contents: [{ uri: 'provar://schema/test-step', mimeType: 'application/json', text }], + }; + } + ); + server.resource( 'provar-tool-guide', 'provar://docs/tool-guide', @@ -335,3 +381,39 @@ export function readCatalogSource(docsDir: string): string { ); } } + +/** + * Resolve the rules directory for bundled MCP JSON resources. The rules/ dir is + * a sibling of this module in both compiled output (lib/mcp/rules) and dev mode + * (src/mcp/rules); fall back one level up if a future layout moves it. + */ +export function resolveRulesDir(currentDir: string): string { + const sibling = join(currentDir, 'rules'); + return existsSync(sibling) ? sibling : join(currentDir, '..', 'rules'); +} + +/** + * Read provar_test_step_schema.json from the rules directory and return it as a + * JSON string. The file is parsed once to verify it is valid JSON before being + * returned verbatim, so the resource never advertises `application/json` while + * serving a truncated/corrupted body; on a missing or unparseable file it + * returns a small `schema_not_found` fallback object, mirroring the + * graceful-degradation shape of the other resource handlers. + */ +export function readTestStepSchema(rulesDir: string): string { + try { + const raw = readFileSync(join(rulesDir, 'provar_test_step_schema.json'), 'utf-8'); + JSON.parse(raw); // validate only — return the original text untouched if it parses + return raw; + } catch { + return JSON.stringify( + { + error: 'schema_not_found', + message: + 'provar_test_step_schema.json not found. If you are developing from source, rebuild the package; otherwise reinstall or upgrade the plugin and try again.', + }, + null, + 2 + ); + } +} diff --git a/src/mcp/tools/bestPracticesEngine.ts b/src/mcp/tools/bestPracticesEngine.ts index 184867d0..e4316f5d 100644 --- a/src/mcp/tools/bestPracticesEngine.ts +++ b/src/mcp/tools/bestPracticesEngine.ts @@ -617,7 +617,9 @@ function validateDetectDuplicatesLiterals(tc: XmlNode, rule: BPRule): BPViolatio if (!valElem || typeof valElem !== 'object') continue; if ((valElem['@_class'] as string | undefined) !== 'value') continue; // skip variables/compounds - const text = ((valElem['#text'] as string | undefined) ?? '').trim(); + // Read through nodeText: fast-xml-parser yields a NUMBER for a numeric tag + // value (e.g. 123), so a bare `.trim()` would throw. + const text = nodeText(valElem); if (!text || text.length <= 3) continue; if (DEFAULT_LITERAL_VALUES.has(text)) continue; if (text.toLowerCase() === 'true' || text.toLowerCase() === 'false') continue; @@ -935,11 +937,7 @@ function validateUiActionNestingStructure(tc: XmlNode, rule: BPRule): BPViolatio const tid = call['@_testItemId'] as string | undefined; const shortApi = apiId.split('.').pop() ?? apiId; const tidSuffix = tid ? ` (testItemId=${tid})` : ''; - const requiredContainer = verdict.includes('UiWithRow') - ? 'UiWithRow' - : verdict.includes('UiWithScreen') - ? 'UiWithScreen' - : 'UiWithScreen'; + const requiredContainer = verdict.includes('UiWithRow') ? 'UiWithRow' : 'UiWithScreen'; const message = `${shortApi} '${title}' is ${verdict} - must be nested inside a parent ` + `${requiredContainer}'s block${tidSuffix}`; @@ -1091,6 +1089,1066 @@ function validateNitroxVariantArgRequired(tc: XmlNode, rule: BPRule): BPViolatio return violations; } +/** + * Value `class` attributes that, when present on a condition/expression argument, + * are themselves meaningful content — comparison and boolean-logic operators that + * Provar emits for `If`/`DoWhile`/`WaitFor` conditions (e.g. `{Count(Rows) > 0}` + * is stored as ``). Mirrors the backend operator allow-list. + */ +const MEANINGFUL_VALUE_OPERATOR_CLASSES: ReadonlySet = new Set([ + 'gt', + 'lt', + 'eq', + 'ne', + 'ge', + 'le', + 'and', + 'or', + 'not', +]); + +/** + * True when an `` carries a *meaningful* value, mirroring the Quality + * Hub `MustContainArgumentValidator` content checks exactly. A `variable` value + * counts only if it references a `` (or has non-empty text) — a bare + * `` is effectively empty; `funcCall` and the comparison + * / logic operator classes always count; `compound` counts only if `` has + * children; any other (simple) value counts only if it has non-empty text. An + * `` with no `` child, or an empty ``, is NOT meaningful + * and is treated as a missing required argument. + */ +function argumentHasMeaningfulValue(arg: XmlNode): boolean { + for (const value of toArr(arg['value'] as XmlNode | string | Array)) { + if (value == null) continue; + if (typeof value === 'string') { + if (value.trim().length > 0) return true; + continue; + } + if (typeof value !== 'object') continue; + const v = value; + const vClass = (v['@_class'] as string | undefined) ?? ''; + // nodeText coerces a numeric #text to string first — a bare `.trim()` throws on it. + const text = nodeText(v); + + if (vClass === 'variable') { + if (v['path'] != null || text.length > 0) return true; + continue; // bare — not meaningful + } + if (vClass === 'funcCall' || MEANINGFUL_VALUE_OPERATOR_CLASSES.has(vClass)) return true; + if (vClass === 'compound') { + const parts = v['parts']; + if (parts && typeof parts === 'object' && Object.keys(parts).some((k) => !k.startsWith('@_'))) return true; + continue; + } + if (text.length > 0) return true; + } + return false; +} + +/** Find an `` for a call, tolerating both the `` wrapper and direct children. */ +function findArgumentById(call: XmlNode, argId: string): XmlNode | undefined { + return getCallArguments(call).find((a) => a['@_id'] === argId) ?? getArguments(call).find((a) => a['@_id'] === argId); +} + +/** Human-readable step label for a violation message: `'' (testItemId=N)`. */ +function stepLabel(call: XmlNode): string { + const label = (call['@_title'] as string | undefined) ?? (call['@_name'] as string | undefined) ?? '(unnamed)'; + const tid = call['@_testItemId'] as string | undefined; + return `'${label}'${tid ? ` (testItemId=${tid})` : ''}`; +} + +/** + * True when `call` satisfies a `mustContainArgument` requirement — either the + * argument is present with a meaningful value, or (for `If`/`DoWhile` conditions) + * the legacy condition-in-title format is used. + */ +function callSatisfiesRequiredArg(call: XmlNode, requiredArg: string, conditionInTitleAllowed: boolean): boolean { + const arg = findArgumentById(call, requiredArg); + if (arg && argumentHasMeaningfulValue(arg)) return true; + if (!arg && conditionInTitleAllowed) { + const title = (call['@_title'] as string | undefined) ?? ''; + if (title.includes('{') && title.includes('}')) return true; // legacy condition-in-title format + } + return false; +} + +/** + * mustContainArgument — every apiCall whose apiId equals `check.apiId` must carry a + * populated ``. Faithful TypeScript port of the + * Quality Hub `MustContainArgumentValidator`, so the local (offline) result and + * the back-end agree: present-AND-non-empty semantics via + * {@link argumentHasMeaningfulValue} (an absent argument OR an empty + * ``/`` is a violation); exact apiId match (no substring / + * variant widening); the legacy exception where `If`/`DoWhile` may carry the + * condition in the step `title` (`If: {expr}`) instead of a `condition` argument; + * disabled steps are NOT skipped (a missing required argument is load/exec + * blocking regardless of the disabled flag); and one violation per rule (the + * back-end returns the first offender), so the weighted-deduction score stays in + * parity with the Lambda. The message still names every offending step without + * inflating `count`. + */ +function validateMustContainArgument(tc: XmlNode, rule: BPRule): BPViolation | null { + const targetApiId = (rule.check['apiId'] as string | undefined) ?? ''; + const requiredArg = (rule.check['argument'] as string | undefined) ?? ''; + if (!targetApiId || !requiredArg) return null; + + const conditionInTitleAllowed = + requiredArg === 'condition' && + (targetApiId === 'com.provar.plugins.bundled.apis.If' || + targetApiId === 'com.provar.plugins.bundled.apis.control.DoWhile'); + + const offending: string[] = []; + for (const call of getAllApiCalls(tc)) { + if (call['@_apiId'] !== targetApiId) continue; + if (callSatisfiesRequiredArg(call, requiredArg, conditionInTitleAllowed)) continue; + offending.push(stepLabel(call)); + } + + if (!offending.length) return null; + const apiName = targetApiId.split('.').pop() ?? targetApiId; + let msg = `${apiName} step missing required '${requiredArg}' argument: ${offending.slice(0, 2).join(', ')}`; + if (offending.length > 2) msg += ` (and ${offending.length - 2} more)`; + // Intentionally no `count`: the back-end reports a single violation per rule, so + // omitting count keeps the weighted-deduction score in parity with the Lambda. + return makeViolation(rule, msg); +} + +// ── Render / load-blocking validators (Tier 2) ─────────────────────────────── +// Faithful ports of the Quality Hub XMLRendering / InvalidValueClass / +// DateValueClassFormat / ApexConnect* / SetValuesInvalidElements validators. +// These check types map 1:1 to load-blocking rules (mostly `critical`) that +// stop a test case rendering or loading in the Provar IDE. Each returns a +// single BPViolation per rule (the back-end reports the first offender and sets +// `count = len(offenders)` only when > 1), so the weighted-deduction score stays +// in parity with the Lambda. + +const APEX_CONNECT_API_ID = 'com.provar.plugins.forcedotcom.core.testapis.ApexConnect'; +const SETVALUES_API_ID = 'com.provar.plugins.bundled.apis.control.SetValues'; + +/** + * Recursively collect every element that appears under `tag` anywhere in the + * subtree (the fast-xml-parser equivalent of ElementTree's `.//tag`). Returns + * the raw values (object, string, or — for repeated tags — each array member); + * callers filter to objects when they need attributes. Mirrors the back-end's + * descendant search so nested-step double-counting matches exactly. + */ +function collectElementsByTag(node: unknown, tag: string): unknown[] { + const out: unknown[] = []; + function walk(n: unknown): void { + if (!n || typeof n !== 'object') return; + if (Array.isArray(n)) { + for (const item of n) walk(item); + return; + } + for (const [k, v] of Object.entries(n as XmlNode)) { + if (k.startsWith('@_') || k === '#text') continue; + if (k === tag) for (const item of toArr(v)) out.push(item); + walk(v); + } + } + walk(node); + return out; +} + +/** Object-form `` descendants of a node (string-only text values are skipped). */ +function collectValueElements(node: unknown): XmlNode[] { + return collectElementsByTag(node, 'value').filter((v): v is XmlNode => v != null && typeof v === 'object'); +} + +/** + * Trimmed text of an element's `#text`, coercing to string first. fast-xml-parser + * parses numeric tag text to a `number` (e.g. an epoch-millis date), so a raw + * `.trim()` on `#text` would throw — always read element text through here. + */ +function nodeText(node: XmlNode): string { + const t = node['#text']; + return t == null ? '' : String(t).trim(); +} + +/** Step context used in load-blocking violation messages, mirroring the back-end defaults. */ +function stepContext(call: XmlNode): { apiName: string; title: string; tid: string } { + const apiId = (call['@_apiId'] as string | undefined) ?? ''; + const apiName = apiId ? apiId.split('.').pop() ?? apiId : 'Unknown'; + const title = (call['@_title'] as string | undefined) ?? apiName; + const tid = (call['@_testItemId'] as string | undefined) ?? 'N/A'; + return { apiName, title, tid }; +} + +/** A `` element paired with the id of the `` that encloses it (`unknown` if none). */ +interface StepValueElem { + value: XmlNode; + argId: string; +} + +/** + * Every `` element within an apiCall, tagged with its parent ``. + * Replicates the back-end's "find the first enclosing argument" lookup so violation + * messages name the right argument. Values not inside any argument get `unknown`. + */ +function getStepValueElements(call: XmlNode): StepValueElem[] { + const argOf = new Map(); + for (const arg of collectElementsByTag(call, 'argument')) { + if (!arg || typeof arg !== 'object') continue; + const id = ((arg as XmlNode)['@_id'] as string | undefined) ?? 'unknown'; + for (const v of collectValueElements(arg)) if (!argOf.has(v)) argOf.set(v, id); + } + return collectValueElements(call).map((value) => ({ value, argId: argOf.get(value) ?? 'unknown' })); +} + +/** Trimmed text of an argument's direct `` child (handles string and object forms). */ +function directValueText(arg: XmlNode): string { + const v = Array.isArray(arg['value']) ? (arg['value'] as unknown[])[0] : arg['value']; + if (v == null) return ''; + if (typeof v === 'string') return v.trim(); + if (typeof v !== 'object') return String(v).trim(); + return nodeText(v as XmlNode); +} + +// RENDER-CASE-001 — the valueClass values that actually exist. This validator only +// inspects the `valueClass` attribute, and a full-corpus scan (AllPOCProjects) shows +// exactly SIX distinct valueClass values: string, boolean, decimal, id, date, dateTime +// — matching the back-end's VALID_VALUE_CLASSES. (The earlier list also carried +// `class="..."` tokens — variable/compound/funcCall/value/valueList/operators — and +// `integer`; none of those ever appear as a valueClass, so they were dead entries, and +// `id` — a real corpus valueClass — was missing. Coordinated with the QH back-end.) +const VALUE_CLASS_CASING_VALID: ReadonlySet = new Set([ + 'string', + 'boolean', + 'decimal', + 'id', + 'date', + 'datetime', +]); + +// Canonical Provar spelling for valueClasses whose correct form is NOT all-lowercase. +// The corpus uses camelCase `dateTime` exclusively (lowercase `datetime` never appears), +// so the casing check must expect `dateTime`; every other valueClass is all-lowercase. +const VALUE_CLASS_CANONICAL_CASE: Record = { datetime: 'dateTime' }; + +/** RENDER-CASE-001 — a known valueClass spelled with wrong case (e.g. `Boolean` → `boolean`). */ +function validateValueClassCasing(tc: XmlNode, rule: BPRule): BPViolation | null { + const offenders: Array<{ valueClass: string; expected: string }> = []; + for (const v of collectValueElements(tc)) { + const vc = v['@_valueClass'] as string | undefined; + if (!vc) continue; + const lower = vc.toLowerCase(); + if (!VALUE_CLASS_CASING_VALID.has(lower)) continue; + const expected = VALUE_CLASS_CANONICAL_CASE[lower] ?? lower; + if (vc !== expected) offenders.push({ valueClass: vc, expected }); + } + if (!offenders.length) return null; + const MAX = 5; + const reported = Math.min(offenders.length, MAX); + let msg = offenders + .slice(0, 3) + .map((o) => `valueClass='${o.valueClass}' should be '${o.expected}'`) + .join('; '); + if (reported > 3) msg += ` (+${reported - 3} more)`; + if (offenders.length > MAX) msg += ` (total: ${offenders.length})`; + return makeViolation(rule, msg, reported); +} + +const BOOLEAN_CASING_BAD: ReadonlySet = new Set(['True', 'False', 'TRUE', 'FALSE']); + +/** RENDER-BOOL-001 — `` text must be lowercase `true`/`false`. */ +function validateBooleanCasing(tc: XmlNode, rule: BPRule): BPViolation | null { + const offenders: string[] = []; + for (const v of collectValueElements(tc)) { + if (v['@_valueClass'] !== 'boolean') continue; + const text = nodeText(v); + if (BOOLEAN_CASING_BAD.has(text)) offenders.push(text); + } + if (!offenders.length) return null; + const MAX = 5; + const reported = Math.min(offenders.length, MAX); + let msg = `Boolean values must be lowercase: ${offenders + .slice(0, 3) + .map((t) => `'${t}' should be '${t.toLowerCase()}'`) + .join('; ')}`; + if (reported > 3) msg += ` (+${reported - 3} more)`; + if (offenders.length > MAX) msg += ` (total: ${offenders.length})`; + return makeViolation(rule, msg, reported); +} + +// VALUE-CLASS-001 — the back-end HARDCODES these sets and ignores the rule JSON's +// validClasses list, so we mirror the hardcoded sets (note `invalid` and +// `namedValues` are accepted, and `dateTime` is camelCase) to stay score-exact. +const VALID_VALUE_ELEMENT_CLASSES: ReadonlySet = new Set([ + 'value', + 'variable', + 'compound', + 'funcCall', + 'valueList', + 'namedValues', + 'uiWait', + 'uiLocator', + 'uiTarget', + 'uiInteraction', + 'restTarget', + 'excelTarget', + 'csvTarget', + 'url', + 'template', + 'add', + 'sub', + 'mult', + 'div', + 'eq', + 'ne', + 'gt', + 'lt', + 'ge', + 'le', + 'and', + 'or', + 'match', + 'invalid', +]); +const VALID_VALUE_CLASS_TYPES: ReadonlySet = new Set([ + 'string', + 'boolean', + 'decimal', + 'id', + 'date', + 'dateTime', +]); + +/** Classify one `` for VALUE-CLASS-001, or `null` when it is valid / unattributed. */ +function classifyInvalidValueClass(v: XmlNode): { kind: 'class' | 'valueClass'; bad: string } | null { + const cls = (v['@_class'] as string | undefined) ?? ''; + if (!cls) return null; + if (!VALID_VALUE_ELEMENT_CLASSES.has(cls)) return { kind: 'class', bad: cls }; + const vc = (v['@_valueClass'] as string | undefined) ?? ''; + if (cls === 'value' && vc && !VALID_VALUE_CLASS_TYPES.has(vc)) return { kind: 'valueClass', bad: vc }; + return null; +} + +/** VALUE-CLASS-001 — `` must use a valid class (and valueClass when class="value"). */ +function validateInvalidValueClass(tc: XmlNode, rule: BPRule): BPViolation | null { + const offenders: Array<{ + argId: string; + ctx: ReturnType; + kind: 'class' | 'valueClass'; + bad: string; + }> = []; + for (const call of getAllApiCalls(tc)) { + const ctx = stepContext(call); + for (const { value, argId } of getStepValueElements(call)) { + const c = classifyInvalidValueClass(value); + if (c) offenders.push({ argId, ctx, kind: c.kind, bad: c.bad }); + } + } + if (!offenders.length) return null; + const f = offenders[0]; + const message = + f.kind === 'class' + ? `Step '${f.ctx.title}' has invalid class="${f.bad}" in argument '${f.argId}'. Valid class values: value, ` + + 'variable, compound, funcCall, valueList, uiWait, uiLocator, uiTarget, etc. For empty arguments, omit ' + + ` entirely: (testItemId=${f.ctx.tid})` + : `Step '${f.ctx.title}' has invalid valueClass="${f.bad}" in argument '${f.argId}'. Valid valueClass values: ` + + `string, boolean, decimal, id, date, dateTime. (testItemId=${f.ctx.tid})`; + return makeViolation(rule, message, offenders.length); +} + +/** RENDER-DATE-VALUECLASS-001 — `valueClass="date"|"dateTime"` text must be an epoch-millis integer. */ +function validateDateValueClassFormat(tc: XmlNode, rule: BPRule): BPViolation | null { + const offenders: Array<{ argId: string; ctx: ReturnType; vc: string; text: string }> = []; + for (const call of getAllApiCalls(tc)) { + const ctx = stepContext(call); + for (const { value, argId } of getStepValueElements(call)) { + if (value['@_class'] !== 'value') continue; + const vc = (value['@_valueClass'] as string | undefined) ?? ''; + if (vc !== 'date' && vc !== 'dateTime') continue; + const text = nodeText(value); + if (!text || /^\d+$/.test(text)) continue; + offenders.push({ argId, ctx, vc, text: text.slice(0, 50) }); + } + } + if (!offenders.length) return null; + const f = offenders[0]; + const message = + `Step '${f.ctx.title}' uses valueClass='${f.vc}' with invalid string value '${f.text}' in argument ` + + `'${f.argId}'. valueClass='${f.vc}' requires an epoch timestamp (milliseconds), not a date string. This ` + + `causes test case loading failures in Provar. (testItemId=${f.ctx.tid})`; + return makeViolation(rule, message, offenders.length); +} + +/** Return the ApexConnect calls in a test case (exact apiId match, nested-aware). */ +function getApexConnectCalls(tc: XmlNode): XmlNode[] { + return getAllApiCalls(tc).filter((c) => c['@_apiId'] === APEX_CONNECT_API_ID); +} + +/** APEX-REUSE-CONN-001 — ApexConnect `reuseConnectionName` must be blank. */ +function validateApexConnectReuseConnection(tc: XmlNode, rule: BPRule): BPViolation | null { + const offenders: Array<{ ctx: ReturnType; value: string }> = []; + for (const call of getApexConnectCalls(tc)) { + const arg = getCallArguments(call).find((a) => a['@_id'] === 'reuseConnectionName'); + if (!arg) continue; + const text = directValueText(arg); + if (text) offenders.push({ ctx: stepContext(call), value: text }); + } + if (!offenders.length) return null; + const f = offenders[0]; + const message = + `ApexConnect step '${f.ctx.title}' has non-empty reuseConnectionName value '${f.value}'. The ` + + `reuseConnectionName argument should be left blank: (testItemId=${f.ctx.tid})`; + return makeViolation(rule, message, offenders.length); +} + +const APEX_CONNECT_VALID_ARGS_DEFAULT: readonly string[] = [ + 'connectionName', + 'resultName', + 'resultScope', + 'uiApplicationName', + 'quickUiLogin', + 'closeAllPrimaryTabs', + 'reuseConnectionName', + 'alreadyOpenBehaviour', + 'autoCleanup', + 'cleanupConnectionName', + 'logFileLocation', + 'connectionId', + 'enableObjectIdLogging', + 'privateBrowsingMode', + 'lightningMode', + 'username', + 'password', + 'securityToken', + 'environment', + 'webBrowser', +]; + +/** APEX-CONNECT-ARGS-001 — every ApexConnect `` must be in the valid whitelist. */ +function validateApexConnectValidArguments(tc: XmlNode, rule: BPRule): BPViolation | null { + const validIds = new Set( + (rule.check['validArgumentIds'] as string[] | undefined) ?? APEX_CONNECT_VALID_ARGS_DEFAULT + ); + const offenders: Array<{ ctx: ReturnType; id: string }> = []; + for (const call of getApexConnectCalls(tc)) { + if (!call['arguments'] || typeof call['arguments'] !== 'object') continue; + for (const arg of getCallArguments(call)) { + const id = arg['@_id'] as string | undefined; + if (id && !validIds.has(id)) offenders.push({ ctx: stepContext(call), id }); + } + } + if (!offenders.length) return null; + const f = offenders[0]; + const message = + `ApexConnect step '${f.ctx.title}' uses invalid argument ID(s): ${offenders.map((o) => o.id).join(', ')}. ` + + 'Only the documented ApexConnect argument IDs are valid (connectionName, resultName, resultScope, …, ' + + 'webBrowser). Leave unused arguments empty (e.g. ) rather than inventing IDs. ' + + `(testItemId=${f.ctx.tid})`; + return makeViolation(rule, message, offenders.length); +} + +/** APEX-CONNECT-CONNID-001 — `connectionId` value must use `valueClass="id"`, not string/other. */ +function validateApexConnectConnectionIdValueClass(tc: XmlNode, rule: BPRule): BPViolation | null { + const offenders: Array<{ ctx: ReturnType; wrong: string; value: string }> = []; + for (const call of getApexConnectCalls(tc)) { + const arg = getCallArguments(call).find((a) => a['@_id'] === 'connectionId'); + if (!arg) continue; + const ve = Array.isArray(arg['value']) ? (arg['value'] as unknown[])[0] : arg['value']; + if (!ve || typeof ve !== 'object') continue; + const v = ve as XmlNode; + const vc = (v['@_valueClass'] as string | undefined) ?? ''; + if (v['@_class'] !== 'value' || !vc || vc === 'id') continue; + offenders.push({ ctx: stepContext(call), wrong: vc, value: nodeText(v) }); + } + if (!offenders.length) return null; + const f = offenders[0]; + const message = + `ApexConnect step '${f.ctx.title}' uses incorrect valueClass='${f.wrong}' for connectionId argument. The ` + + "connectionId must use valueClass='id' with a GUID value, NOT valueClass='string'. Current value: " + + `'${f.value}'. If you have no specific connection GUID, leave it empty: ` + + `(testItemId=${f.ctx.tid})`; + return makeViolation(rule, message, offenders.length); +} + +/** Tags found directly under a `` container that are not the allowed `namedValue`. */ +function namedValuesInvalidChildren(nv: XmlNode): string[] { + const bad: string[] = []; + for (const [k, v] of Object.entries(nv)) { + if (k.startsWith('@_') || k === '#text' || k === 'namedValue') continue; + const instances = toArr(v).length; // one entry per element instance (mirrors the back-end count) + for (let i = 0; i < instances; i++) bad.push(k); + } + return bad; +} + +/** SETVALUES-INVALID-ELEMENT-001 — reject hallucinated ``/`` and bad namedValues children. */ +function validateSetValuesInvalidElements(tc: XmlNode, rule: BPRule): BPViolation | null { + const invalidElements = (rule.check['invalidElements'] as string[] | undefined) ?? ['namedValueSet', 'name']; + let count = 0; + let first: { ctx: ReturnType; elem: string; context?: string } | null = null; + for (const call of getAllApiCalls(tc)) { + if (call['@_apiId'] !== SETVALUES_API_ID) continue; + const ctx = stepContext(call); + for (const elem of invalidElements) { + if (collectElementsByTag(call, elem).length) { + count++; + first ??= { ctx, elem }; + } + } + for (const nv of collectElementsByTag(call, 'namedValues')) { + if (!nv || typeof nv !== 'object') continue; + for (const childTag of namedValuesInvalidChildren(nv as XmlNode)) { + count++; + first ??= { ctx, elem: childTag, context: 'inside ' }; + } + } + } + if (!first) return null; + const ctxStr = first.context ? ` ${first.context}` : ''; + const message = + `SetValues step '${first.ctx.title}' contains invalid element <${first.elem}>${ctxStr}. SetValues must use ` + + ' with children. Do not use ' + + ` or elements. (testItemId=${first.ctx.tid})`; + return makeViolation(rule, message, count); +} + +// ── Back-end-only rules (Tier 4) ───────────────────────────────────────────── +// Ports of seven Quality Hub validators that existed only in the back-end rule +// set. All seven rules are severity=major / weight=5. Score parity: six emit a +// single violation (the back-end returns the first offender; `count` is set only +// for the two UI-locator checks, and only when >1 offender); `varStringLiteral` +// emits ONE violation per offending value (the back-end returns a list, scored +// linearly) — do not collapse it. + +const ASSERT_VALUES_API_ID = 'com.provar.plugins.bundled.apis.AssertValues'; +const DB_CONNECT_API_ID = 'com.provar.plugins.bundled.apis.db.DbConnect'; +const UI_DO_ACTION_API_ID = 'com.provar.plugins.forcedotcom.core.ui.UiDoAction'; + +// DB operation steps whose dbConnectionName must reference a DbConnect resultName +// (both the modern `db.*` and legacy `data.*` namespaces, mirroring the back-end). +const DB_OPERATION_API_IDS: ReadonlySet = new Set([ + 'com.provar.plugins.bundled.apis.db.SqlQuery', + 'com.provar.plugins.bundled.apis.db.DbRead', + 'com.provar.plugins.bundled.apis.db.DbInsert', + 'com.provar.plugins.bundled.apis.db.DbUpdate', + 'com.provar.plugins.bundled.apis.db.DbDelete', + 'com.provar.plugins.bundled.apis.data.SqlQuery', + 'com.provar.plugins.bundled.apis.data.DbRead', + 'com.provar.plugins.bundled.apis.data.DbInsert', + 'com.provar.plugins.bundled.apis.data.DbUpdate', + 'com.provar.plugins.bundled.apis.data.DbDelete', +]); + +/** Escape a literal for safe embedding in a RegExp (mirrors Python's re.escape). */ +function escapeRegExp(s: string): string { + return s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +/** + * Resolve an argument's text value, mirroring the back-end `get_argument_value`: + * `variable` → first `` element name, `compound` → concatenated `` + * text, otherwise the element text. Wrapper-aware (handles both the `` + * wrapper and direct `` children) via {@link findArgumentById}. + */ +function resolvedArgText(call: XmlNode, argId: string): string { + const arg = findArgumentById(call, argId); + if (!arg) return ''; + const raw = arg['value']; + const ve = Array.isArray(raw) ? (raw as unknown[])[0] : raw; + if (ve == null) return ''; + if (typeof ve === 'string') return ve.trim(); + if (typeof ve !== 'object') return String(ve).trim(); + const v = ve as XmlNode; + const cls = v['@_class'] as string | undefined; + if (cls === 'variable') { + const firstPath = toArr(v['path'] as XmlNode | XmlNode[])[0] as XmlNode | undefined; + return ((firstPath?.['@_element'] as string | undefined) ?? '').trim(); + } + if (cls === 'compound') { + const partsNode = (v['parts'] as XmlNode | undefined)?.['value']; + return toArr(partsNode as XmlNode | XmlNode[]) + .filter((p) => p && typeof p === 'object') + .map((p) => nodeText(p)) + .join(''); + } + return nodeText(v); +} + +// Matches a bare variable token only: `{Name}` or `{Obj.Field}`. The character +// class `[\w.]` excludes `:`, so binding-style expressions such as +// `{targetUrl:object}` never match — they are inherently safe and need no +// argument-name exemption. +const VAR_LITERAL_PATTERN = /^\{[\w.]+\}$/; + +/** + * VAR-STRING-LITERAL-001 — a `{Var}`/`{Obj.Field}` token stored as + * `class="value" valueClass="string"` instead of `class="variable"`. Provar does + * not resolve it at runtime, so the API silently receives the literal text. Emits + * ONE violation per offending value (the back-end returns a list). + * + * Local correction (PDX-508): the back-end exempts `sfUiTargetObjectId` / + * `sfUiTargetResultName` from this check, but field evidence shows a bare + * `{Variable}` in those UI-target args is NOT interpolated — it lands in the URL + * as `%7B…%7D` and the step hard-fails (a load/exec stopper, not a warning). We + * therefore do NOT exempt those args. Binding-style `{ns:key}` expressions stay + * safe because {@link VAR_LITERAL_PATTERN} already excludes them (the colon). A + * matching change is queued for the Quality Hub back-end so the score parity is + * restored once both ship. + */ +function validateVarStringLiteral(tc: XmlNode, rule: BPRule): BPViolation[] { + const violations: BPViolation[] = []; + for (const call of getAllApiCalls(tc)) { + const ctx = stepContext(call); + for (const { value } of getStepValueElements(call)) { + if (value['@_class'] !== 'value' || value['@_valueClass'] !== 'string') continue; + const text = nodeText(value); + if (!VAR_LITERAL_PATTERN.test(text)) continue; + violations.push( + makeViolation( + rule, + `Argument value "${text}" looks like a variable reference but is stored as a plain string — Provar ` + + 'will not resolve it at runtime. Use instead ' + + `(step '${ctx.title}', testItemId=${ctx.tid})` + ) + ); + } + } + return violations; +} + +/** CONN-DB-002 — every DB operation's dbConnectionName must match a DbConnect resultName in the test. */ +function validateDbConnectResultNameMismatch(tc: XmlNode, rule: BPRule): BPViolation | null { + const calls = getAllApiCalls(tc); + const resultNames = new Set(); + for (const call of calls) { + if (call['@_apiId'] !== DB_CONNECT_API_ID) continue; + const rn = resolvedArgText(call, 'resultName'); + if (rn) resultNames.add(rn); + } + if (!resultNames.size) return null; // no DbConnect resultName — CONN-DB-001 covers a missing DbConnect + + const offenders: string[] = []; + for (const call of calls) { + if (!DB_OPERATION_API_IDS.has(call['@_apiId'] as string)) continue; + const ref = resolvedArgText(call, 'dbConnectionName'); + if (!ref || resultNames.has(ref)) continue; + offenders.push(`'${stepContext(call).title.slice(0, 40)}' uses dbConnectionName='${ref}'`); + } + if (!offenders.length) return null; + const names = [...resultNames].sort().join(', '); + return makeViolation( + rule, + `DbConnect resultName does not match dbConnectionName in ${offenders.length} DB operation(s): ` + + `${offenders[0]} but DbConnect resultName(s) are: ${names}` + ); +} + +/** The `` children of a SetValues `` (the assigned-value slot). */ +function getSetValuesValueElements(call: XmlNode): XmlNode[] { + const out: XmlNode[] = []; + for (const nv of collectElementsByTag(call, 'namedValue')) { + if (!nv || typeof nv !== 'object' || (nv as XmlNode)['@_name'] !== 'value') continue; + for (const v of toArr((nv as XmlNode)['value'] as XmlNode | XmlNode[])) { + if (v && typeof v === 'object') out.push(v); + } + } + return out; +} + +const SETVALUES_FUNC_EXPR = /\{[A-Za-z][A-Za-z0-9]*\s*\([^)]*\)\s*\}/; +const SETVALUES_ZERO_IDX = /\{[^}]*\[0\][^}]*\}/; + +/** First SetValues `valueClass="string"` value whose text matches `re` (string-template anti-patterns). */ +function firstSetValuesStringMatch( + tc: XmlNode, + re: RegExp +): { ctx: ReturnType; text: string } | null { + for (const call of getAllApiCalls(tc)) { + if (call['@_apiId'] !== SETVALUES_API_ID) continue; + for (const ve of getSetValuesValueElements(call)) { + if (ve['@_class'] !== 'value' || ve['@_valueClass'] !== 'string') continue; + const text = nodeText(ve); + if (re.test(text)) return { ctx: stepContext(call), text }; + } + } + return null; +} + +/** SETVALUES-FUNC-STR-001 — SetValues uses `{Func(args)}` string interpolation instead of a `funcCall` value. */ +function validateSetValuesFuncCallString(tc: XmlNode, rule: BPRule): BPViolation | null { + const hit = firstSetValuesStringMatch(tc, SETVALUES_FUNC_EXPR); + if (!hit) return null; + return makeViolation( + rule, + 'SetValues uses string interpolation for a function call — the value will not be evaluated: ' + + `'${hit.ctx.title.slice(0, 50)}' value='${hit.text.slice(0, 60)}' (testItemId=${hit.ctx.tid})` + ); +} + +/** SETVALUES-ZERO-IDX-001 — SetValues string template uses a 0 index (templates are 1-indexed). */ +function validateSetValuesZeroIndexString(tc: XmlNode, rule: BPRule): BPViolation | null { + const hit = firstSetValuesStringMatch(tc, SETVALUES_ZERO_IDX); + if (!hit) return null; + return makeViolation( + rule, + 'SetValues string expression uses a [0] index — Provar string templates are 1-indexed, causing an ' + + `out-of-bounds error at runtime: '${hit.ctx.title.slice(0, 50)}' value='${hit.text.slice(0, 60)}' — use ` + + `[1] for the first item (testItemId=${hit.ctx.tid})` + ); +} + +const ASSERT_WHOLE_EXPR = /^\s*\{[^{}]+\}\s*$/; + +/** The direct `` element child of an argument (wrapper-aware), or undefined. */ +function directArgValueElement(call: XmlNode, argId: string): XmlNode | undefined { + const arg = findArgumentById(call, argId); + if (!arg) return undefined; + const raw = arg['value']; + const ve = Array.isArray(raw) ? (raw as unknown[])[0] : raw; + return ve && typeof ve === 'object' ? (ve as XmlNode) : undefined; +} + +/** ASSERT-STR-VAR-001 — AssertValues references a variable via a `{Var}` string literal instead of `class="variable"`. */ +function validateAssertValuesStringExpr(tc: XmlNode, rule: BPRule): BPViolation | null { + for (const call of getAllApiCalls(tc)) { + if (call['@_apiId'] !== ASSERT_VALUES_API_ID) continue; + for (const argId of ['expectedValue', 'actualValue']) { + const v = directArgValueElement(call, argId); + if (!v || v['@_class'] !== 'value' || v['@_valueClass'] !== 'string') continue; + const text = nodeText(v); + if (!ASSERT_WHOLE_EXPR.test(text)) continue; + const ctx = stepContext(call); + return makeViolation( + rule, + 'AssertValues uses a string literal to reference a variable — the assertion compares the literal text, ' + + `not the variable value: '${ctx.title.slice(0, 50)}' ${argId}='${text.slice(0, 60)}' should use ` + + ` (testItemId=${ctx.tid})` + ); + } + } + return null; +} + +/** The `uri` of a UiDoAction's `locator` argument value (`class="uiLocator"`), or '' if absent. */ +function getUiDoActionLocatorUri(call: XmlNode): string { + const arg = getCallArguments(call).find((a) => a['@_id'] === 'locator'); + if (!arg) return ''; + for (const v of toArr(arg['value'] as XmlNode | XmlNode[])) { + if (v && typeof v === 'object' && v['@_class'] === 'uiLocator') { + return (v['@_uri'] as string | undefined) ?? ''; + } + } + return ''; +} + +// Standard SF flow buttons whose locator name must use the corpus-validated casing/path. +const UI_WRONG_BUTTONS: ReadonlyArray = [ + ['Cancel', "use 'name=cancel' (lowercase)"], + ['continue', "the Continue button on record type selection screens uses 'name=save&path=selectRecordType'"], + ['Continue', "the Continue button on record type selection screens uses 'name=save&path=selectRecordType'"], +]; + +/** UI-LOCATOR-BUTTON-CASING-001 — Cancel/Continue flow buttons must use the correct locator name. */ +function validateUiLocatorButtonCasing(tc: XmlNode, rule: BPRule): BPViolation | null { + const offenders: Array<{ ctx: ReturnType; wrong: string; explanation: string; uri: string }> = []; + for (const call of getAllApiCalls(tc)) { + if (call['@_apiId'] !== UI_DO_ACTION_API_ID) continue; + const uri = getUiDoActionLocatorUri(call); + if (!uri) continue; + for (const [wrong, explanation] of UI_WRONG_BUTTONS) { + if (new RegExp(`name=${escapeRegExp(wrong)}(&|$)`).test(uri)) { + offenders.push({ ctx: stepContext(call), wrong, explanation, uri }); + break; // only report the first match per step (mirrors the back-end) + } + } + } + if (!offenders.length) return null; + const f = offenders[0]; + return makeViolation( + rule, + `Step '${f.ctx.title}' uses incorrect button name 'name=${f.wrong}': ${f.explanation}. Incorrect button ` + + `names cause Provar to show 'Not Available' and fail at runtime. Current URI: ${f.uri.slice(0, 120)} ` + + `(testItemId=${f.ctx.tid})`, + offenders.length + ); +} + +const UI_RECORDTYPE_WRONG = /name=recordType(Id)?(&|$)/; + +/** UI-LOCATOR-RECORDTYPE-001 — the Record Type picker locator must use `name=RecordType`, not `name=recordType(Id)`. */ +function validateUiLocatorRecordTypeField(tc: XmlNode, rule: BPRule): BPViolation | null { + const offenders: Array<{ ctx: ReturnType; uri: string }> = []; + for (const call of getAllApiCalls(tc)) { + if (call['@_apiId'] !== UI_DO_ACTION_API_ID) continue; + const uri = getUiDoActionLocatorUri(call); + if (!uri || !UI_RECORDTYPE_WRONG.test(uri)) continue; + offenders.push({ ctx: stepContext(call), uri }); + } + if (!offenders.length) return null; + const f = offenders[0]; + return makeViolation( + rule, + `Step '${f.ctx.title}' uses an incorrect Record Type field locator. The Record Type picker must use ` + + "'name=RecordType' (not 'name=recordTypeId' or 'name=recordType') with 'field=RecordTypeId' in the binding. " + + `Current URI: ${f.uri.slice(0, 150)} (testItemId=${f.ctx.tid})`, + offenders.length + ); +} + +// ── Structural / load-affecting check types (Tier 5) ───────────────────────── +// Faithful ports of nine Quality Hub structural validators. Six emit a single +// first-offender violation with no `count`; validFuncCallId, the two UiAssert +// structure checks, and bindingParameterOrder collect every offender and emit +// one violation carrying `count` (capped at 5 for funcCall), matching the +// back-end so the weighted-deduction score stays in parity with the Lambda. + +const UI_ASSERT_API_ID = 'com.provar.plugins.forcedotcom.core.ui.UiAssert'; + +// FUNCCALL-VALID-001 — Provar's built-in funcCall ids (exact back-end whitelist, 20 entries). +const VALID_FUNCCALL_IDS: ReadonlySet = new Set([ + 'TestCaseName', + 'TestCasePath', + 'TestCaseOutcome', + 'TestCaseSuccessful', + 'TestCaseErrors', + 'TestRunErrors', + 'StringReplace', + 'StringTrim', + 'StringNormalize', + 'DateAdd', + 'DateFormat', + 'DateParse', + 'Count', + 'Round', + 'NumberFormat', + 'Not', + 'IsSorted', + 'GetEnvironmentVariable', + 'GetSelectedEnvironment', + 'UniqueId', +]); + +/** FUNCCALL-VALID-001 — every `` `id` must be a real Provar built-in function. */ +function validateValidFuncCallId(tc: XmlNode, rule: BPRule): BPViolation | null { + const offenders: string[] = []; + for (const v of collectValueElements(tc)) { + if (v['@_class'] !== 'funcCall') continue; + const id = (v['@_id'] as string | undefined) ?? ''; + if (id && !VALID_FUNCCALL_IDS.has(id)) offenders.push(id); + } + if (!offenders.length) return null; + const MAX = 5; + let msg = `Unknown funcCall id(s) — these functions do not exist in Provar: ${offenders + .slice(0, 3) + .map((id) => `'${id}'`) + .join(', ')}`; + if (offenders.length > 3) msg += ` (+${offenders.length - 3} more)`; + msg += + '. Valid functions include Count, DateAdd, DateFormat, Round, StringReplace, UniqueId, etc. ' + + 'For string concatenation use ….'; + return makeViolation(rule, msg, Math.min(offenders.length, MAX)); +} + +// RENDER-ROOT-001 — the only attributes allowed on the root element. +const VALID_ROOT_ATTRS: ReadonlySet = new Set([ + 'guid', + 'id', + 'name', + 'visibility', + 'registryId', + 'failureBehaviour', +]); + +/** RENDER-ROOT-001 — the root `` element must not carry unknown attributes. */ +function validateRootAttributes(tc: XmlNode, rule: BPRule): BPViolation | null { + const unknown = Object.keys(tc) + .filter((k) => k.startsWith('@_')) + .map((k) => k.slice(2)) + .filter((a) => !VALID_ROOT_ATTRS.has(a)); + if (!unknown.length) return null; + return makeViolation(rule, `Unknown root attributes: ${unknown.join(', ')}`); +} + +/** SETVALUES-STRUCTURE-001 — every SetValues step must contain a `` container. */ +function validateSetValuesStructure(tc: XmlNode, rule: BPRule): BPViolation | null { + for (const call of getAllApiCalls(tc)) { + if (call['@_apiId'] !== SETVALUES_API_ID) continue; + // Data-driven SetValues pull their values from an external source declared in + // (e.g. an Excel/CSV binding) and legitimately carry an + // empty with no inline . Not a defect. + if (call['parameterValueSources'] != null) continue; + if (collectElementsByTag(call, 'namedValues').length) continue; + return makeViolation(rule, `SetValues step missing container (testItemId=${stepContext(call).tid})`); + } + return null; +} + +/** SETVALUES-NAME-001 — every `` in a SetValues step must carry a `name` attribute. */ +function validateNamedValueName(tc: XmlNode, rule: BPRule): BPViolation | null { + for (const call of getAllApiCalls(tc)) { + if (call['@_apiId'] !== SETVALUES_API_ID) continue; + for (const nv of collectElementsByTag(call, 'namedValue')) { + if (!nv || typeof nv !== 'object' || (nv as XmlNode)['@_name']) continue; + return makeViolation( + rule, + `namedValue in SetValues step missing name attribute (testItemId=${stepContext(call).tid})` + ); + } + } + return null; +} + +// The QEditor SetValues form stores each assignment as a triple of namedValue slots: +// `valuePath` (target field), `value` (data), `valueScope`. Any of these may be left +// empty — a blank `value` sets the field to blank, and a wholly-blank row is an unused +// row Provar simply ignores. So an empty structural slot is NOT a "missing value" defect. +const SETVALUES_BLANKABLE_SLOTS: ReadonlySet = new Set(['valuePath', 'value', 'valueScope']); + +/** SETVALUES-VALUE-001 — every `` in a SetValues step must contain a child ``. */ +function validateNamedValueValue(tc: XmlNode, rule: BPRule): BPViolation | null { + for (const call of getAllApiCalls(tc)) { + if (call['@_apiId'] !== SETVALUES_API_ID) continue; + for (const nv of collectElementsByTag(call, 'namedValue')) { + if (!nv || typeof nv !== 'object') continue; + const node = nv as XmlNode; + if (node['value'] != null) continue; + // A blank structural slot (valuePath/value/valueScope) is valid (empty value / + // unused row). Only a non-standard namedValue missing its value is a real defect. + if (SETVALUES_BLANKABLE_SLOTS.has((node['@_name'] as string | undefined) ?? '')) continue; + return makeViolation( + rule, + `namedValue in SetValues step missing value element (testItemId=${stepContext(call).tid})` + ); + } + } + return null; +} + +/** UI-ASSERT-STRUCT-002 — UiAssert steps must not contain a (hallucinated) `` element. */ +function validateUiAssertHallucinatedGeneratedParameters(tc: XmlNode, rule: BPRule): BPViolation | null { + const offenders: Array> = []; + for (const call of getAllApiCalls(tc)) { + if (call['@_apiId'] !== UI_ASSERT_API_ID || call['generatedParameters'] == null) continue; + offenders.push(stepContext(call)); + } + if (!offenders.length) return null; + const f = offenders[0]; + return makeViolation( + rule, + `UiAssert step '${f.title}' contains a hallucinated element — UiAssert steps never ` + + `contain generatedParameters; remove the entire section (testItemId=${f.tid})`, + offenders.length + ); +} + +// UI-ASSERT-STRUCT-001 — arguments every UiAssert step must declare (even if empty). +const UI_ASSERT_REQUIRED_ARGS: readonly string[] = [ + 'fieldAssertions', + 'columnAssertions', + 'pageAssertions', + 'resultScope', + 'captureAfter', + 'beforeWait', + 'autoRetry', +]; + +/** UI-ASSERT-STRUCT-001 — a UiAssert step is missing one or more of its required arguments. */ +function validateUiAssertMissingArguments(tc: XmlNode, rule: BPRule): BPViolation | null { + const offenders: Array<{ ctx: ReturnType; missing: string[] }> = []; + for (const call of getAllApiCalls(tc)) { + if (call['@_apiId'] !== UI_ASSERT_API_ID) continue; + const argsNode = call['arguments']; + let missing: string[]; + if (!argsNode || typeof argsNode !== 'object') { + missing = [...UI_ASSERT_REQUIRED_ARGS]; + } else { + const existing = new Set(); + for (const a of getCallArguments(call)) { + const id = a['@_id'] as string | undefined; + if (id) existing.add(id); + } + missing = UI_ASSERT_REQUIRED_ARGS.filter((r) => !existing.has(r)); + } + if (missing.length) offenders.push({ ctx: stepContext(call), missing }); + } + if (!offenders.length) return null; + const f = offenders[0]; + return makeViolation( + rule, + `UiAssert step '${f.ctx.title}' is missing required arguments: ${f.missing.join(', ')}. All UiAssert steps ` + + 'must include fieldAssertions, columnAssertions, pageAssertions, resultScope, captureAfter, beforeWait, and ' + + `autoRetry (even if empty) (testItemId=${f.ctx.tid})`, + offenders.length + ); +} + +// UI-BINDING-ORDER-001 — binding URIs must list object= before action=/field= (percent-encoded). +const BINDING_WRONG_ACTION_FIRST = /object%3Faction%3D[^%]+%26object%3D/; +const BINDING_WRONG_FIELD_FIRST = /object%3Ffield%3D[^%]+%26object%3D/; +const BINDING_ACTION_EXTRACT = /action%3D([^%&]+)%26object%3D([^%&"]+)/; +const BINDING_FIELD_EXTRACT = /field%3D([^%&]+)%26object%3D([^%&"]+)/; + +/** Classify a binding URI's parameter order, returning the wrong/correct pair or null when fine. */ +function classifyBindingOrder(uri: string): { wrong: string; correct: string } | null { + if (BINDING_WRONG_ACTION_FIRST.test(uri)) { + const m = BINDING_ACTION_EXTRACT.exec(uri); + if (m) { + const o = m[2].replace(/&/g, '').replace(/&/g, ''); + return { wrong: `action=${m[1]}&object=${o}`, correct: `object=${o}&action=${m[1]}` }; + } + } else if (BINDING_WRONG_FIELD_FIRST.test(uri)) { + const m = BINDING_FIELD_EXTRACT.exec(uri); + if (m) { + const o = m[2].replace(/&/g, '').replace(/&/g, ''); + return { wrong: `field=${m[1]}&object=${o}`, correct: `object=${o}&field=${m[1]}` }; + } + } + return null; +} + +/** UI-BINDING-ORDER-001 — a `uiLocator` binding lists object= after action=/field= (non-standard order). */ +function validateBindingParameterOrder(tc: XmlNode, rule: BPRule): BPViolation | null { + const seen = new Set(); + const offenders: Array<{ ctx: ReturnType; wrong: string; correct: string }> = []; + for (const call of getAllApiCalls(tc)) { + const ctx = stepContext(call); + for (const v of collectValueElements(call)) { + if (v['@_class'] !== 'uiLocator' || seen.has(v)) continue; + seen.add(v); + const uri = (v['@_uri'] as string | undefined) ?? ''; + if (!uri || !uri.includes('binding=')) continue; + const verdict = classifyBindingOrder(uri); + if (verdict) offenders.push({ ctx, ...verdict }); + } + } + if (!offenders.length) return null; + const f = offenders[0]; + return makeViolation( + rule, + `UI binding in step '${f.ctx.title}' lists parameters in a non-standard order: found '${f.wrong}', the ` + + `corpus-majority convention lists object= first ('${f.correct}'). (testItemId=${f.ctx.tid})`, + offenders.length + ); +} + +// UI-CONN-LITERAL-001 — UI step types whose uiConnectionName must be a literal, not a variable. +const UI_CONN_LITERAL_APIS: ReadonlySet = new Set([ + 'com.provar.plugins.forcedotcom.core.ui.UiWithScreen', + 'com.provar.plugins.forcedotcom.core.ui.UiDoAction', + 'com.provar.plugins.forcedotcom.core.ui.UiAssert', + 'com.provar.plugins.forcedotcom.core.ui.UiWithRow', +]); + +/** UI-CONN-LITERAL-001 — a UI step's `uiConnectionName` uses a `class="variable"` value instead of a literal. */ +function validateUiConnectionNameLiteral(tc: XmlNode, rule: BPRule): BPViolation | null { + for (const call of getAllApiCalls(tc)) { + if (!UI_CONN_LITERAL_APIS.has(call['@_apiId'] as string)) continue; + const v = directArgValueElement(call, 'uiConnectionName'); + if (!v || v['@_class'] !== 'variable') continue; + const ctx = stepContext(call); + return makeViolation( + rule, + `UI step '${ctx.title}' uses a variable reference for uiConnectionName; it must be a literal string ` + + `(testItemId=${ctx.tid})` + ); + } + return null; +} + // ── Validator dispatch map ──────────────────────────────────────────────────── type ValidatorFn = (tc: XmlNode, rule: BPRule) => BPViolation | null; @@ -1107,6 +2165,33 @@ const VALIDATOR_REGISTRY: Record = { detectDuplicatesLiterals: validateDetectDuplicatesLiterals, uniqueResultNames: validateUniqueResultNames, uiWithScreenTarget: validateUiWithScreenTarget, + mustContainArgument: validateMustContainArgument, + // Tier 2 — render / load-blocking check types (ports of the QH load-blocking validators) + valueClassCasing: validateValueClassCasing, + booleanCasing: validateBooleanCasing, + invalidValueClass: validateInvalidValueClass, + dateValueClassFormat: validateDateValueClassFormat, + apexConnectReuseConnection: validateApexConnectReuseConnection, + apexConnectValidArguments: validateApexConnectValidArguments, + apexConnectConnectionIdValueClass: validateApexConnectConnectionIdValueClass, + setValuesInvalidElements: validateSetValuesInvalidElements, + // Tier 4 — back-end-only rules (single-violation ports) + dbConnectResultNameMismatch: validateDbConnectResultNameMismatch, + setValuesFuncCallString: validateSetValuesFuncCallString, + setValuesZeroIndexString: validateSetValuesZeroIndexString, + assertValuesStringExpr: validateAssertValuesStringExpr, + uiLocatorButtonCasing: validateUiLocatorButtonCasing, + uiLocatorRecordTypeField: validateUiLocatorRecordTypeField, + // Tier 5 — structural / load-affecting check types + validFuncCallId: validateValidFuncCallId, + rootAttributes: validateRootAttributes, + setValuesStructure: validateSetValuesStructure, + namedValueName: validateNamedValueName, + namedValueValue: validateNamedValueValue, + uiAssertHallucinatedGeneratedParameters: validateUiAssertHallucinatedGeneratedParameters, + uiAssertMissingArguments: validateUiAssertMissingArguments, + bindingParameterOrder: validateBindingParameterOrder, + uiConnectionNameLiteral: validateUiConnectionNameLiteral, // 'regex' is dispatched separately (needs metadata) // 'uiActionNestingStructure' is dispatched separately (emits one violation per offending step) }; @@ -1118,6 +2203,8 @@ const MULTI_VALIDATOR_REGISTRY: Record = { uiActionNestingStructure: validateUiActionNestingStructure, nitroxConnectInvalidArgs: validateNitroxConnectInvalidArgs, nitroxVariantArgRequired: validateNitroxVariantArgRequired, + // Tier 4 — emits one violation per offending value (back-end returns a list) + varStringLiteral: validateVarStringLiteral, }; // ── XML parser (shared settings) ───────────────────────────────────────────── diff --git a/src/mcp/tools/hierarchyValidate.ts b/src/mcp/tools/hierarchyValidate.ts index b8a742de..0436c0b9 100644 --- a/src/mcp/tools/hierarchyValidate.ts +++ b/src/mcp/tools/hierarchyValidate.ts @@ -93,7 +93,7 @@ export interface HierarchyViolation { export interface TestCaseResult { name: string; level: 'test_case'; - status: 'valid' | 'invalid' | 'error'; + status: 'valid' | 'needs_improvement' | 'invalid' | 'error'; quality_score: number; validity_score: number; is_valid: boolean; @@ -137,6 +137,7 @@ export interface ProjectResult { export interface HierarchySummary { total_test_cases: number; test_cases_valid: number; + test_cases_needs_improvement: number; test_cases_invalid: number; test_cases_error: number; total_violations: number; @@ -370,7 +371,10 @@ export function validateHierarchyTestCase(tc: TestCaseInput, qualityThreshold: n const xmlSource = tc.xml ?? tc.xml_content ?? ''; try { const r = validateTestCase(xmlSource, tc.name); - const status = !r.is_valid ? 'invalid' : r.quality_score < qualityThreshold ? 'invalid' : 'valid'; + // PDX-509 tri-state: criticals flip is_valid → 'invalid'; a loadable case + // below the quality bar is 'needs_improvement' (was previously collapsed to + // 'invalid'); at or above the bar is 'valid'. + const status = !r.is_valid ? 'invalid' : r.quality_score < qualityThreshold ? 'needs_improvement' : 'valid'; return { name: tc.name, level: 'test_case', @@ -1054,6 +1058,7 @@ export function buildHierarchySummary(result: SuiteResult | PlanResult | Project const stats: HierarchySummary = { total_test_cases: 0, test_cases_valid: 0, + test_cases_needs_improvement: 0, test_cases_invalid: 0, test_cases_error: 0, total_violations: 0, @@ -1078,6 +1083,7 @@ export function buildHierarchySummary(result: SuiteResult | PlanResult | Project if (node.level === 'test_case') { stats.total_test_cases++; if (node.status === 'valid') stats.test_cases_valid++; + else if (node.status === 'needs_improvement') stats.test_cases_needs_improvement++; else if (node.status === 'invalid') stats.test_cases_invalid++; else stats.test_cases_error++; scores.push(node.quality_score); diff --git a/src/mcp/tools/projectValidateFromPath.ts b/src/mcp/tools/projectValidateFromPath.ts index a27c160a..7662ed31 100644 --- a/src/mcp/tools/projectValidateFromPath.ts +++ b/src/mcp/tools/projectValidateFromPath.ts @@ -203,11 +203,10 @@ export function registerProjectValidateFromPath(server: McpServer, config: Serve .min(0) .max(100) .optional() - .default(80) .describe( desc( - 'Minimum quality score for a test case to be considered valid (default: 80)', - 'number 0–100, optional; minimum quality score threshold' + 'Minimum quality score for a test case to be considered valid. Precedence: this arg → PROVAR_MCP_QUALITY_THRESHOLD env → 90.', + 'number 0–100, optional; default 90' ) ), save_results: z diff --git a/src/mcp/tools/testCaseGenerate.ts b/src/mcp/tools/testCaseGenerate.ts index c44f7406..4a7b380e 100644 --- a/src/mcp/tools/testCaseGenerate.ts +++ b/src/mcp/tools/testCaseGenerate.ts @@ -15,6 +15,7 @@ import type { ServerConfig } from '../server.js'; import { assertPathAllowed, PathPolicyError } from '../security/pathPolicy.js'; import { makeError, makeRequestId } from '../schemas/common.js'; import { log } from '../logging/logger.js'; +import { allocateTestCaseId, DEFAULT_TESTCASE_ID } from '../utils/testCaseId.js'; import { validateTestCase } from './testCaseValidate.js'; import { desc } from './descHelper.js'; import { UI_ACTION_API_IDS, UI_SCREEN_CONTAINER_API_IDS, UI_LOCATOR_BEARING_API_IDS } from './uiActionApiIds.js'; @@ -201,7 +202,7 @@ const TOOL_DESCRIPTION = [ // ── Existing description (unchanged below) ─────────────────────────────────── 'Generate a Provar XML test case skeleton with proper UUID v4 guids, sequential testItemId values, and structure.', 'Returns XML content. Writes to disk only when dry_run=false.', - 'Generated structure: with (id is always the integer literal "1" as required by the Provar runtime), a child, then .', + 'Generated structure: with (a numeric integer id), a child, then . The unique identifier is the guid; id is a human-facing label. When writing into an existing project the id is auto-allocated as the highest in-use id + 1; otherwise it defaults to 1.', 'URI-aware generation: use target_uri to control the XML nesting structure.', ' - sf:ui:target (or omit target_uri) → flat Salesforce XML structure (existing behaviour).', ' - ui:pageobject:target?pageId=pageobjects.PageClass → wraps all steps in a UiWithScreen element targeting that non-SF page object.', @@ -226,9 +227,9 @@ const TOOL_DESCRIPTION = [ 'target argument (UiWithScreen/UiWithRow): pass the URI value; emitted as class="uiTarget" uri="...".', 'locator argument (UiDoAction/UiAssert/UiRead/UiFill): pass the URI value; emitted as class="uiLocator" uri="...".', 'valueClass auto-detection: argument values are typed automatically before XML emission. ' + - 'ISO-8601 date "YYYY-MM-DD" → valueClass="date"; ISO-8601 datetime "YYYY-MM-DDTHH:MM:SS" (optional fractional seconds + timezone) → "datetime"; ' + + 'ISO-8601 date "YYYY-MM-DD" → valueClass="date"; ISO-8601 datetime "YYYY-MM-DDTHH:MM:SS" (optional fractional seconds + timezone) → "dateTime"; ' + '"true"/"false" → "boolean"; numeric string (e.g. "42", "-5", "3.14") → "decimal"; otherwise "string". ' + - 'Pass dates / booleans / numbers in those formats — Provar runtime silently discards date fields emitted as valueClass="string". ' + + 'Pass dates in ISO-8601 — the generator converts them to the epoch-millisecond value Provar requires (a date/dateTime field emitted as an ISO string, or as valueClass="string", fails to load in the IDE). A bare date is treated as UTC midnight; a datetime without a timezone is treated as UTC. ' + 'Note: numbers always emit valueClass="decimal" per the canonical Provar reference (there is no separate "integer" valueClass).', 'Edit page objects: action=Edit targets require a compiled page object for the SF object. ' + 'If none exists in the project page-objects directory, the locator binding will fail at runtime. ' + @@ -389,13 +390,23 @@ export function registerTestCaseGenerate(server: McpServer, config: ServerConfig } try { - const xmlContent = buildTestCaseXml(input); const filePath: string | undefined = input.output_path ? path.resolve(input.output_path) : undefined; - let written = false; + // Allocate the root testCase id from surrounding project context, but only + // when actually persisting (a write path that has cleared the path policy). + // Preview/dry-run runs have no project anchor, so they keep the default id. + let idAllocation = { id: DEFAULT_TESTCASE_ID, basis: 'default' as const } as ReturnType< + typeof allocateTestCaseId + >; if (filePath && !input.dry_run) { assertPathAllowed(filePath, config.allowedPaths); + idAllocation = allocateTestCaseId(filePath, config.allowedPaths); + } + const xmlContent = buildTestCaseXml(input, idAllocation.id); + let written = false; + + if (filePath && !input.dry_run) { if (fs.existsSync(filePath) && !input.overwrite) { const err = makeError( 'FILE_EXISTS', @@ -408,7 +419,12 @@ export function registerTestCaseGenerate(server: McpServer, config: ServerConfig fs.mkdirSync(path.dirname(filePath), { recursive: true }); fs.writeFileSync(filePath, xmlContent, 'utf-8'); written = true; - log('info', 'provar_testcase_generate: wrote file', { requestId, filePath }); + log('info', 'provar_testcase_generate: wrote file', { + requestId, + filePath, + testCaseId: idAllocation.id, + idBasis: idAllocation.basis, + }); } const warnings = buildStepWarnings(input.steps); @@ -420,6 +436,7 @@ export function registerTestCaseGenerate(server: McpServer, config: ServerConfig written, dry_run: input.dry_run, step_count: input.steps.length, + test_case_id: idAllocation.id, idempotency_key: input.idempotency_key, ...(warnings.length > 0 ? { warnings } : {}), }; @@ -557,14 +574,49 @@ function buildArgumentValue(key: string, val: string, indent: string, inNamedVal if (key === 'locator' && UI_LOCATOR_BEARING_API_IDS.has(apiId)) { return `${indent}`; } + // D2: 'interaction' argument → class="uiInteraction" (UiDoAction Action widget). + // PDX-506: the IDE step editor binds its Action only from a typed uiInteraction; + // a plain string runs green from the CLI but renders the Action field blank. + // Gated on the shared UI-action API set so generator + validator stay aligned. + if (key === 'interaction' && UI_ACTION_API_IDS.has(apiId)) { + return `${indent}`; + } } // PDX-493 (H3): infer valueClass for date / datetime / boolean / decimal / string. The // `fieldTypeHint` parameter on `inferSalesforceValueClass` is intentionally not threaded // through here yet — it lands in PDX-492 (H2b) along with the `field_type_hints` tool input. const inferred = inferSalesforceValueClass(key, val); + if (inferred === 'date' || inferred === 'datetime') { + // PDX-509: Provar stores date/dateTime as epoch MILLISECONDS, never an ISO string — + // an ISO string fails to load in the IDE (RENDER-DATE-VALUECLASS-001), and the real + // corpus uses epoch ms exclusively with camelCase `dateTime`. Convert here. + const ms = isoToEpochMs(val, inferred); + if (ms !== null) { + const valueClass = inferred === 'datetime' ? 'dateTime' : 'date'; + return `${indent}${ms}`; + } + // A value that matches the ISO shape but is not a real date (e.g. "2026-99-99") + // cannot be converted — fall back to a plain string rather than emit a + // load-breaking date/dateTime with a non-epoch value. + return `${indent}${escapeXmlContent(val)}`; + } return `${indent}${escapeXmlContent(val)}`; } +/** + * Convert a validated ISO-8601 date / datetime string to epoch milliseconds. + * A date-only string ("YYYY-MM-DD") is parsed as UTC midnight per the ES spec. A + * datetime WITHOUT an explicit timezone would otherwise parse as machine-local + * time (non-deterministic), so we pin it to UTC by appending 'Z' — matching the + * Provar corpus, where date/dateTime values are stored as UTC epoch ms. + * Returns null if the value cannot be parsed (caller falls back to the raw text). + */ +function isoToEpochMs(val: string, kind: 'date' | 'datetime'): number | null { + const needsUtc = kind === 'datetime' && !/(Z|[+-]\d{2}:?\d{2})$/.test(val); + const ms = Date.parse(needsUtc ? `${val}Z` : val); + return Number.isNaN(ms) ? null : ms; +} + function buildArgumentsXml(attributes: Record, baseIndent = ' ', apiId = ''): string { const entries = Object.entries(attributes); if (entries.length === 0) return ''; @@ -603,6 +655,94 @@ function buildSetValuesXml(attributes: Record, baseIndent: strin ); } +// PDX-507: derive the field name for a uiFieldAssertion from a locator URI's +// `name=` parameter (e.g. ui:locator?name=LastName&binding=… → "LastName"). +function extractFieldName(uri: string): string { + const m = /[?&]name=([^&]*)/.exec(uri); + if (!m) return 'Field'; + try { + return decodeURIComponent(m[1]) || 'Field'; + } catch { + return m[1] || 'Field'; + } +} + +// PDX-507: UiAssert — assemble the nested fieldAssertions / uiFieldAssertion +// structure the Provar IDE Result Assertions tab binds from. The generator +// previously emitted the flat shape (top-level fieldLocator / attributeName / +// comparisonType / expectedValue arguments), which runs green from the CLI but +// renders the Result Assertions tab blank in the IDE. Shape confirmed against +// the real test corpus (AllPOCProjects): 3,743/3,778 UiAssert steps use this +// nested form, with a BARE element (never class="uiLocator" +// — see best-practice rule UI-ASSERT-FIELDLOCATOR-002) and no `assertionType`. +// Only a `fieldLocator` attribute triggers the nested form; a UiAssert that +// carries a plain `locator` argument keeps the documented locator→uiLocator +// contract via the flat fallback (buildArgumentValue dispatches it). When no +// fieldLocator is supplied the step is not a field assertion, so fall back to +// flat argument emission and leave other UiAssert shapes untouched. +function buildUiAssertXml(attributes: Record, baseIndent: string, apiId: string): string { + const a = attributes; + const fieldLocator = a['fieldLocator'] ?? ''; + if (!fieldLocator) return buildArgumentsXml(attributes, baseIndent, apiId); + + const i = (n: number): string => baseIndent + ' '.repeat(n); + const resultName = a['resultName'] ?? 'Values'; + const resultScope = a['resultScope'] ?? 'Test'; + const captureAfter = a['captureAfter'] ?? 'false'; + const attributeName = a['attributeName'] ?? 'value'; + const comparisonType = a['comparisonType'] ?? 'EqualTo'; + const fieldName = extractFieldName(fieldLocator); + const expected = a['expectedValue']; + + // resultName / resultScope / captureAfter are control-plane literals that the + // real corpus ALWAYS emits as valueClass="string" — including captureAfter + // ("true"/"false"), which is intentionally NOT valueClass="boolean" here. + // Routing these through inferSalesforceValueClass would diverge from the + // corpus (it would infer boolean), so they are emitted as string literals. + const strVal = (v: string): string => `${escapeXmlContent(v)}`; + + // uiAttributeAssertion — self-closing when there is no expected value. This is + // a real corpus form (e.g. ADP_POV "UI Assert - Date Empty" emits + // ); + // otherwise it carries a typed child built via buildArgumentValue so + // {Var} expands to class="variable". + const attrOpen = `${i(5)}\n${buildArgumentValue('expectedValue', expected, i(6), true)}\n${i(5)}` + : `${attrOpen}/>`; + + const fieldAssertion = + `${i(4)}\n` + + `${i(5)}\n` + + `${i(5)}\n` + + `${attrAssertion}\n` + + `${i(5)}\n` + + `${i(4)}`; + + return ( + `\n${i(0)}\n` + + `${i(0)}\n${i(1)}${strVal(resultName)}\n${i(0)}\n` + + `${i(0)}\n${i(1)}${strVal(resultScope)}\n${i(0)}\n` + + `${i(0)}\n` + + `${i(1)}\n` + + `${fieldAssertion}\n` + + `${i(1)}\n` + + `${i(0)}\n` + + `${i(0)}\n${i(1)}${strVal(captureAfter)}\n${i(0)}\n` + + `${i(0)}\n${i(1)}\n${i( + 0 + )}\n` + + `${i(0)}\n${i(1)}\n${i(0)}\n` + + `${i(0)}\n` + + `${i(0)}\n` + + `${i(0)}\n` + + `${baseIndent.slice(0, -2)}` + ); +} + function buildFlatStepXml( step: { api_id: string; name: string; attributes: Record }, testItemId: number, @@ -631,9 +771,15 @@ function buildStepXmlWithChildren( const resolvedApiId = resolveApiId(step.api_id); const baseIndent = indent + ' '; // Use SetValues structure for any SetValues API (string-match mirrors the validator). - const argumentsXml = resolvedApiId.includes('SetValues') - ? buildSetValuesXml(step.attributes, baseIndent) - : buildArgumentsXml(step.attributes, baseIndent, resolvedApiId); + // PDX-507: UiAssert uses the nested fieldAssertions/uiFieldAssertion structure. + let argumentsXml: string; + if (resolvedApiId.includes('SetValues')) { + argumentsXml = buildSetValuesXml(step.attributes, baseIndent); + } else if (resolvedApiId.endsWith('.UiAssert') || resolvedApiId === 'UiAssert') { + argumentsXml = buildUiAssertXml(step.attributes, baseIndent, resolvedApiId); + } else { + argumentsXml = buildArgumentsXml(step.attributes, baseIndent, resolvedApiId); + } const hasClauses = substepsTestItemId !== undefined; const open = `${indent} before . + // Provar requires: standalone="no", a numeric integer id, no name attr, before . + // The id is a human-facing label, not a uniqueness key (guid is) — see allocateTestCaseId: + // when writing into an existing project we pass highest-in-use + 1; otherwise it defaults to 1. return ( '\n' + - `\n` + + `\n` + ' \n' + ' \n' + stepLines + diff --git a/src/mcp/tools/testCaseValidate.ts b/src/mcp/tools/testCaseValidate.ts index 17325030..c99307cf 100644 --- a/src/mcp/tools/testCaseValidate.ts +++ b/src/mcp/tools/testCaseValidate.ts @@ -9,6 +9,7 @@ import fs from 'node:fs'; import path from 'node:path'; import { createHash } from 'node:crypto'; +import { fileURLToPath } from 'node:url'; import { z } from 'zod'; import { XMLParser } from 'fast-xml-parser'; import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'; @@ -25,6 +26,7 @@ import { REQUEST_ACCESS_URL, } from '../../services/qualityHub/client.js'; import { applyDetailLevel, type DetailLevel } from '../utils/detailLevel.js'; +import { resolveQualityThreshold } from '../utils/qualityThreshold.js'; import { calcCompletenessScore, calcNextAction } from '../utils/validationScore.js'; import { generateRunId, @@ -46,7 +48,7 @@ import { } from '../rules/comparisonTypeSets.js'; import { runBestPractices } from './bestPracticesEngine.js'; import { desc } from './descHelper.js'; -import { UI_SCREEN_CONTAINER_API_IDS, UI_LOCATOR_BEARING_API_IDS } from './uiActionApiIds.js'; +import { UI_ACTION_API_IDS, UI_SCREEN_CONTAINER_API_IDS, UI_LOCATOR_BEARING_API_IDS } from './uiActionApiIds.js'; const ONBOARDING_MESSAGE = 'Quality Hub validation unavailable — running local validation only (structural rules, no quality scoring).\n' + @@ -67,6 +69,9 @@ const UNREACHABLE_WARNING = const TC_VALIDATE_SUMMARY_FIELDS = [ 'requestId', 'is_valid', + 'status', + 'quality_threshold', + 'meets_quality_threshold', 'validity_score', 'quality_score', 'validation_source', @@ -80,6 +85,25 @@ function tcStorageDir(): string { return resolveValidationDir('testcase'); } +/** PDX-509 tri-state verdict: validity gate (criticals) + quality bar (threshold). */ +interface QualityVerdict { + status: 'invalid' | 'needs_improvement' | 'valid'; + quality_threshold: number; + meets_quality_threshold: boolean; +} + +/** + * Combine the validity gate and the quality bar into one verdict. + * `invalid` when a critical defect blocks loading; otherwise `needs_improvement` + * when the score is below the (resolved) threshold; otherwise `valid`. + */ +function deriveQualityVerdict(isValid: boolean, qualityScore: number, qualityThresholdArg?: number): QualityVerdict { + const quality_threshold = resolveQualityThreshold(qualityThresholdArg); + const meets_quality_threshold = qualityScore >= quality_threshold; + const status = !isValid ? 'invalid' : meets_quality_threshold ? 'valid' : 'needs_improvement'; + return { status, quality_threshold, meets_quality_threshold }; +} + /** Resolve validation result from QualityHub API or fall back to local. */ async function resolveBaseResult( source: string, @@ -141,7 +165,7 @@ export function registerTestCaseValidate(server: McpServer, config: ServerConfig { title: 'Validate Test Case', description: desc( - "Validate a Provar XML test case for structural correctness and quality. Checks XML declaration, root element, required attributes (guid UUID v4, testItemId integer), presence, and applies best-practice rules. When a Provar API key is configured (via sf provar auth login or PROVAR_API_KEY env var), calls the Quality Hub API for full 170-rule scoring. Falls back to local validation if no key is set or the API is unavailable. Returns validity_score (schema compliance), quality_score (best practices, 0–100), and validation_source indicating which ruleset was applied. Every response includes run_id — pass it as baseline_run_id in the next call to receive only new/resolved issues. Data-driven note (DATA-001): when file_path is supplied and the project's provardx-properties.json references the test case directly via top-level `testCase` / `testCases` rather than via a `.testinstance` inside a plan, the validator emits DATA-001 warning a declaration will resolve all variables to null in direct testCase-mode — wire the test into a plan via provar_testplan_add-instance to enable data-driven iteration. When structural errors are returned, consult the provar://docs/step-reference MCP resource for correct step attribute schemas.", + "Validate a Provar XML test case for structural correctness and quality. Checks XML declaration, root element, required attributes (guid UUID v4, testItemId integer), presence, and applies best-practice rules. When a Provar API key is configured (via sf provar auth login or PROVAR_API_KEY env var), calls the Quality Hub API for full 170-rule scoring. Falls back to local validation if no key is set or the API is unavailable. Returns validity_score (schema compliance), quality_score (best practices, 0–100), and validation_source indicating which ruleset was applied. Returns a tri-state status: 'invalid' (a critical defect — the test will not load in Provar, is_valid=false), 'needs_improvement' (loads but quality_score is below quality_threshold), or 'valid' (loads and clears the bar); meets_quality_threshold and the effective quality_threshold are also returned. Note: a critical best-practice violation (e.g. an unknown apiId) now gates is_valid the same way a structural error does — it surfaces in issues[] as an ERROR. major/minor/info best-practice violations affect quality_score (and the status verdict) only. Every response includes run_id — pass it as baseline_run_id in the next call to receive only new/resolved issues. Data-driven note (DATA-001): when file_path is supplied and the project's provardx-properties.json references the test case directly via top-level `testCase` / `testCases` rather than via a `.testinstance` inside a plan, the validator emits DATA-001 warning a declaration will resolve all variables to null in direct testCase-mode — wire the test into a plan via provar_testplan_add-instance to enable data-driven iteration. When structural errors are returned, consult the provar://docs/step-reference MCP resource for correct step attribute schemas.", 'Validate a Provar XML test case: structure, UUIDs, steps, quality scoring; run_id for baseline diff.' ), inputSchema: { @@ -164,6 +188,17 @@ export function registerTestCaseValidate(server: McpServer, config: ServerConfig 'enum summary|standard|full, optional; default standard' ) ), + quality_threshold: z + .number() + .min(0) + .max(100) + .optional() + .describe( + desc( + 'Minimum quality_score for status to be "valid" rather than "needs_improvement". Does NOT affect is_valid (only critical defects do). Precedence: this arg → PROVAR_MCP_QUALITY_THRESHOLD env → 90.', + 'number 0–100, optional; default 90' + ) + ), baseline_run_id: z .string() .optional() @@ -175,7 +210,7 @@ export function registerTestCaseValidate(server: McpServer, config: ServerConfig ), }, }, - async ({ content, xml, file_path, detail, baseline_run_id }) => { + async ({ content, xml, file_path, detail, baseline_run_id, quality_threshold }) => { const requestId = makeRequestId(); log('info', 'provar_testcase_validate', { requestId, has_content: !!(content ?? xml), file_path }); @@ -208,7 +243,15 @@ export function registerTestCaseValidate(server: McpServer, config: ServerConfig const context = tcRunContext(file_path, source); const contextHash = computeContextHash('tc', context); const runId = generateRunId(context); - const bpViolations = (baseResult.best_practices_violations ?? []) as unknown as DiffableViolation[]; + // A bridged critical appears twice — as the surfaced issue AND as its original + // BP violation — both carrying the same BP rule_id. Hand-coded issue rule_ids + // (TC_*, UI-*, COMPARISON-TYPE-001) never collide with BP rule_ids, so a BP + // rule_id appearing in issues[] can only mean it was bridged. Drop those from + // the BP list here so the baseline diff counts each finding once. + const issueRuleIds = new Set(baseResult.issues.map((i) => i.rule_id)); + const bpViolations = (baseResult.best_practices_violations ?? []).filter( + (v) => !issueRuleIds.has(v.rule_id) + ) as unknown as DiffableViolation[]; const currentViolations: DiffableViolation[] = [ ...(baseResult.issues as unknown as DiffableViolation[]), ...bpViolations, @@ -262,12 +305,18 @@ export function registerTestCaseValidate(server: McpServer, config: ServerConfig const completeness_score = calcCompletenessScore(baseResult.is_valid ? 1 : 0, 1); const recommended_next_action = calcNextAction(completeness_score, hasBaseline, currentViolations.length); + // PDX-509 tri-state verdict. is_valid answers "will it load" (gated by + // criticals only); status layers the quality bar on top so an AI client + // gets an explicit "loads but fix before running" signal. + const verdict = deriveQualityVerdict(baseResult.is_valid, baseResult.quality_score, quality_threshold); + const result = { requestId, run_id: runId, completeness_score, recommended_next_action, ...baseResult, + ...verdict, }; const detailLevel = (detail ?? 'standard') as DetailLevel; @@ -336,21 +385,18 @@ export function validateTestCaseXml(filePath: string, config: ServerConfig): Tes /** TC_010/TC_011: validate testCase id and guid attributes. */ function checkTestCaseIdAndGuid(tcId: string | null, tcGuid: string | undefined, issues: ValidationIssue[]): void { - if (!tcId) { - issues.push({ - rule_id: 'TC_010', - severity: 'ERROR', - message: 'testCase missing required id attribute.', - applies_to: 'testCase', - suggestion: 'Add id="1" to testCase element (Provar requires the integer literal "1").', - }); - } else if (tcId !== '1') { + // The testCase `id` is OPTIONAL — the guid is the unique identifier (real Provar + // projects routinely omit id). When an id IS present it must be a non-negative + // integer: test case ids are project-unique sequential numbers (corpus shows 0…N), + // NOT the literal "1". Only a present-but-non-numeric id (e.g. a UUID) is rejected. + if (tcId !== null && !/^\d+$/.test(tcId)) { issues.push({ rule_id: 'TC_010', severity: 'ERROR', - message: `testCase id="${tcId}" is invalid — Provar requires id="1" (integer literal).`, + message: `testCase id="${tcId}" is invalid — when present, the id must be a non-negative integer (test case ids are unique within a project, numbered sequentially). The guid is the cross-project unique identifier.`, applies_to: 'testCase', - suggestion: 'Set id="1" on the testCase element. The unique identifier is the guid attribute, not id.', + suggestion: + 'Set id to a project-unique non-negative integer (typically the highest in use + 1), or omit the id attribute entirely and rely on the guid.', }); } if (!tcGuid) { @@ -672,6 +718,90 @@ function checkUiTarget( } } +// UI-INTERACTION-001 (PDX-506): a UI action step's `interaction` argument must +// use class="uiInteraction". A plain string runs green from the CLI but renders +// the Provar IDE Action widget blank. Mirrors checkUiTarget / UI-LOCATOR-001. +function checkUiInteraction(call: Record, stepName: string, issues: ValidationIssue[]): void { + const interactionArg = getArgList(call).find((a) => (a['@_id'] as string | undefined) === 'interaction'); + if (!interactionArg) return; + const interactionNode = interactionArg['value'] as Record | undefined; + if (interactionNode == null) return; + const valClass = interactionNode['@_class'] as string | undefined; + if (valClass !== 'uiInteraction') { + issues.push({ + rule_id: 'UI-INTERACTION-001', + severity: 'ERROR', + message: `"${stepName}" interaction argument uses class="${ + valClass ?? '(missing)' + }" — must be class="uiInteraction".`, + applies_to: 'apiCall', + suggestion: + 'Emit the interaction as: . ' + + 'In provar_testcase_generate the "interaction" attribute is converted automatically.', + }); + } +} + +// Flat-form argument ids that the broken/CLI-only UiAssert shape carries at the +// top level. The IDE-renderable contract nests these inside a uiFieldAssertion. +const UI_ASSERT_FLAT_ARG_IDS = ['fieldLocator', 'attributeName', 'comparisonType', 'expectedValue']; + +// UI-ASSERT-STRUCTURE-001 (PDX-507, local rule): a UiAssert field assertion must +// be nested inside fieldAssertions → uiFieldAssertion (with a bare +// element), not emitted as flat top-level arguments. +// The flat shape runs green from the CLI but renders the IDE Result Assertions +// tab blank. Confirmed against the real corpus (AllPOCProjects): 0 of 3,778 +// UiAssert steps use a flat fieldLocator argument; ~99% use the nested form. +// (Named "-STRUCTURE-001" to match the local SETVALUES-STRUCTURE-001 convention +// and avoid colliding with the best-practice JSON rule UI-ASSERT-STRUCT-001.) +function checkUiAssertStructure(call: Record, stepName: string, issues: ValidationIssue[]): void { + const flatArg = getArgList(call).find((a) => + UI_ASSERT_FLAT_ARG_IDS.includes((a['@_id'] as string | undefined) ?? '') + ); + if (!flatArg) return; + const offendingId = (flatArg['@_id'] as string | undefined) ?? ''; + issues.push({ + rule_id: 'UI-ASSERT-STRUCTURE-001', + severity: 'ERROR', + message: `UiAssert step "${stepName}" carries a flat "${offendingId}" argument — field assertions must be nested inside fieldAssertions/uiFieldAssertion or the Provar IDE Result Assertions tab renders blank.`, + applies_to: 'apiCall', + suggestion: + 'Nest field assertions: ' + + '' + + '' + + ' plus empty columnAssertions/pageAssertions. The fieldLocator is a bare ' + + 'uri element, NOT class="uiLocator". In provar_testcase_generate pass fieldLocator/attributeName/comparisonType/' + + 'expectedValue as flat attributes — the generator builds this structure automatically.', + }); +} + +// SETVALUES-STRUCTURE-001 (mirrors quality-hub-agents SETVALUES-STRUCTURE-001): +// SetValues values argument must use class="valueList" with children. +// A plain string value causes an immediate ClassCastException at runtime. +// Extracted to keep validateApiCallArgs within the complexity budget. +function checkSetValuesStructure(call: Record, stepName: string, issues: ValidationIssue[]): void { + const valuesArg = getArgList(call).find((a) => (a['@_id'] as string | undefined) === 'values'); + if (!valuesArg) return; + const valuesNode = valuesArg['value'] as Record | undefined; + if (valuesNode == null) return; + const valClass = valuesNode['@_class'] as string | undefined; + if (valClass !== 'valueList') { + issues.push({ + rule_id: 'SETVALUES-STRUCTURE-001', + severity: 'ERROR', + message: `SetValues step "${stepName}" values argument uses class="${ + valClass ?? '(missing)' + }" — must use class="valueList" with children.`, + applies_to: 'apiCall', + suggestion: + 'Wrap variable assignments in: ' + + 'value' + + '. In provar_testcase_generate pass each variable as a flat key/value pair ' + + 'in attributes — the generator builds the valueList structure automatically.', + }); + } +} + function validateApiCallArgs( call: Record, apiId: string, @@ -715,32 +845,18 @@ function validateApiCallArgs( } } - // SETVALUES-STRUCTURE-001 (mirrors quality-hub-agents SETVALUES-STRUCTURE-001): - // SetValues values argument must use class="valueList" with children. - // A plain string value causes an immediate ClassCastException at runtime. + // UI-INTERACTION-001 (PDX-506) for any UI action; UI-ASSERT-STRUCTURE-001 + // (PDX-507) additionally for UiAssert field-assertion structure. Both are + // local rules gated on the shared UI-action set and extracted to helpers to + // keep validateApiCallArgs within the complexity budget. + if (UI_ACTION_API_IDS.has(apiId)) { + checkUiInteraction(call, stepName, issues); + if (apiId.endsWith('UiAssert')) checkUiAssertStructure(call, stepName, issues); + } + + // SETVALUES-STRUCTURE-001: SetValues values argument must use class="valueList". if (apiId.includes('SetValues') && !apiId.includes('AssertValues')) { - const valuesArg = getArgList(call).find((a) => (a['@_id'] as string | undefined) === 'values'); - if (valuesArg) { - const valuesNode = valuesArg['value'] as Record | undefined; - if (valuesNode != null) { - const valClass = valuesNode['@_class'] as string | undefined; - if (valClass !== 'valueList') { - issues.push({ - rule_id: 'SETVALUES-STRUCTURE-001', - severity: 'ERROR', - message: `SetValues step "${stepName}" values argument uses class="${ - valClass ?? '(missing)' - }" — must use class="valueList" with children.`, - applies_to: 'apiCall', - suggestion: - 'Wrap variable assignments in: ' + - 'value' + - '. In provar_testcase_generate pass each variable as a flat key/value pair ' + - 'in attributes — the generator builds the valueList structure automatically.', - }); - } - } - } + checkSetValuesStructure(call, stepName, issues); } // ASSERT-001: AssertValues using UI namedValues format instead of variable format @@ -845,6 +961,107 @@ function validateComparisonTypes(tc: Record, issues: Validation } } +/** + * The Layer-1 structural-validity rule catalog — the single source of truth for + * Layer-1 rule metadata (id / severity / applies_to / description) and the + * best-practice ownership mapping. Loaded from `provar_layer1_rules.json` + * (copied into `lib/mcp/rules/` at compile, same as the best-practices ruleset). + * The imperative DETECTION below is unchanged; centralizing only the metadata + * keeps the published Validation Rule Registry from drifting from the validator. + * The same JSON is read by `scripts/build-validation-rule-registry.cjs` (renders + * the registry rows) and guarded by `validationRuleRegistry.test.ts`. + */ +export interface Layer1RuleCatalogEntry { + id: string; + severity: 'ERROR' | 'WARNING'; + applies_to: string; + description: string; + /** Layer-2 `critical` BP rule ids whose concept this Layer-1 check owns (suppressed from the bridge). */ + owns_bp_rules?: string[]; +} + +export const LAYER1_RULE_CATALOG: readonly Layer1RuleCatalogEntry[] = ((): Layer1RuleCatalogEntry[] => { + const catalogPath = path.join( + path.dirname(fileURLToPath(import.meta.url)), + '..', + 'rules', + 'provar_layer1_rules.json' + ); + const parsed = JSON.parse(fs.readFileSync(catalogPath, 'utf-8')) as { rules: Layer1RuleCatalogEntry[] }; + return parsed.rules; +})(); + +/** + * PDX-509 — validity bridge. + * + * Best-practice violations carry a `critical|major|minor|info` severity, but + * historically that severity only moved `quality_score` and never gated + * `is_valid`. Per the canonical taxonomy, `critical` means "Provar will not + * load/render the test case" — exactly the class of defect that MUST flip + * validity. The bridge surfaces every un-suppressed `critical` BP violation as + * an ERROR issue so it gates `is_valid`. + * + * `major`/`minor`/`info` are intentionally NOT bridged — a major is a runtime + * ERROR that still loads, so it influences the quality score (and the + * `needs_improvement` verdict) without flipping `is_valid`. + * + * These BP rules cover concepts the hand-coded Layer-1 checks already OWN + * (root element, identifier, steps presence, testItemId integers, comparisonType + * enums). Layer-1 is authoritative for those: it fires only on the genuinely + * load-blocking condition (e.g. TC_020 fires on a MISSING , but an + * empty-but-present still loads — it just does nothing), whereas the + * ported BP rule is coarser (VALID-STEPS-001 also flags an empty , a + * "does nothing" design smell, not a load failure). So we never bridge these — + * Layer-1 already gates the load-blocking case, and the BP version still counts + * toward quality_score. The bridge's real value is the criticals Layer-1 does + * NOT check (unknown apiId, missing required control/connection args, invalid + * render value-class casing, NitroX connect args…). + */ +// Derived from the catalog's `owns_bp_rules` — the single source shared with the +// registry generator. A Layer-2 `critical` listed here is NOT bridged into +// `is_valid` (the named Layer-1 check is authoritative; bridging would +// double-report). Ownership today: TC_003→SCHEMA-ROOT-001, TC_010→SCHEMA-ID-001, +// TC_011→VALID-GUID-001, TC_020→SCHEMA-STEPS-001/VALID-STEPS-001, +// TC_034→STEP-ITEMID-001, COMPARISON-TYPE-001→COMPARISON-TYPE-ENUM-001. +export const LAYER1_OWNED_BP_RULES: ReadonlySet = new Set( + LAYER1_RULE_CATALOG.flatMap((rule) => rule.owns_bp_rules ?? []) +); + +/** Map a Best-Practices `appliesTo` token to the issue `applies_to` vocabulary. */ +const BP_APPLIES_TO_ISSUE: Record = { + TestCase: 'testCase', + Step: 'apiCall', + Argument: 'argument', + Document: 'document', +}; + +/** + * Surface every un-suppressed `critical` best-practice violation as an ERROR + * issue so it gates `is_valid`. Mutates `issues` in place. The BP list itself is + * left untouched, so `quality_score` (and its Lambda parity) is unchanged — a + * bridged critical deliberately appears in BOTH `issues[]` (validity) and + * `best_practices_violations[]` (scoring). + */ +function bridgeCriticalViolations( + bpViolations: Array, + issues: ValidationIssue[] +): void { + const present = new Set(issues.map((i) => i.rule_id)); + for (const v of bpViolations) { + if (v.severity !== 'critical') continue; + if (present.has(v.rule_id)) continue; + if (LAYER1_OWNED_BP_RULES.has(v.rule_id)) continue; // Layer-1 is authoritative for these + issues.push({ + rule_id: v.rule_id, + severity: 'ERROR', + message: v.message, + applies_to: BP_APPLIES_TO_ISSUE[v.applies_to[0] ?? ''] ?? 'testCase', + suggestion: v.recommendation, + }); + present.add(v.rule_id); + } +} + function finalize( issues: ValidationIssue[], testCaseId: string | null, @@ -853,15 +1070,19 @@ function finalize( xmlContent: string, testName?: string ): TestCaseValidationResult { + // Layer 2: quality score (best practices engine — same rules & formula as Quality Hub API) + const bp = runBestPractices(xmlContent, { testName }); + + // PDX-509: bridge critical BP violations into Layer-1 issues BEFORE counting, + // so a "won't load" best-practice defect flips is_valid like a structural one. + bridgeCriticalViolations(bp.violations, issues); + const errorCount = issues.filter((i) => i.severity === 'ERROR').length; const warningCount = issues.filter((i) => i.severity === 'WARNING').length; - // Layer 1: validity score (schema compliance — existing rules) + // Layer 1: validity score (schema compliance — existing rules + bridged criticals) const validity_score = Math.max(0, 100 - errorCount * 20); - // Layer 2: quality score (best practices engine — same rules & formula as Quality Hub API) - const bp = runBestPractices(xmlContent, { testName }); - return { is_valid: errorCount === 0, validity_score, diff --git a/src/mcp/tools/testPlanValidate.ts b/src/mcp/tools/testPlanValidate.ts index d1e91fc9..c6830627 100644 --- a/src/mcp/tools/testPlanValidate.ts +++ b/src/mcp/tools/testPlanValidate.ts @@ -12,6 +12,7 @@ import { makeError, makeRequestId } from '../schemas/common.js'; import { log } from '../logging/logger.js'; import { applyDetailLevel, type DetailLevel } from '../utils/detailLevel.js'; import { calcCompletenessScore, calcNextAction } from '../utils/validationScore.js'; +import { resolveQualityThreshold } from '../utils/qualityThreshold.js'; import { validatePlan, buildHierarchySummary, @@ -142,7 +143,10 @@ export function registerTestPlanValidate(server: McpServer): void { .max(100) .optional() .describe( - desc('Minimum quality score for a test case to be considered valid (default: 80)', 'number 0–100, optional') + desc( + 'Minimum quality score for a test case to be considered valid. Precedence: this arg → PROVAR_MCP_QUALITY_THRESHOLD env → 90.', + 'number 0–100, optional; default 90' + ) ), detail: z .enum(['summary', 'standard', 'full']) @@ -161,7 +165,7 @@ export function registerTestPlanValidate(server: McpServer): void { log('info', 'provar_testplan_validate', { requestId, plan_name }); try { - const threshold = quality_threshold ?? 80; + const threshold = resolveQualityThreshold(quality_threshold); const input: TestPlanInput = { name: plan_name, test_suites: test_suites ?? [], diff --git a/src/mcp/tools/testSuiteValidate.ts b/src/mcp/tools/testSuiteValidate.ts index 07e96c47..c5be195b 100644 --- a/src/mcp/tools/testSuiteValidate.ts +++ b/src/mcp/tools/testSuiteValidate.ts @@ -12,6 +12,7 @@ import { makeError, makeRequestId } from '../schemas/common.js'; import { log } from '../logging/logger.js'; import { applyDetailLevel, type DetailLevel } from '../utils/detailLevel.js'; import { calcCompletenessScore, calcNextAction } from '../utils/validationScore.js'; +import { resolveQualityThreshold } from '../utils/qualityThreshold.js'; import { generateRunId, saveRun, @@ -114,7 +115,10 @@ export function registerTestSuiteValidate(server: McpServer): void { .max(100) .optional() .describe( - desc('Minimum quality score for a test case to be considered valid (default: 80)', 'number 0–100, optional') + desc( + 'Minimum quality score for a test case to be considered valid. Precedence: this arg → PROVAR_MCP_QUALITY_THRESHOLD env → 90.', + 'number 0–100, optional; default 90' + ) ), detail: z .enum(['summary', 'standard', 'full']) @@ -142,7 +146,7 @@ export function registerTestSuiteValidate(server: McpServer): void { log('info', 'provar_testsuite_validate', { requestId, suite_name }); try { - const threshold = quality_threshold ?? 80; + const threshold = resolveQualityThreshold(quality_threshold); const input: TestSuiteInput = { name: suite_name, test_cases: test_cases ?? [], diff --git a/src/mcp/utils/qualityThreshold.ts b/src/mcp/utils/qualityThreshold.ts new file mode 100644 index 00000000..39f46031 --- /dev/null +++ b/src/mcp/utils/qualityThreshold.ts @@ -0,0 +1,45 @@ +/* + * Copyright (c) 2024 Provar Limited. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.md file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ + +/* + * Quality-threshold resolution for the validation tools. + * + * A test case "meets quality" when its best-practices `quality_score` is at or + * above this threshold. The bar is resolved with the precedence: + * per-call `quality_threshold` arg → PROVAR_MCP_QUALITY_THRESHOLD env → default (90). + * + * The default was raised from 80 to 90 so that "valid" means genuinely + * production-ready, not merely "loads". The env var lets a team pin a house + * standard (e.g. relax to 80 during a migration) without threading an argument + * through every call. Out-of-range or unparseable values fall through to the + * next source — mirroring the depth-guard convention in server.ts. + */ + +export const DEFAULT_QUALITY_THRESHOLD = 90; + +/** True when `n` is a finite number inside the inclusive 0–100 score range. */ +function inRange(n: number): boolean { + return Number.isFinite(n) && n >= 0 && n <= 100; +} + +/** + * Resolve the effective quality threshold. + * + * @param perCallArg The tool's `quality_threshold` argument, if the caller set one. + * @returns A number in 0–100; never NaN. + */ +export function resolveQualityThreshold(perCallArg?: number): number { + if (typeof perCallArg === 'number' && inRange(perCallArg)) return perCallArg; + + const raw = process.env['PROVAR_MCP_QUALITY_THRESHOLD']; + if (raw !== undefined && raw.trim() !== '') { + const parsed = Number(raw); + if (inRange(parsed)) return parsed; + } + + return DEFAULT_QUALITY_THRESHOLD; +} diff --git a/src/mcp/utils/testCaseId.ts b/src/mcp/utils/testCaseId.ts new file mode 100644 index 00000000..11c00354 --- /dev/null +++ b/src/mcp/utils/testCaseId.ts @@ -0,0 +1,170 @@ +/* + * Copyright (c) 2024 Provar Limited. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.md file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ + +import fs from 'node:fs'; +import path from 'node:path'; +import { assertPathAllowed } from '../security/pathPolicy.js'; + +/* + * Root-`` id allocation for the generator. + * + * Important: the `id` attribute is NOT a uniqueness key. A corpus sweep of 651 + * real `.testcase` files shows id values duplicate freely within a single + * project (id="0" appears 9× in one project), there is no project-level "next + * id" counter file, and the Quality Hub backend requires only ONE of + * id/guid/registryId — never checking id uniqueness or integer-ness. The + * generator already emits a unique `guid`, which is the real identifier. + * + * So duplicate ids cause no runtime or validation failure. This allocator is a + * convention-alignment nicety: when we can see the surrounding project, pick the + * next integer after the highest in use so a freshly generated case does not + * carry a confusing duplicate id="1". Where there is no project context (preview + * runs, or output outside the allowed roots) we keep the historical default. + */ + +export const DEFAULT_TESTCASE_ID = 1; + +/** How an id was chosen — surfaced for logging and the tool response. */ +export type TestCaseIdBasis = 'preserved-existing' | 'project-max-plus-1' | 'default'; + +export interface TestCaseIdAllocation { + id: number; + basis: TestCaseIdBasis; + /** Project root that was scanned, when basis is project-max-plus-1. */ + projectRoot?: string; + /** Highest numeric id found in the project, when basis is project-max-plus-1. */ + highestExistingId?: number; +} + +/** Non-throwing wrapper around assertPathAllowed. */ +function isAllowed(p: string, allowedPaths: string[]): boolean { + try { + assertPathAllowed(p, allowedPaths); + return true; + } catch { + return false; + } +} + +/** Read the first `bytes` of a file as UTF-8 (the root element lives at the top). */ +function readPrefix(file: string, bytes = 8192): string { + const fd = fs.openSync(file, 'r'); + try { + const buf = Buffer.alloc(bytes); + const read = fs.readSync(fd, buf, 0, bytes, 0); + return buf.toString('utf-8', 0, read); + } finally { + fs.closeSync(fd); + } +} + +/** + * The numeric `id` of the root `` element, or undefined when the file + * is unreadable, has no id, or the id is non-numeric (e.g. a UUID). `[^>]*?` + * keeps the match inside the opening tag, so only the root id is considered. + */ +function readRootTestCaseId(file: string): number | undefined { + try { + const match = readPrefix(file).match(/]*?\bid=["'](\d+)["']/); + if (!match) return undefined; + const id = Number.parseInt(match[1], 10); + // Ignore ids too large to round-trip as an exact integer: otherwise `max + 1` + // would render in scientific notation (e.g. id="1e+22"), an invalid id. + return Number.isSafeInteger(id) ? id : undefined; + } catch { + return undefined; + } +} + +/** + * Walk up from `startDir` to the nearest ancestor containing a `.testproject` + * marker, staying within the allowed roots. Returns undefined when no project + * marker is reachable. + */ +function findProjectRoot(startDir: string, allowedPaths: string[]): string | undefined { + let dir = path.resolve(startDir); + // Bound the walk: a `.testproject` we cannot read (outside allowed roots) is + // not ours to scan, and the loop terminates at the filesystem root. + for (;;) { + if (!isAllowed(dir, allowedPaths)) return undefined; + if (fs.existsSync(path.join(dir, '.testproject'))) return dir; + const parent = path.dirname(dir); + if (parent === dir) return undefined; + dir = parent; + } +} + +/** + * Highest numeric root-`` id under `projectRoot` (scanning `tests/` + * when present, else the root), ignoring `excludeFile`. Undefined when no + * numeric id is found. + */ +function maxProjectTestCaseId(projectRoot: string, excludeFile: string): number | undefined { + const scanRoot = fs.existsSync(path.join(projectRoot, 'tests')) ? path.join(projectRoot, 'tests') : projectRoot; + let max: number | undefined; + + const walk = (dir: string): void => { + let entries: fs.Dirent[]; + try { + entries = fs.readdirSync(dir, { withFileTypes: true }); + } catch { + return; + } + for (const entry of entries) { + if (entry.name.startsWith('.') || entry.name === 'node_modules') continue; + const full = path.join(dir, entry.name); + if (entry.isDirectory()) { + walk(full); + } else if ( + // Only read real files inside the project. A symlinked `.testcase` could + // point outside the allowed roots; skip it so the scan honours the + // path-policy containment promised by findProjectRoot. (Dir symlinks are + // already skipped — isDirectory() is false for them.) + entry.isFile() && + !entry.isSymbolicLink() && + entry.name.endsWith('.testcase') && + path.resolve(full) !== excludeFile + ) { + const id = readRootTestCaseId(full); + if (id !== undefined && (max === undefined || id > max)) max = id; + } + } + }; + + walk(scanRoot); + return max; +} + +/** + * Choose the `id` for a test case the generator is about to write to disk. + * + * Precedence: + * 1. Overwriting an existing file with a numeric id → preserve it (stable regeneration). + * 2. Output sits inside a reachable Provar project → highest project id + 1. + * 3. Otherwise → DEFAULT_TESTCASE_ID. + * + * @param outputPath The `.testcase` file that will be written (need not exist yet). + * @param allowedPaths The MCP path-policy roots; the project scan never leaves them. + */ +export function allocateTestCaseId(outputPath: string, allowedPaths: string[]): TestCaseIdAllocation { + const resolved = path.resolve(outputPath); + + if (fs.existsSync(resolved)) { + const selfId = readRootTestCaseId(resolved); + if (selfId !== undefined) return { id: selfId, basis: 'preserved-existing' }; + } + + const projectRoot = findProjectRoot(path.dirname(resolved), allowedPaths); + if (projectRoot) { + const max = maxProjectTestCaseId(projectRoot, resolved); + if (max !== undefined) { + return { id: max + 1, basis: 'project-max-plus-1', projectRoot, highestExistingId: max }; + } + } + + return { id: DEFAULT_TESTCASE_ID, basis: 'default' }; +} diff --git a/src/services/projectValidation.ts b/src/services/projectValidation.ts index f4bea14a..5744eeb2 100644 --- a/src/services/projectValidation.ts +++ b/src/services/projectValidation.ts @@ -23,6 +23,7 @@ import { type HierarchyViolation, type HierarchySummary, } from '../mcp/tools/hierarchyValidate.js'; +import { resolveQualityThreshold } from '../mcp/utils/qualityThreshold.js'; // ── Public error type ───────────────────────────────────────────────────────── @@ -39,7 +40,7 @@ export class ProjectValidationError extends Error { export interface ProjectValidationOptions { project_path: string; - quality_threshold?: number; // default 80 + quality_threshold?: number; // resolved via resolveQualityThreshold: arg → PROVAR_MCP_QUALITY_THRESHOLD → 90 save_results?: boolean; // default true (any value !== false means save) results_dir?: string; // default '{project_path}/provardx/validation' } @@ -842,7 +843,7 @@ export function validateProjectFromPath(options: ProjectValidationOptions): Proj ); } - const threshold = quality_threshold ?? 80; + const threshold = resolveQualityThreshold(quality_threshold); // 1. Read project context from .testproject const { projectName, context } = readProjectContext(projectRoot); diff --git a/test/fixtures/testcases/Contact_Lead_nested.testcase b/test/fixtures/testcases/Contact_Lead_nested.testcase index ea3bcd3b..606ac57d 100644 --- a/test/fixtures/testcases/Contact_Lead_nested.testcase +++ b/test/fixtures/testcases/Contact_Lead_nested.testcase @@ -304,25 +304,28 @@ - - Field + + Values - - ui:locator?name=LastName - - - value - - - EqualTo + + Test - - - + + + + + + + + + + + + - - + + false @@ -330,37 +333,34 @@ - - Test - - - false - - - Field + + Values - - ui:locator?name=Company - - - value - - - EqualTo + + Test - - - + + + + + + + + + + + + - - + + false @@ -368,12 +368,6 @@ - - Test - - - false - diff --git a/test/unit/mcp/bestPracticesEngine.test.ts b/test/unit/mcp/bestPracticesEngine.test.ts index 4c2b0bb1..350a5fda 100644 --- a/test/unit/mcp/bestPracticesEngine.test.ts +++ b/test/unit/mcp/bestPracticesEngine.test.ts @@ -573,3 +573,1057 @@ describe('runBestPractices', () => { }); }); }); + +// ── mustContainArgument validator (PDX-508) ────────────────────────────────── + +describe('mustContainArgument validator', () => { + const GUID_MCA_TC = '550e8400-e29b-41d4-a716-4466554409a0'; + const GUID_MCA_S1 = '550e8400-e29b-41d4-a716-4466554409a1'; + const GUID_MCA_S2 = '550e8400-e29b-41d4-a716-4466554409a2'; + + // Build a single-step (or multi-step) test case from raw XML. + function buildTc(stepsXml: string): string { + return ` + + +${stepsXml} + +`; + } + + function find(violations: BPViolation[], ruleId: string): BPViolation | undefined { + return violations.find((v) => v.rule_id === ruleId); + } + + // CONTROL-IF-001 — If steps must have a 'condition' argument (critical / weight 8). + const IF_API = 'com.provar.plugins.bundled.apis.If'; + + it('fires when an If step is missing its required condition argument', () => { + const xml = buildTc(` `); + const v = find(runBestPractices(xml).violations, 'CONTROL-IF-001'); + assert.ok(v, 'Expected CONTROL-IF-001 to fire for an If step with no condition'); + assert.equal(v?.severity, 'critical'); + assert.ok(v?.message.includes('condition'), `Message should name the argument: ${v?.message}`); + assert.ok(v?.message.includes('testItemId=1'), `Message should locate the step: ${v?.message}`); + }); + + it('passes when the If step has a populated condition argument', () => { + const xml = buildTc(` + + + {{count}} == 1 + + + `); + const v = find(runBestPractices(xml).violations, 'CONTROL-IF-001'); + assert.ok(!v, `Expected no CONTROL-IF-001 violation, got: ${v?.message}`); + }); + + it('fires when the required argument is present but an empty self-closing tag', () => { + const xml = buildTc(` + + + + `); + const v = find(runBestPractices(xml).violations, 'CONTROL-IF-001'); + assert.ok(v, 'Expected CONTROL-IF-001 to fire when condition has no '); + }); + + // ASSERT-COMPARISON-001 — AssertValues must have a 'comparisonType' argument. + const ASSERT_API = 'com.provar.plugins.bundled.apis.AssertValues'; + + it('emits a single violation per rule for multiple offenders and does not inflate count (score parity)', () => { + // The Quality Hub back-end returns one violation per rule; omitting `count` + // keeps the weighted-deduction score in parity with the Lambda. + const xml = buildTc(` + `); + const matches = runBestPractices(xml).violations.filter((v) => v.rule_id === 'ASSERT-COMPARISON-001'); + assert.equal(matches.length, 1, 'Expected exactly one ASSERT-COMPARISON-001 violation'); + assert.equal(matches[0].count, undefined, 'count must be unset so the score stays in parity with the back-end'); + assert.ok(matches[0].message.includes('and 0 more') === false); + assert.ok( + matches[0].message.includes('testItemId=1') && matches[0].message.includes('testItemId=2'), + `Message should still name both offenders: ${matches[0].message}` + ); + }); + + it('fires for a disabled step missing the argument (back-end does not skip disabled steps)', () => { + const xml = buildTc(` + + disabled + + `); + const v = find(runBestPractices(xml).violations, 'CONTROL-IF-001'); + assert.ok(v, 'A missing required argument is load-blocking even on a disabled step (matches the back-end)'); + }); + + it('fires when the argument is present but its element is empty (present-and-non-empty semantics)', () => { + const xml = buildTc(` + + + + `); + const v = find(runBestPractices(xml).violations, 'CONTROL-IF-001'); + assert.ok(v, 'Expected CONTROL-IF-001 to fire for an empty (mirrors the back-end)'); + }); + + it('passes when the condition is a variable reference with a child', () => { + const xml = buildTc(` + + + + `); + const v = find(runBestPractices(xml).violations, 'CONTROL-IF-001'); + assert.ok(!v, `A variable reference with a is a valid condition, got: ${v?.message}`); + }); + + it('fires for a bare with no path or text (effectively empty)', () => { + const xml = buildTc(` + + + + `); + const v = find(runBestPractices(xml).violations, 'CONTROL-IF-001'); + assert.ok(v, 'A bare variable value with no path/text is treated as missing (mirrors the back-end)'); + }); + + it('passes when the condition is a comparison-operator value (e.g. class="gt")', () => { + const xml = buildTc(` + + + + `); + const v = find(runBestPractices(xml).violations, 'CONTROL-IF-001'); + assert.ok(!v, `A comparison-operator condition is valid, got: ${v?.message}`); + }); + + it('passes an If with no condition argument when the condition is carried in the title (legacy format)', () => { + const xml = buildTc( + ` ` + ); + const v = find(runBestPractices(xml).violations, 'CONTROL-IF-001'); + assert.ok(!v, `Legacy condition-in-title format should pass, got: ${v?.message}`); + }); + + it('does not fire for a different apiId that happens to lack the argument', () => { + const xml = buildTc( + ` ` + ); + const v = find(runBestPractices(xml).violations, 'CONTROL-IF-001'); + assert.ok(!v, 'CONTROL-IF-001 must only apply to If steps'); + }); + + it('checks steps nested inside control-flow containers (recursive)', () => { + const xml = + buildTc(` + + {{rows}} + row + + + + + + + + + `); + const v = find(runBestPractices(xml).violations, 'CONTROL-IF-001'); + assert.ok(v, 'Expected CONTROL-IF-001 to fire for an If nested inside a ForEach'); + assert.ok(v?.message.includes('testItemId=3'), `Message should locate the nested step: ${v?.message}`); + }); +}); + +describe('render / load-blocking validators (Tier 2)', () => { + const GUID_T2_TC = '550e8400-e29b-41d4-a716-4466554408b0'; + const GUID_T2_S1 = '550e8400-e29b-41d4-a716-4466554408b1'; + const APEX_CONNECT = 'com.provar.plugins.forcedotcom.core.testapis.ApexConnect'; + const SET_VALUES = 'com.provar.plugins.bundled.apis.control.SetValues'; + + // Wrap raw XML in a minimal, schema-valid test case. + function buildTc(stepsXml: string): string { + return ` + + +${stepsXml} + +`; + } + + // Wrap a single in an AssertValues argument (whole-tree validators don't need apiCall context). + function buildValueStep(valueXml: string): string { + return buildTc(` + + ${valueXml} + + `); + } + + function find(violations: BPViolation[], ruleId: string): BPViolation | undefined { + return violations.find((v) => v.rule_id === ruleId); + } + + // ── valueClassCasing (RENDER-CASE-001) ────────────────────────────────────── + describe('valueClassCasing — RENDER-CASE-001', () => { + it('fires when a known valueClass is spelled with wrong case', () => { + const v = find( + runBestPractices(buildValueStep('true')).violations, + 'RENDER-CASE-001' + ); + assert.ok(v, 'Expected RENDER-CASE-001 to fire for valueClass="Boolean"'); + assert.equal(v?.severity, 'critical'); + assert.ok(v?.message.includes("'boolean'"), `Message should suggest lowercase: ${v?.message}`); + }); + + it('passes when the valueClass is already lowercase', () => { + const v = find( + runBestPractices(buildValueStep('true')).violations, + 'RENDER-CASE-001' + ); + assert.ok(!v, `Lowercase valueClass should pass, got: ${v?.message}`); + }); + + it('does not fire for class-attribute tokens used as a valueClass (e.g. FuncCall) — out of scope', () => { + // funcCall/valueList/variable/operators are `class="..."` values, never `valueClass` + // values, so they are not in the corpus-confirmed valueClass set and are not normalized. + const v = find( + runBestPractices(buildValueStep('x')).violations, + 'RENDER-CASE-001' + ); + assert.ok(!v, 'valueClass="FuncCall" must NOT fire — funcCall is a class value, not a valueClass'); + }); + + it('normalizes valueClass="ID" → "id" (id is a real corpus valueClass)', () => { + const v = find( + runBestPractices(buildValueStep('001x')).violations, + 'RENDER-CASE-001' + ); + assert.ok(v, 'Expected RENDER-CASE-001 to fire for valueClass="ID"'); + assert.ok(v?.message.includes("'id'"), `Message should suggest lowercase 'id': ${v?.message}`); + }); + + it('passes for camelCase dateTime and does not fire for already-correct id', () => { + const r = runBestPractices( + buildValueStep('1736899200000') + ).violations.find((x) => x.rule_id === 'RENDER-CASE-001'); + assert.ok(!r, 'valueClass="dateTime" is canonical and must pass'); + }); + }); + + // ── booleanCasing (RENDER-BOOL-001) ───────────────────────────────────────── + describe('booleanCasing — RENDER-BOOL-001', () => { + it('fires for an uppercase boolean value', () => { + const v = find( + runBestPractices(buildValueStep('True')).violations, + 'RENDER-BOOL-001' + ); + assert.ok(v, 'Expected RENDER-BOOL-001 to fire for "True"'); + assert.equal(v?.severity, 'critical'); + }); + + it('passes for a lowercase boolean value', () => { + const v = find( + runBestPractices(buildValueStep('false')).violations, + 'RENDER-BOOL-001' + ); + assert.ok(!v, `Lowercase boolean should pass, got: ${v?.message}`); + }); + + it('does not fire when the value is not a boolean valueClass', () => { + const v = find( + runBestPractices(buildValueStep('True')).violations, + 'RENDER-BOOL-001' + ); + assert.ok(!v, 'Only valueClass="boolean" values are checked for boolean casing'); + }); + }); + + // ── invalidValueClass (VALUE-CLASS-001) ───────────────────────────────────── + describe('invalidValueClass — VALUE-CLASS-001', () => { + it('fires for a hallucinated class like "null"', () => { + const v = find(runBestPractices(buildValueStep('')).violations, 'VALUE-CLASS-001'); + assert.ok(v, 'Expected VALUE-CLASS-001 to fire for class="null"'); + assert.equal(v?.severity, 'critical'); + assert.ok(v?.message.includes('actualValue'), `Message should name the argument: ${v?.message}`); + }); + + it('fires for an invalid valueClass on a class="value" element', () => { + const v = find( + runBestPractices(buildValueStep('1')).violations, + 'VALUE-CLASS-001' + ); + assert.ok(v, 'Expected VALUE-CLASS-001 to fire for valueClass="money"'); + assert.ok(v?.message.includes('valueClass'), `Message should call out the valueClass: ${v?.message}`); + }); + + it('passes for a valid class="value" valueClass="string"', () => { + const v = find( + runBestPractices(buildValueStep('x')).violations, + 'VALUE-CLASS-001' + ); + assert.ok(!v, `A valid value class should pass, got: ${v?.message}`); + }); + + it('accepts class="invalid" (back-end hardcodes it in the valid set — parity)', () => { + const v = find(runBestPractices(buildValueStep('')).violations, 'VALUE-CLASS-001'); + assert.ok(!v, 'class="invalid" is accepted by the back-end and must not fire here'); + }); + }); + + // ── dateValueClassFormat (RENDER-DATE-VALUECLASS-001) ─────────────────────── + describe('dateValueClassFormat — RENDER-DATE-VALUECLASS-001', () => { + it('fires for an ISO date string with valueClass="date"', () => { + const v = find( + runBestPractices(buildValueStep('2025-01-15')).violations, + 'RENDER-DATE-VALUECLASS-001' + ); + assert.ok(v, 'Expected the date-format rule to fire for an ISO string'); + assert.equal(v?.severity, 'critical'); + }); + + it('passes for an epoch-millis integer with valueClass="date"', () => { + const v = find( + runBestPractices(buildValueStep('1736899200000')).violations, + 'RENDER-DATE-VALUECLASS-001' + ); + assert.ok(!v, `Epoch millis should pass, got: ${v?.message}`); + }); + + it('does not fire when the same string is stored as valueClass="string"', () => { + const v = find( + runBestPractices(buildValueStep('2025-01-15')).violations, + 'RENDER-DATE-VALUECLASS-001' + ); + assert.ok(!v, 'Only date/dateTime valueClasses are checked for epoch format'); + }); + }); + + // ── apexConnectReuseConnection (APEX-REUSE-CONN-001) ──────────────────────── + describe('apexConnectReuseConnection — APEX-REUSE-CONN-001', () => { + it('fires when reuseConnectionName carries a non-empty value', () => { + const xml = buildTc(` + + Reuse + + `); + const v = find(runBestPractices(xml).violations, 'APEX-REUSE-CONN-001'); + assert.ok(v, 'Expected APEX-REUSE-CONN-001 to fire for a non-empty reuseConnectionName'); + assert.equal(v?.severity, 'major'); + }); + + it('passes when reuseConnectionName is left blank', () => { + const xml = buildTc(` + + + + `); + const v = find(runBestPractices(xml).violations, 'APEX-REUSE-CONN-001'); + assert.ok(!v, `A blank reuseConnectionName should pass, got: ${v?.message}`); + }); + }); + + // ── apexConnectValidArguments (APEX-CONNECT-ARGS-001) ─────────────────────── + describe('apexConnectValidArguments — APEX-CONNECT-ARGS-001', () => { + it('fires for a hallucinated argument id', () => { + const xml = buildTc(` + + SF + 30 + + `); + const v = find(runBestPractices(xml).violations, 'APEX-CONNECT-ARGS-001'); + assert.ok(v, 'Expected APEX-CONNECT-ARGS-001 to fire for commandTimeout'); + assert.equal(v?.severity, 'critical'); + assert.ok(v?.message.includes('commandTimeout'), `Message should name the bad arg: ${v?.message}`); + }); + + it('passes when all argument ids are in the whitelist', () => { + const xml = buildTc(` + + SF + conn + + `); + const v = find(runBestPractices(xml).violations, 'APEX-CONNECT-ARGS-001'); + assert.ok(!v, `All-valid arguments should pass, got: ${v?.message}`); + }); + }); + + // ── apexConnectConnectionIdValueClass (APEX-CONNECT-CONNID-001) ───────────── + describe('apexConnectConnectionIdValueClass — APEX-CONNECT-CONNID-001', () => { + it('fires when connectionId uses valueClass="string"', () => { + const xml = buildTc(` + + default + + `); + const v = find(runBestPractices(xml).violations, 'APEX-CONNECT-CONNID-001'); + assert.ok(v, 'Expected APEX-CONNECT-CONNID-001 to fire for valueClass="string"'); + assert.equal(v?.severity, 'critical'); + }); + + it('passes when connectionId uses valueClass="id"', () => { + const xml = buildTc(` + + bce7fd3f-0f81-4c5c-ab68-c3edd44b5d1e + + `); + const v = find(runBestPractices(xml).violations, 'APEX-CONNECT-CONNID-001'); + assert.ok(!v, `valueClass="id" should pass, got: ${v?.message}`); + }); + + it('passes when connectionId is left empty', () => { + const xml = buildTc(` + + + + `); + const v = find(runBestPractices(xml).violations, 'APEX-CONNECT-CONNID-001'); + assert.ok(!v, `An empty connectionId should pass, got: ${v?.message}`); + }); + }); + + // ── setValuesInvalidElements (SETVALUES-INVALID-ELEMENT-001) ──────────────── + describe('setValuesInvalidElements — SETVALUES-INVALID-ELEMENT-001', () => { + it('fires for hallucinated / elements', () => { + const xml = buildTc(` + + + + + + LeadId + + + + + + + `); + const v = find(runBestPractices(xml).violations, 'SETVALUES-INVALID-ELEMENT-001'); + assert.ok(v, 'Expected SETVALUES-INVALID-ELEMENT-001 to fire for /'); + assert.equal(v?.severity, 'critical'); + }); + + it('passes for a correctly-structured SetValues step', () => { + const xml = buildTc(` + + + + + MyVar + 1 + Test + + + + + `); + const v = find(runBestPractices(xml).violations, 'SETVALUES-INVALID-ELEMENT-001'); + assert.ok(!v, `A correctly-structured SetValues step should pass, got: ${v?.message}`); + }); + }); +}); + +describe('back-end-only rules (Tier 4)', () => { + const GUID_T4_TC = '550e8400-e29b-41d4-a716-4466554407c0'; + const GUID_T4_S1 = '550e8400-e29b-41d4-a716-4466554407c1'; + const APEX_CREATE = 'com.provar.plugins.forcedotcom.core.testapis.ApexCreateObject'; + const ASSERT_VALUES = 'com.provar.plugins.bundled.apis.AssertValues'; + const SET_VALUES = 'com.provar.plugins.bundled.apis.control.SetValues'; + const DB_CONNECT = 'com.provar.plugins.bundled.apis.db.DbConnect'; + const SQL_QUERY = 'com.provar.plugins.bundled.apis.db.SqlQuery'; + const UI_DO_ACTION = 'com.provar.plugins.forcedotcom.core.ui.UiDoAction'; + + function buildTc(stepsXml: string): string { + return ` + + +${stepsXml} + +`; + } + + function find(violations: BPViolation[], ruleId: string): BPViolation | undefined { + return violations.find((v) => v.rule_id === ruleId); + } + + function countOf(violations: BPViolation[], ruleId: string): number { + return violations.filter((v) => v.rule_id === ruleId).length; + } + + // Wrap one around a raw on an ApexCreateObject step (avoids triggering other Tier 4 rules). + function buildArgStep(valueXml: string): string { + return buildTc(` + + ${valueXml} + + `); + } + + // ── numeric tag-value robustness (no .trim crash) ────────────────────────── + describe('numeric text does not crash the engine', () => { + it('runs cleanly when a value element holds a bare number (parsed as a JS number)', () => { + // fast-xml-parser yields a NUMBER for 123456; a bare `.trim()` + // on that previously threw "(...).trim is not a function" and turned real + // validation into a VALIDATE_ERROR. nodeText coerces to string first. + const xml = buildArgStep('123456'); + let result: ReturnType | undefined; + assert.doesNotThrow(() => { + result = runBestPractices(xml); + }, 'numeric tag value must not crash runBestPractices'); + assert.ok(result && typeof result.quality_score === 'number'); + }); + }); + + // ── varStringLiteral (VAR-STRING-LITERAL-001) — multi-violation ───────────── + describe('varStringLiteral — VAR-STRING-LITERAL-001', () => { + it('fires for a {Var} stored as class="value" valueClass="string"', () => { + const v = find( + runBestPractices(buildArgStep('{AccountId}')).violations, + 'VAR-STRING-LITERAL-001' + ); + assert.ok(v, 'Expected VAR-STRING-LITERAL-001 to fire for {AccountId}'); + assert.equal( + v?.severity, + 'major', + 'a runtime execution error (the test still loads) — major, not minor/critical' + ); + }); + + it('emits one violation per offending value (back-end returns a list)', () => { + const xml = buildTc(` + + {AccountId} + {Obj.Field} + + `); + assert.equal( + countOf(runBestPractices(xml).violations, 'VAR-STRING-LITERAL-001'), + 2, + 'each offending value is its own violation' + ); + }); + + it('passes when the value is a proper class="variable" reference', () => { + const v = find( + runBestPractices(buildArgStep('')).violations, + 'VAR-STRING-LITERAL-001' + ); + assert.ok(!v, `A class="variable" reference should pass, got: ${v?.message}`); + }); + + it('fires for UI-target args (sfUiTargetObjectId) — a bare {Var} is NOT interpolated there', () => { + // Field evidence: a literal {AccountId} in sfUiTargetObjectId lands in the + // URL as %7BAccountId%7D and the step hard-fails. The back-end exemption was + // wrong; we removed it locally, so the rule must now fire. + const xml = buildTc(` + + {AccountId} + + `); + const v = find(runBestPractices(xml).violations, 'VAR-STRING-LITERAL-001'); + assert.ok(v, 'sfUiTargetObjectId must flag a bare {Var} literal — it is not interpolated at runtime'); + assert.equal(v?.severity, 'major'); + }); + + it('still passes for binding-style {ns:key} expressions (the colon is excluded)', () => { + const xml = buildTc(` + + {targetUrl:object} + + `); + const v = find(runBestPractices(xml).violations, 'VAR-STRING-LITERAL-001'); + assert.ok(!v, 'binding-style {ns:key} is safe and must not be flagged'); + }); + + it('does not fire when the surrounding text is more than a bare token', () => { + const v = find( + runBestPractices(buildArgStep('Hello {Name}')).violations, + 'VAR-STRING-LITERAL-001' + ); + assert.ok(!v, 'only a whole-string {Token} is flagged, not embedded references'); + }); + }); + + // ── dbConnectResultNameMismatch (CONN-DB-002) ────────────────────────────── + describe('dbConnectResultNameMismatch — CONN-DB-002', () => { + function dbStep(apiId: string, argId: string, value: string, tid: string): string { + return ` + + ${value} + + `; + } + + it('fires when a DB operation references a dbConnectionName that no DbConnect produced', () => { + const xml = buildTc( + `${dbStep(DB_CONNECT, 'resultName', 'SQLServer', '1')}\n${dbStep( + SQL_QUERY, + 'dbConnectionName', + 'WrongName', + '2' + )}` + ); + const v = find(runBestPractices(xml).violations, 'CONN-DB-002'); + assert.ok(v, 'Expected CONN-DB-002 to fire for a mismatched dbConnectionName'); + assert.equal(v?.severity, 'major'); + assert.ok(v?.message.includes('SQLServer'), `Message should name the valid resultName(s): ${v?.message}`); + }); + + it('passes when the dbConnectionName matches the DbConnect resultName', () => { + const xml = buildTc( + `${dbStep(DB_CONNECT, 'resultName', 'SQLServer', '1')}\n${dbStep( + SQL_QUERY, + 'dbConnectionName', + 'SQLServer', + '2' + )}` + ); + const v = find(runBestPractices(xml).violations, 'CONN-DB-002'); + assert.ok(!v, `A matching dbConnectionName should pass, got: ${v?.message}`); + }); + + it('does not fire when there is no DbConnect (delegated to CONN-DB-001)', () => { + const xml = buildTc(dbStep(SQL_QUERY, 'dbConnectionName', 'SQLServer', '1')); + const v = find(runBestPractices(xml).violations, 'CONN-DB-002'); + assert.ok(!v, 'with no DbConnect resultName the mismatch rule defers to CONN-DB-001'); + }); + }); + + // ── setValuesFuncCallString (SETVALUES-FUNC-STR-001) ─────────────────────── + describe('setValuesFuncCallString — SETVALUES-FUNC-STR-001', () => { + function setValuesStep(valueXml: string): string { + return buildTc(` + + + + + ${valueXml} + + + + + `); + } + + it('fires for a {Func(args)} string interpolation', () => { + const v = find( + runBestPractices(setValuesStep('{Count(AccountList)}')) + .violations, + 'SETVALUES-FUNC-STR-001' + ); + assert.ok(v, 'Expected SETVALUES-FUNC-STR-001 to fire for {Count(...)}'); + assert.equal(v?.severity, 'major'); + }); + + it('passes when the value is a proper funcCall element', () => { + const v = find( + runBestPractices( + setValuesStep( + '' + ) + ).violations, + 'SETVALUES-FUNC-STR-001' + ); + assert.ok(!v, `A funcCall value should pass, got: ${v?.message}`); + }); + }); + + // ── setValuesZeroIndexString (SETVALUES-ZERO-IDX-001) ────────────────────── + describe('setValuesZeroIndexString — SETVALUES-ZERO-IDX-001', () => { + function setValuesStep(text: string): string { + return buildTc(` + + + + + ${text} + + + + + `); + } + + it('fires for a [0] index in a string template', () => { + const v = find(runBestPractices(setValuesStep('{AccountList[0].Name}')).violations, 'SETVALUES-ZERO-IDX-001'); + assert.ok(v, 'Expected SETVALUES-ZERO-IDX-001 to fire for [0]'); + }); + + it('passes for a 1-indexed string template', () => { + const v = find(runBestPractices(setValuesStep('{AccountList[1].Name}')).violations, 'SETVALUES-ZERO-IDX-001'); + assert.ok(!v, `A [1] index should pass, got: ${v?.message}`); + }); + }); + + // ── assertValuesStringExpr (ASSERT-STR-VAR-001) ──────────────────────────── + describe('assertValuesStringExpr — ASSERT-STR-VAR-001', () => { + function assertStep(argId: string, valueXml: string): string { + return buildTc(` + + ${valueXml} + + `); + } + + it('fires when expectedValue is a {Var} string literal', () => { + const v = find( + runBestPractices(assertStep('expectedValue', '{RowCount}')) + .violations, + 'ASSERT-STR-VAR-001' + ); + assert.ok(v, 'Expected ASSERT-STR-VAR-001 to fire for a {RowCount} literal'); + assert.equal(v?.severity, 'major'); + }); + + it('passes when the value is a proper class="variable" reference', () => { + const v = find( + runBestPractices(assertStep('expectedValue', '')) + .violations, + 'ASSERT-STR-VAR-001' + ); + assert.ok(!v, `A class="variable" reference should pass, got: ${v?.message}`); + }); + + it('does not fire for a plain literal that is not a brace expression', () => { + const v = find( + runBestPractices(assertStep('expectedValue', 'Acme')) + .violations, + 'ASSERT-STR-VAR-001' + ); + assert.ok(!v, 'a literal value that is not a {…} expression is fine'); + }); + }); + + // ── uiLocatorButtonCasing (UI-LOCATOR-BUTTON-CASING-001) ─────────────────── + describe('uiLocatorButtonCasing — UI-LOCATOR-BUTTON-CASING-001', () => { + function uiStep(uri: string): string { + return buildTc(` + + + + `); + } + + it('fires for the wrong-cased Cancel button (name=Cancel)', () => { + const v = find(runBestPractices(uiStep('ui:locator?name=Cancel')).violations, 'UI-LOCATOR-BUTTON-CASING-001'); + assert.ok(v, 'Expected UI-LOCATOR-BUTTON-CASING-001 to fire for name=Cancel'); + assert.equal(v?.severity, 'major'); + }); + + it('fires for name=Continue (record-type Continue needs name=save&path=selectRecordType)', () => { + const v = find(runBestPractices(uiStep('ui:locator?name=Continue')).violations, 'UI-LOCATOR-BUTTON-CASING-001'); + assert.ok(v, 'Expected UI-LOCATOR-BUTTON-CASING-001 to fire for name=Continue'); + }); + + it('passes for the correct lowercase name=cancel', () => { + const v = find(runBestPractices(uiStep('ui:locator?name=cancel')).violations, 'UI-LOCATOR-BUTTON-CASING-001'); + assert.ok(!v, `name=cancel is correct and should pass, got: ${v?.message}`); + }); + + it('does not match a prefix (name=Cancelled must not fire)', () => { + const v = find(runBestPractices(uiStep('ui:locator?name=Cancelled')).violations, 'UI-LOCATOR-BUTTON-CASING-001'); + assert.ok(!v, 'the name= boundary must not match a longer name'); + }); + }); + + // ── uiLocatorRecordTypeField (UI-LOCATOR-RECORDTYPE-001) ─────────────────── + describe('uiLocatorRecordTypeField — UI-LOCATOR-RECORDTYPE-001', () => { + function uiStep(uri: string): string { + return buildTc(` + + + + `); + } + + it('fires for name=recordTypeId', () => { + const v = find( + runBestPractices(uiStep('ui:locator?name=recordTypeId&field=RecordTypeId')).violations, + 'UI-LOCATOR-RECORDTYPE-001' + ); + assert.ok(v, 'Expected UI-LOCATOR-RECORDTYPE-001 to fire for name=recordTypeId'); + assert.equal(v?.severity, 'major'); + }); + + it('fires for the lowercase name=recordType', () => { + const v = find(runBestPractices(uiStep('ui:locator?name=recordType')).violations, 'UI-LOCATOR-RECORDTYPE-001'); + assert.ok(v, 'Expected UI-LOCATOR-RECORDTYPE-001 to fire for name=recordType'); + }); + + it('passes for the correct name=RecordType', () => { + const v = find( + runBestPractices(uiStep('ui:locator?name=RecordType&field=RecordTypeId')).violations, + 'UI-LOCATOR-RECORDTYPE-001' + ); + assert.ok(!v, `name=RecordType is correct and should pass, got: ${v?.message}`); + }); + }); +}); + +describe('structural / load-affecting validators (Tier 5)', () => { + const GUID_T5_TC = '550e8400-e29b-41d4-a716-4466554406d0'; + const GUID_T5_S1 = '550e8400-e29b-41d4-a716-4466554406d1'; + const SET_VALUES = 'com.provar.plugins.bundled.apis.control.SetValues'; + const UI_ASSERT = 'com.provar.plugins.forcedotcom.core.ui.UiAssert'; + const UI_DO_ACTION = 'com.provar.plugins.forcedotcom.core.ui.UiDoAction'; + const GENERIC = 'com.provar.plugins.forcedotcom.core.testapis.ApexExecute'; + + function buildTc(stepsXml: string): string { + return ` + + +${stepsXml} + +`; + } + + function find(violations: BPViolation[], ruleId: string): BPViolation | undefined { + return violations.find((v) => v.rule_id === ruleId); + } + + // ── validFuncCallId (FUNCCALL-VALID-001) ─────────────────────────────────── + describe('validFuncCallId — FUNCCALL-VALID-001', () => { + function funcStep(valueXml: string): string { + return buildTc(` + + ${valueXml} + + `); + } + + it('fires for a hallucinated funcCall id', () => { + const v = find( + runBestPractices(funcStep('')).violations, + 'FUNCCALL-VALID-001' + ); + assert.ok(v, 'Expected FUNCCALL-VALID-001 to fire for id="Concatenate"'); + assert.equal(v?.severity, 'major'); + }); + + it('passes for a real Provar built-in (Count)', () => { + const v = find( + runBestPractices(funcStep('')).violations, + 'FUNCCALL-VALID-001' + ); + assert.ok(!v, `id="Count" is valid and should pass, got: ${v?.message}`); + }); + }); + + // ── rootAttributes (RENDER-ROOT-001) ─────────────────────────────────────── + describe('rootAttributes — RENDER-ROOT-001', () => { + it('fires when the root testCase carries an unknown attribute', () => { + const xml = ` +`; + const v = find(runBestPractices(xml).violations, 'RENDER-ROOT-001'); + assert.ok(v, 'Expected RENDER-ROOT-001 to fire for an unknown root attribute'); + assert.ok(v?.message.includes('bogusAttr'), `Message should name the bad attr: ${v?.message}`); + }); + + it('passes when the root has only known attributes', () => { + const xml = ` +`; + const v = find(runBestPractices(xml).violations, 'RENDER-ROOT-001'); + assert.ok(!v, `Only known root attrs should pass, got: ${v?.message}`); + }); + }); + + // ── setValuesStructure (SETVALUES-STRUCTURE-001) ─────────────────────────── + describe('setValuesStructure — SETVALUES-STRUCTURE-001', () => { + it('fires for a SetValues step with no container', () => { + const xml = buildTc(` + + `); + const v = find(runBestPractices(xml).violations, 'SETVALUES-STRUCTURE-001'); + assert.ok(v, 'Expected SETVALUES-STRUCTURE-001 to fire when is missing'); + assert.equal(v?.severity, 'critical'); + }); + + it('passes when the SetValues step contains a container', () => { + const xml = buildTc(` + 1 + `); + const v = find(runBestPractices(xml).violations, 'SETVALUES-STRUCTURE-001'); + assert.ok(!v, `A SetValues with should pass, got: ${v?.message}`); + }); + + it('does NOT fire for a data-driven SetValues (values from an external source)', () => { + // Excel/CSV-driven SetValues declares and carries an + // empty with no inline — corpus-confirmed valid. + const xml = buildTc(` + + + `); + const v = find(runBestPractices(xml).violations, 'SETVALUES-STRUCTURE-001'); + assert.ok(!v, `A data-driven SetValues must pass, got: ${v?.message}`); + }); + }); + + // ── namedValueName (SETVALUES-NAME-001) ──────────────────────────────────── + describe('namedValueName — SETVALUES-NAME-001', () => { + it('fires when a namedValue lacks a name attribute', () => { + const xml = buildTc(` + 1 + `); + const v = find(runBestPractices(xml).violations, 'SETVALUES-NAME-001'); + assert.ok(v, 'Expected SETVALUES-NAME-001 to fire for a nameless namedValue'); + assert.equal(v?.severity, 'critical'); + }); + + it('passes when every namedValue has a name attribute', () => { + const xml = buildTc(` + 1 + `); + const v = find(runBestPractices(xml).violations, 'SETVALUES-NAME-001'); + assert.ok(!v, `A named namedValue should pass, got: ${v?.message}`); + }); + }); + + // ── namedValueValue (SETVALUES-VALUE-001) ────────────────────────────────── + describe('namedValueValue — SETVALUES-VALUE-001', () => { + it('fires when a non-structural namedValue has no child ', () => { + const xml = buildTc(` + + `); + const v = find(runBestPractices(xml).violations, 'SETVALUES-VALUE-001'); + assert.ok(v, 'Expected SETVALUES-VALUE-001 to fire for a non-slot namedValue with no child'); + assert.equal(v?.severity, 'critical'); + }); + + it('does NOT fire for an empty value slot (blank field) — corpus-confirmed valid', () => { + const xml = buildTc(` + Field__c + `); + const v = find(runBestPractices(xml).violations, 'SETVALUES-VALUE-001'); + assert.ok(!v, `An empty name="value" slot blanks the field and must pass, got: ${v?.message}`); + }); + + it('does NOT fire for a wholly-blank row (empty valuePath + value) — an unused row', () => { + const xml = buildTc(` + + `); + const v = find(runBestPractices(xml).violations, 'SETVALUES-VALUE-001'); + assert.ok(!v, `A blank/unused SetValues row must pass, got: ${v?.message}`); + }); + + it('passes when every namedValue has a child ', () => { + const xml = buildTc(` + 1 + `); + const v = find(runBestPractices(xml).violations, 'SETVALUES-VALUE-001'); + assert.ok(!v, `A namedValue with a child should pass, got: ${v?.message}`); + }); + }); + + // ── uiAssertHallucinatedGeneratedParameters (UI-ASSERT-STRUCT-002) ───────── + describe('uiAssertHallucinatedGeneratedParameters — UI-ASSERT-STRUCT-002', () => { + it('fires for a UiAssert step containing ', () => { + const xml = buildTc(` + + `); + const v = find(runBestPractices(xml).violations, 'UI-ASSERT-STRUCT-002'); + assert.ok(v, 'Expected UI-ASSERT-STRUCT-002 to fire for hallucinated generatedParameters'); + assert.equal(v?.severity, 'critical'); + }); + + it('passes for a UiAssert step with no generatedParameters', () => { + const xml = buildTc(` `); + const v = find(runBestPractices(xml).violations, 'UI-ASSERT-STRUCT-002'); + assert.ok(!v, `A UiAssert with no generatedParameters should pass, got: ${v?.message}`); + }); + }); + + // ── uiAssertMissingArguments (UI-ASSERT-STRUCT-001) ──────────────────────── + describe('uiAssertMissingArguments — UI-ASSERT-STRUCT-001', () => { + it('fires for a UiAssert step missing required arguments', () => { + const xml = buildTc(` + + `); + const v = find(runBestPractices(xml).violations, 'UI-ASSERT-STRUCT-001'); + assert.ok(v, 'Expected UI-ASSERT-STRUCT-001 to fire when required args are missing'); + assert.equal(v?.severity, 'critical'); + assert.ok(v?.message.includes('fieldAssertions'), `Message should list a missing arg: ${v?.message}`); + }); + + it('passes when all required UiAssert arguments are present', () => { + const args = [ + 'fieldAssertions', + 'columnAssertions', + 'pageAssertions', + 'resultScope', + 'captureAfter', + 'beforeWait', + 'autoRetry', + ] + .map((a) => ``) + .join(''); + const xml = buildTc(` + ${args} + `); + const v = find(runBestPractices(xml).violations, 'UI-ASSERT-STRUCT-001'); + assert.ok(!v, `All required args present should pass, got: ${v?.message}`); + }); + }); + + // ── bindingParameterOrder (UI-BINDING-ORDER-001) ─────────────────────────── + describe('bindingParameterOrder — UI-BINDING-ORDER-001', () => { + function locStep(uri: string): string { + return buildTc(` + + `); + } + + it('fires for an action-before-object binding order', () => { + const v = find( + runBestPractices(locStep('ui:locator?binding=object%3Faction%3DNEW%26object%3DAccount')).violations, + 'UI-BINDING-ORDER-001' + ); + assert.ok(v, 'Expected UI-BINDING-ORDER-001 to fire for action-first order'); + assert.equal( + v?.severity, + 'major', + 'wrong binding order errors at runtime (the test loads) — major, not critical' + ); + }); + + it('passes for the object-first binding order', () => { + const v = find( + runBestPractices(locStep('ui:locator?binding=object%3Fobject%3DAccount%26action%3DNEW')).violations, + 'UI-BINDING-ORDER-001' + ); + assert.ok(!v, `object-first order should pass, got: ${v?.message}`); + }); + + it('does not fire for a locator with no binding= parameter', () => { + const v = find(runBestPractices(locStep('ui:locator?name=save')).violations, 'UI-BINDING-ORDER-001'); + assert.ok(!v, 'a locator without a binding is out of scope for this rule'); + }); + }); + + // ── uiConnectionNameLiteral (UI-CONN-LITERAL-001) ────────────────────────── + describe('uiConnectionNameLiteral — UI-CONN-LITERAL-001', () => { + function uiConnStep(valueXml: string): string { + return buildTc(` + ${valueXml} + `); + } + + it('fires when uiConnectionName is a variable reference', () => { + const v = find( + runBestPractices(uiConnStep('')).violations, + 'UI-CONN-LITERAL-001' + ); + assert.ok(v, 'Expected UI-CONN-LITERAL-001 to fire for a variable uiConnectionName'); + assert.equal(v?.severity, 'critical'); + }); + + it('passes when uiConnectionName is a literal string', () => { + const v = find( + runBestPractices(uiConnStep('MyConnection')).violations, + 'UI-CONN-LITERAL-001' + ); + assert.ok(!v, `A literal uiConnectionName should pass, got: ${v?.message}`); + }); + }); +}); diff --git a/test/unit/mcp/qualityThreshold.test.ts b/test/unit/mcp/qualityThreshold.test.ts new file mode 100644 index 00000000..34008e66 --- /dev/null +++ b/test/unit/mcp/qualityThreshold.test.ts @@ -0,0 +1,55 @@ +/* + * Copyright (c) 2024 Provar Limited. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.md file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ + +import { strict as assert } from 'node:assert'; +import { describe, it, afterEach } from 'mocha'; +import { resolveQualityThreshold, DEFAULT_QUALITY_THRESHOLD } from '../../../src/mcp/utils/qualityThreshold.js'; + +describe('resolveQualityThreshold (PDX-509)', () => { + const saved = process.env.PROVAR_MCP_QUALITY_THRESHOLD; + + afterEach(() => { + if (saved !== undefined) process.env.PROVAR_MCP_QUALITY_THRESHOLD = saved; + else delete process.env.PROVAR_MCP_QUALITY_THRESHOLD; + }); + + it('defaults to 90', () => { + delete process.env.PROVAR_MCP_QUALITY_THRESHOLD; + assert.equal(DEFAULT_QUALITY_THRESHOLD, 90); + assert.equal(resolveQualityThreshold(), 90); + }); + + it('honours a valid per-call arg over everything', () => { + process.env.PROVAR_MCP_QUALITY_THRESHOLD = '50'; + assert.equal(resolveQualityThreshold(80), 80); + }); + + it('falls back to the env var when no arg is given', () => { + process.env.PROVAR_MCP_QUALITY_THRESHOLD = '70'; + assert.equal(resolveQualityThreshold(), 70); + }); + + it('ignores an out-of-range arg and falls through to env, then default', () => { + process.env.PROVAR_MCP_QUALITY_THRESHOLD = '65'; + assert.equal(resolveQualityThreshold(150), 65, 'arg 150 is out of range → env'); + delete process.env.PROVAR_MCP_QUALITY_THRESHOLD; + assert.equal(resolveQualityThreshold(-5), 90, 'arg -5 is out of range, no env → default'); + }); + + it('ignores an unparseable or out-of-range env var', () => { + process.env.PROVAR_MCP_QUALITY_THRESHOLD = 'not-a-number'; + assert.equal(resolveQualityThreshold(), 90); + process.env.PROVAR_MCP_QUALITY_THRESHOLD = '999'; + assert.equal(resolveQualityThreshold(), 90); + }); + + it('accepts the boundary values 0 and 100', () => { + delete process.env.PROVAR_MCP_QUALITY_THRESHOLD; + assert.equal(resolveQualityThreshold(0), 0); + assert.equal(resolveQualityThreshold(100), 100); + }); +}); diff --git a/test/unit/mcp/server.test.ts b/test/unit/mcp/server.test.ts index f981c18c..6296d6bf 100644 --- a/test/unit/mcp/server.test.ts +++ b/test/unit/mcp/server.test.ts @@ -10,7 +10,7 @@ import path from 'node:path'; import fs from 'node:fs'; import os from 'node:os'; import { describe, it, afterEach } from 'mocha'; -import { resolveDocsDir, readCatalogSource } from '../../../src/mcp/server.js'; +import { resolveDocsDir, readCatalogSource, resolveRulesDir, readTestStepSchema } from '../../../src/mcp/server.js'; describe('resolveDocsDir', () => { const tmpDirs: string[] = []; @@ -127,3 +127,80 @@ describe('readCatalogSource', () => { assert.equal(result['schemasUpdated'], null); }); }); + +describe('resolveRulesDir', () => { + const tmpDirs: string[] = []; + + afterEach(() => { + for (const d of tmpDirs) { + try { + fs.rmSync(d, { recursive: true, force: true }); + } catch { + // ignore cleanup errors + } + } + tmpDirs.length = 0; + }); + + function makeTmpDir(): string { + const d = fs.mkdtempSync(path.join(os.tmpdir(), 'provar-rules-test-')); + tmpDirs.push(d); + return d; + } + + it('returns sibling rules/ when it exists (compiled lib/mcp/ and dev src/mcp/ modes)', () => { + const base = makeTmpDir(); + const sibling = path.join(base, 'rules'); + fs.mkdirSync(sibling); + assert.equal(resolveRulesDir(base), sibling); + }); + + it('falls back one level to ../rules when the sibling is absent', () => { + const base = makeTmpDir(); + assert.equal(resolveRulesDir(base), path.join(base, '..', 'rules')); + }); +}); + +describe('readTestStepSchema', () => { + const tmpDirs: string[] = []; + + afterEach(() => { + for (const d of tmpDirs) { + try { + fs.rmSync(d, { recursive: true, force: true }); + } catch { + // ignore + } + } + tmpDirs.length = 0; + }); + + function makeTmpDir(): string { + const d = fs.mkdtempSync(path.join(os.tmpdir(), 'provar-rules-test-')); + tmpDirs.push(d); + return d; + } + + it('returns the schema file verbatim when present (parseable JSON with expected keys)', () => { + const rulesDir = makeTmpDir(); + const schema = { title: 'Provar Test Case XML Structure Schema', version: '2.0.0', testCase: {}, apiCalls: {} }; + fs.writeFileSync(path.join(rulesDir, 'provar_test_step_schema.json'), JSON.stringify(schema)); + const parsed = JSON.parse(readTestStepSchema(rulesDir)) as typeof schema; + assert.equal(parsed.version, '2.0.0'); + assert.ok('testCase' in parsed && 'apiCalls' in parsed, 'schema should expose testCase and apiCalls'); + }); + + it('returns a schema_not_found fallback object when the file is absent', () => { + const rulesDir = makeTmpDir(); + const result = JSON.parse(readTestStepSchema(rulesDir)) as Record; + assert.equal(result['error'], 'schema_not_found'); + assert.ok(typeof result['message'] === 'string' && result['message'].length > 0); + }); + + it('returns the schema_not_found fallback when the file is present but corrupted (invalid JSON)', () => { + const rulesDir = makeTmpDir(); + fs.writeFileSync(path.join(rulesDir, 'provar_test_step_schema.json'), '{ "truncated": '); + const result = JSON.parse(readTestStepSchema(rulesDir)) as Record; + assert.equal(result['error'], 'schema_not_found', 'a corrupt file must not be served verbatim as application/json'); + }); +}); diff --git a/test/unit/mcp/testCaseGenerate.test.ts b/test/unit/mcp/testCaseGenerate.test.ts index 9532a43a..2616b996 100644 --- a/test/unit/mcp/testCaseGenerate.test.ts +++ b/test/unit/mcp/testCaseGenerate.test.ts @@ -407,17 +407,20 @@ describe('provar_testcase_generate', () => { assert.ok(!xml.includes('Test &'), 'escaped name must not appear in XML'); }); - it('escapes XML special characters in step api_id and name', () => { + it('escapes XML special characters in the step name', () => { + // A real apiId never contains XML metacharacters (and an unknown apiId is now + // a load-blocking error via the bridge), so escaping is exercised through the + // step name — which legitimately can contain &, <, > and ". const result = server.call('provar_testcase_generate', { test_case_name: 'Escape Step Test', - steps: [{ api_id: 'Api', name: 'Step & "Name"', attributes: {} }], + steps: [{ api_id: 'UiConnect', name: 'Step & ', attributes: {} }], dry_run: true, overwrite: false, }); const xml = parseText(result)['xml_content'] as string; - assert.ok(xml.includes('<') && xml.includes('>'), 'Expected < > escaped in apiId'); assert.ok(xml.includes('&'), 'Expected & escaped in step name'); + assert.ok(xml.includes('<') && xml.includes('>'), 'Expected < > escaped in step name'); }); }); @@ -501,6 +504,76 @@ describe('provar_testcase_generate', () => { }); }); + // ── testCase id allocation (highest-in-use + 1 within an existing project) ────── + describe('testCase id allocation', () => { + const SMOKE_STEPS = [{ api_id: 'UiConnect', name: 'Connect', attributes: {} }]; + + /** Pull the root testCase id from emitted xml_content. */ + function emittedId(result: unknown): string | undefined { + const xml = parseText(result)['xml_content'] as string; + return xml.match(/]*?\bid="([^"]+)"/)?.[1]; + } + + function seedProject(): void { + fs.writeFileSync(path.join(tmpDir, '.testproject'), '', 'utf-8'); + const testsDir = path.join(tmpDir, 'tests'); + fs.mkdirSync(testsDir, { recursive: true }); + const guid = '22222222-2222-4222-8222-222222222222'; + fs.writeFileSync( + path.join(testsDir, 'Existing.testcase'), + ``, + 'utf-8' + ); + } + + it('defaults to id="1" when output is not inside a project', () => { + const outPath = path.join(tmpDir, 'Loose.testcase'); + const result = server.call('provar_testcase_generate', { + test_case_name: 'Loose', + steps: SMOKE_STEPS, + output_path: outPath, + dry_run: false, + overwrite: false, + }); + + assert.equal(isError(result), false); + assert.equal(emittedId(result), '1'); + assert.equal(parseText(result)['test_case_id'], 1); + }); + + it('allocates highest-in-use + 1 when writing into an existing project', () => { + seedProject(); + const outPath = path.join(tmpDir, 'tests', 'New.testcase'); + const result = server.call('provar_testcase_generate', { + test_case_name: 'New', + steps: SMOKE_STEPS, + output_path: outPath, + dry_run: false, + overwrite: false, + }); + + assert.equal(isError(result), false); + assert.equal(emittedId(result), '6', 'project max id is 5 → next is 6'); + assert.equal(parseText(result)['test_case_id'], 6); + }); + + it('does not scan the project for a dry_run preview (keeps default id)', () => { + seedProject(); + const outPath = path.join(tmpDir, 'tests', 'Preview.testcase'); + const result = server.call('provar_testcase_generate', { + test_case_name: 'Preview', + steps: SMOKE_STEPS, + output_path: outPath, + dry_run: true, + overwrite: false, + }); + + assert.equal(isError(result), false); + assert.equal(emittedId(result), '1'); + assert.equal(parseText(result)['test_case_id'], 1); + }); + }); + // ── PDX-483 runtime guard: reject empty steps[] on non-dry-run with output_path ── // The PDX-479 regression class arose from agents calling generate with steps:[] // intending to append later via step_edit. The passive contract (PDX-482) lives in @@ -794,9 +867,10 @@ describe('provar_testcase_generate', () => { }); const xml = parseText(result)['xml_content'] as string; + // Provar stores dates as epoch milliseconds (UTC midnight), not an ISO string. assert.ok( - xml.includes('valueClass="date">2026-05-19'), - `Expected valueClass="date" for ISO date; got: ${xml}` + xml.includes('valueClass="date">1779148800000'), + `Expected valueClass="date" with epoch ms for ISO date; got: ${xml}` ); }); @@ -815,12 +889,35 @@ describe('provar_testcase_generate', () => { }); const xml = parseText(result)['xml_content'] as string; + // Provar uses camelCase valueClass="dateTime" with an epoch-ms value (UTC when no tz). assert.ok( - xml.includes('valueClass="datetime">2026-05-19T10:30:00'), - `Expected valueClass="datetime" for ISO datetime; got: ${xml}` + xml.includes('valueClass="dateTime">1779186600000'), + `Expected valueClass="dateTime" with epoch ms for ISO datetime; got: ${xml}` ); }); + it('falls back to valueClass="string" for an ISO-shaped but invalid date (no load-breaking date)', () => { + const result = server.call('provar_testcase_generate', { + test_case_name: 'BadDateField', + steps: [ + { + api_id: 'ApexCreateObject', + name: 'Create', + attributes: { CloseDate: '2026-99-99' }, + }, + ], + dry_run: true, + overwrite: false, + }); + + const xml = parseText(result)['xml_content'] as string; + assert.ok( + xml.includes('valueClass="string">2026-99-99'), + `Unparseable date must emit valueClass="string", not a non-epoch date; got: ${xml}` + ); + assert.ok(!xml.includes('valueClass="date"'), 'must not emit a load-breaking valueClass="date"'); + }); + it('emits valueClass="boolean" for "true" / "false" literals', () => { const result = server.call('provar_testcase_generate', { test_case_name: 'BoolField', @@ -1041,6 +1138,96 @@ describe('provar_testcase_generate', () => { ); }); + // PDX-506: GENERATOR test — a UiDoAction interaction attribute must round-trip + // to typed class="uiInteraction" XML (not a plain string) and validate clean. + it('emits class="uiInteraction" uri="..." for the interaction argument', () => { + const result = server.call('provar_testcase_generate', { + test_case_name: 'UI Interaction Test', + steps: [ + { + api_id: 'UiDoAction', + name: 'Click button', + attributes: { + locator: 'sf:ui:locator:button?label=Save', + interaction: 'ui:interaction?name=action', + }, + }, + ], + dry_run: true, + overwrite: false, + validate_after_edit: false, + }); + + const xml = parseText(result)['xml_content'] as string; + assert.ok(xml.includes('class="uiInteraction"'), 'Expected class="uiInteraction"'); + assert.ok(xml.includes('uri="ui:interaction?name=action"'), 'Expected uri attribute with interaction value'); + assert.ok( + !xml.includes('valueClass="string">ui:interaction'), + 'Must NOT emit interaction URI as a plain string value' + ); + + // Round-trip: the generated XML must pass the validator with no UI-INTERACTION-001. + const v = validateTestCase(xml); + assert.ok( + !v.issues.some((i) => i.rule_id === 'UI-INTERACTION-001'), + 'Generated UiDoAction interaction must clear UI-INTERACTION-001' + ); + }); + + // PDX-507: GENERATOR test — a UiAssert with flat field-assertion attributes + // must round-trip to the nested fieldAssertions/uiFieldAssertion structure + // (bare , NO top-level fieldLocator argument, NO uiLocator) + // and validate clean. Shape confirmed against the AllPOCProjects corpus. + it('emits the nested fieldAssertions/uiFieldAssertion structure for UiAssert', () => { + const result = server.call('provar_testcase_generate', { + test_case_name: 'UI Assert Structure Test', + steps: [ + { + api_id: 'UiAssert', + name: 'Assert Priority error', + attributes: { + fieldLocator: 'ui:locator?name=Priority', + attributeName: 'error', + comparisonType: 'Contains', + expectedValue: 'Priority must be set', + }, + }, + ], + dry_run: true, + overwrite: false, + validate_after_edit: false, + }); + + const xml = parseText(result)['xml_content'] as string; + assert.ok(xml.includes(''), 'Expected fieldAssertions argument'); + assert.ok(xml.includes(''), 'Expected nested uiFieldAssertion'); + assert.ok( + xml.includes(''), + 'Expected bare fieldLocator element with uri attribute' + ); + assert.ok( + xml.includes(''), + 'Expected uiAttributeAssertion with attributeName + comparisonType' + ); + assert.ok( + xml.includes('') && xml.includes(''), + 'Expected empty columnAssertions/pageAssertions containers' + ); + // Must NOT emit the flat shape or wrap the locator in uiLocator. + assert.ok(!xml.includes(''), 'Must NOT emit a flat fieldLocator argument'); + assert.ok( + !xml.includes('class="uiLocator" uri="ui:locator?name=Priority"'), + 'Must NOT wrap the field locator in class="uiLocator"' + ); + + // Round-trip: generated XML must clear UI-ASSERT-STRUCTURE-001. + const v = validateTestCase(xml); + assert.ok( + !v.issues.some((i) => i.rule_id === 'UI-ASSERT-STRUCTURE-001'), + 'Generated UiAssert must clear UI-ASSERT-STRUCTURE-001' + ); + }); + it('uiTarget also applies inside UiWithScreen wrapper when target_uri is non-SF', () => { const result = server.call('provar_testcase_generate', { test_case_name: 'Non-SF With Target', diff --git a/test/unit/mcp/testCaseId.test.ts b/test/unit/mcp/testCaseId.test.ts new file mode 100644 index 00000000..ab1c5a99 --- /dev/null +++ b/test/unit/mcp/testCaseId.test.ts @@ -0,0 +1,192 @@ +/* + * Copyright (c) 2024 Provar Limited. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.md file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ + +import { strict as assert } from 'node:assert'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { describe, it, beforeEach, afterEach } from 'mocha'; +import { allocateTestCaseId, DEFAULT_TESTCASE_ID } from '../../../src/mcp/utils/testCaseId.js'; + +// ── Fixture helpers ──────────────────────────────────────────────────────────── + +let tmpDir: string; + +/** Write a minimal .testcase file carrying the given root id (string so we can test UUIDs). */ +function writeCase(relPath: string, id: string): string { + const full = path.join(tmpDir, relPath); + fs.mkdirSync(path.dirname(full), { recursive: true }); + const guid = '11111111-1111-4111-8111-111111111111'; + fs.writeFileSync( + full, + `\n\n \n \n\n`, + 'utf-8' + ); + return full; +} + +function markProject(relDir = '.'): void { + const dir = path.join(tmpDir, relDir); + fs.mkdirSync(dir, { recursive: true }); + fs.writeFileSync(path.join(dir, '.testproject'), '', 'utf-8'); +} + +beforeEach(() => { + tmpDir = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'tcid-test-'))); +}); + +afterEach(() => { + fs.rmSync(tmpDir, { recursive: true, force: true }); +}); + +/** Some Windows hosts / CI restrict symlink creation to privileged users. */ +const symlinkSupported = ((): boolean => { + try { + const probe = fs.mkdtempSync(path.join(os.tmpdir(), 'tcid-symcap-')); + fs.writeFileSync(path.join(probe, 'target'), 'x', 'utf-8'); + fs.symlinkSync(path.join(probe, 'target'), path.join(probe, 'link')); + fs.rmSync(probe, { recursive: true, force: true }); + return true; + } catch { + return false; + } +})(); + +// ── Tests ──────────────────────────────────────────────────────────────────── + +describe('allocateTestCaseId', () => { + it('defaults to DEFAULT_TESTCASE_ID when there is no surrounding project', () => { + const out = path.join(tmpDir, 'Loose.testcase'); + const alloc = allocateTestCaseId(out, [tmpDir]); + assert.equal(alloc.id, DEFAULT_TESTCASE_ID); + assert.equal(alloc.basis, 'default'); + }); + + it('allocates highest project id + 1 when writing into an existing project', () => { + markProject(); + writeCase('tests/A.testcase', '3'); + writeCase('tests/sub/B.testcase', '7'); + writeCase('tests/sub/C.testcase', '5'); + + const out = path.join(tmpDir, 'tests', 'New.testcase'); + const alloc = allocateTestCaseId(out, [tmpDir]); + + assert.equal(alloc.id, 8, 'max id is 7 → next is 8'); + assert.equal(alloc.basis, 'project-max-plus-1'); + assert.equal(alloc.highestExistingId, 7); + assert.equal(alloc.projectRoot, tmpDir); + }); + + it('finds the project root by walking up from a nested output directory', () => { + markProject(); + writeCase('tests/area/Existing.testcase', '12'); + + const out = path.join(tmpDir, 'tests', 'area', 'deep', 'New.testcase'); + const alloc = allocateTestCaseId(out, [tmpDir]); + + assert.equal(alloc.id, 13); + assert.equal(alloc.basis, 'project-max-plus-1'); + }); + + it('preserves an existing numeric id on overwrite (stable regeneration)', () => { + markProject(); + writeCase('tests/Other.testcase', '50'); + const out = writeCase('tests/Target.testcase', '42'); + + const alloc = allocateTestCaseId(out, [tmpDir]); + + assert.equal(alloc.id, 42, 'should keep the file’s own id, not jump to 51'); + assert.equal(alloc.basis, 'preserved-existing'); + }); + + it('ignores non-numeric (UUID) ids when computing the max', () => { + markProject(); + writeCase('tests/Uuid.testcase', 'ced6c489-5a6d-4a40-a92f-71986c895b73'); + writeCase('tests/Num.testcase', '2'); + + const out = path.join(tmpDir, 'tests', 'New.testcase'); + const alloc = allocateTestCaseId(out, [tmpDir]); + + assert.equal(alloc.id, 3, 'only the numeric id 2 counts → next is 3'); + }); + + it('defaults when the project has a marker but no numeric ids yet', () => { + markProject(); + const out = path.join(tmpDir, 'tests', 'First.testcase'); + const alloc = allocateTestCaseId(out, [tmpDir]); + + assert.equal(alloc.id, DEFAULT_TESTCASE_ID); + assert.equal(alloc.basis, 'default'); + }); + + it('scans the project root itself when there is no tests/ folder', () => { + markProject(); + writeCase('Flat.testcase', '9'); + + const out = path.join(tmpDir, 'Another.testcase'); + const alloc = allocateTestCaseId(out, [tmpDir]); + + assert.equal(alloc.id, 10); + assert.equal(alloc.basis, 'project-max-plus-1'); + }); + + it('does not cross above the allowed roots to find a project marker', () => { + // .testproject sits at tmpDir, but the allowed root is tmpDir/tests — the walk + // must stop at the boundary and fall back to the default rather than scanning above it. + markProject(); + writeCase('tests/A.testcase', '4'); + const testsDir = path.join(tmpDir, 'tests'); + + const out = path.join(testsDir, 'New.testcase'); + const alloc = allocateTestCaseId(out, [testsDir]); + + assert.equal(alloc.id, DEFAULT_TESTCASE_ID); + assert.equal(alloc.basis, 'default'); + }); + + it('ignores a numeric id too large to be a safe integer', () => { + markProject(); + writeCase('tests/Normal.testcase', '4'); + writeCase('tests/Huge.testcase', '99999999999999999999999'); + + const out = path.join(tmpDir, 'tests', 'New.testcase'); + const alloc = allocateTestCaseId(out, [tmpDir]); + + // The 23-digit id overflows JS integer precision; it must be ignored so the + // generator never emits id="1e+22". Only the safe id 4 counts → next is 5. + assert.equal(alloc.id, 5); + assert.equal(alloc.basis, 'project-max-plus-1'); + assert.equal(alloc.highestExistingId, 4); + }); + + (symlinkSupported ? it : it.skip)('does not read a symlinked .testcase that points outside the allowed roots', () => { + markProject(); + writeCase('tests/Real.testcase', '4'); + + // A "secret" project OUTSIDE the allowed root, carrying a deliberately huge id. + const secretDir = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'tcid-secret-'))); + try { + const secretCase = path.join(secretDir, 'Secret.testcase'); + fs.writeFileSync( + secretCase, + '\n\n', + 'utf-8' + ); + fs.symlinkSync(secretCase, path.join(tmpDir, 'tests', 'Link.testcase')); + + const out = path.join(tmpDir, 'tests', 'New.testcase'); + const alloc = allocateTestCaseId(out, [tmpDir]); + + // The symlink must be skipped: only the in-root id 4 counts → next is 5, + // NOT 88889 (which would mean the out-of-root file was read). + assert.equal(alloc.id, 5); + assert.equal(alloc.highestExistingId, 4); + } finally { + fs.rmSync(secretDir, { recursive: true, force: true }); + } + }); +}); diff --git a/test/unit/mcp/testCaseValidate.test.ts b/test/unit/mcp/testCaseValidate.test.ts index 9c19916c..c3ca6d6f 100644 --- a/test/unit/mcp/testCaseValidate.test.ts +++ b/test/unit/mcp/testCaseValidate.test.ts @@ -11,6 +11,7 @@ import { registerTestCaseValidate, validateTestCaseXml, } from '../../../src/mcp/tools/testCaseValidate.js'; +import { runBestPractices } from '../../../src/mcp/tools/bestPracticesEngine.js'; import type { ServerConfig } from '../../../src/mcp/server.js'; import { ASSERT_VALUES_COMPARISON_TYPES, @@ -31,8 +32,8 @@ const VALID_TC = ` - - + + `; @@ -76,31 +77,48 @@ describe('validateTestCase', () => { }); describe('testCase attribute rules', () => { - it('TC_010: flags missing id', () => { + it('TC_010: does NOT fire for a missing id — id is optional, guid is the identifier', () => { const r = validateTestCase( `` ); assert.ok( - r.issues.some((i) => i.rule_id === 'TC_010'), - 'Expected TC_010' + !r.issues.some((i) => i.rule_id === 'TC_010'), + 'A missing id must not be flagged (real Provar test cases routinely omit it)' ); }); - it('TC_010: flags non-"1" id (e.g. UUID used as id)', () => { + it('TC_010: flags a non-numeric id (e.g. UUID used as id)', () => { const r = validateTestCase( `` ); assert.ok( r.issues.some((i) => i.rule_id === 'TC_010'), - 'Expected TC_010 for UUID used as id' + 'Expected TC_010 for a non-numeric id' ); }); - it('TC_010: does not fire when id="1" (the correct literal)', () => { + it('TC_010: does not fire for id="0" — non-negative integers are valid (corpus uses 0)', () => { + const r = validateTestCase( + `` + ); + assert.ok(!r.issues.some((i) => i.rule_id === 'TC_010'), 'id="0" must be accepted'); + }); + + it('TC_010: does not fire for a valid integer id (id="1")', () => { const r = validateTestCase(VALID_TC); assert.ok(!r.issues.some((i) => i.rule_id === 'TC_010'), 'TC_010 must not fire when id="1"'); }); + it('TC_010: does not fire for a sequential project id (e.g. id="42") — ids are not literally "1"', () => { + const r = validateTestCase( + `` + ); + assert.ok( + !r.issues.some((i) => i.rule_id === 'TC_010'), + 'TC_010 must accept any non-negative-integer id (test case ids are project-unique sequential numbers)' + ); + }); + it('TC_011: flags missing guid', () => { const r = validateTestCase( '' @@ -710,6 +728,235 @@ describe('validateTestCase', () => { }); }); + // PDX-506: UiDoAction `interaction` must serialise as class="uiInteraction". + // The Cox Demo "Create Test Case (Generated)" file ran green from the CLI but + // rendered the IDE Action widget blank because `interaction` was a plain string. + describe('UI-INTERACTION-001', () => { + // REPRODUCE: the exact broken Cox Demo shape — a plain-string `interaction`. + // This assertion FAILS against pre-fix validator code (no rule existed) and + // PASSES once UI-INTERACTION-001 is added. + it('errors when UiDoAction interaction argument uses class="value" (plain string)', () => { + const r = validateTestCase( + ` + + + + + + ui:interaction?name=click + + + + +` + ); + assert.ok( + r.issues.some((i) => i.rule_id === 'UI-INTERACTION-001'), + 'Expected UI-INTERACTION-001' + ); + const issue = r.issues.find((i) => i.rule_id === 'UI-INTERACTION-001')!; + assert.equal(issue.severity, 'ERROR'); + assert.ok(issue.message.includes('uiInteraction'), `Message should mention uiInteraction: ${issue.message}`); + }); + + // CLEARED: the corrected typed-uiInteraction form must pass with no false positive. + it('does not fire when UiDoAction interaction uses class="uiInteraction"', () => { + const r = validateTestCase( + ` + + + + + + + + + + +` + ); + assert.ok( + !r.issues.some((i) => i.rule_id === 'UI-INTERACTION-001'), + 'UI-INTERACTION-001 should not fire for correct uiInteraction class' + ); + }); + + it('fires when UiDoAction interaction has no class attribute', () => { + const r = validateTestCase( + ` + + + + + + ui:interaction?name=click + + + + +` + ); + assert.ok( + r.issues.some((i) => i.rule_id === 'UI-INTERACTION-001'), + 'UI-INTERACTION-001 should fire when has no class attribute' + ); + const issue = r.issues.find((i) => i.rule_id === 'UI-INTERACTION-001')!; + assert.ok(issue.message.includes('(missing)'), `Message should note missing class: ${issue.message}`); + }); + + it('does not fire for a UI action step without an interaction argument', () => { + const r = validateTestCase( + ` + + + + + + + + + + +` + ); + assert.ok( + !r.issues.some((i) => i.rule_id === 'UI-INTERACTION-001'), + 'UI-INTERACTION-001 must not fire when no interaction argument is present' + ); + }); + }); + + // PDX-507: a UiAssert field assertion must be nested inside + // fieldAssertions/uiFieldAssertion (bare ). The flat shape + // (top-level fieldLocator/attributeName/comparisonType/expectedValue args) runs + // green from the CLI but renders the IDE Result Assertions tab blank. Contract + // confirmed against the real corpus (AllPOCProjects): 0/3,778 UiAssert steps + // use a flat fieldLocator argument. + describe('UI-ASSERT-STRUCTURE-001', () => { + // REPRODUCE: the exact broken Cox Demo flat shape. On pre-fix code the rule + // did not exist, so provar_testcase_validate did NOT flag this shape and this + // test's assertion therefore FAILS on main; it PASSES once the rule is added. + it('errors when UiAssert carries a flat fieldLocator argument (Cox Demo shape)', () => { + const r = validateTestCase( + ` + + + + + + Values + + + ui:locator?name=Priority + + + error + + + Contains + + + Priority must be set + + + + +` + ); + assert.ok( + r.issues.some((i) => i.rule_id === 'UI-ASSERT-STRUCTURE-001'), + 'Expected UI-ASSERT-STRUCTURE-001' + ); + const issue = r.issues.find((i) => i.rule_id === 'UI-ASSERT-STRUCTURE-001')!; + assert.equal(issue.severity, 'ERROR'); + assert.ok( + issue.message.includes('fieldAssertions') || issue.message.includes('uiFieldAssertion'), + `Message should mention the nested structure: ${issue.message}` + ); + }); + + // CLEARED: the corpus-confirmed nested form must pass with no false positive. + it('does not fire for the nested uiFieldAssertion structure', () => { + const r = validateTestCase( + ` + + + + + + Values + + + Test + + + + + + + + Test + + + + + + + false + + + + + + + + + + + + +` + ); + assert.ok( + !r.issues.some((i) => i.rule_id === 'UI-ASSERT-STRUCTURE-001'), + 'UI-ASSERT-STRUCTURE-001 should not fire for the nested structure' + ); + }); + + // CLEARED (anchored to the real fixture): the corrected Contact_Lead_nested + // fixture uses the nested form and must pass with no false positive. + it('does not fire for the Contact_Lead_nested.testcase fixture', () => { + const fixturePath = path.resolve(process.cwd(), 'test', 'fixtures', 'testcases', 'Contact_Lead_nested.testcase'); + const xml = fs.readFileSync(fixturePath, 'utf-8'); + const r = validateTestCase(xml); + assert.ok( + !r.issues.some((i) => i.rule_id === 'UI-ASSERT-STRUCTURE-001'), + 'Nested fixture should not trigger UI-ASSERT-STRUCTURE-001' + ); + }); + + it('does not fire for a non-UiAssert step with a fieldLocator-like argument', () => { + const r = validateTestCase( + ` + + + + + + + + + + +` + ); + assert.ok( + !r.issues.some((i) => i.rule_id === 'UI-ASSERT-STRUCTURE-001'), + 'UI-ASSERT-STRUCTURE-001 must only apply to UiAssert steps' + ); + }); + }); + describe('SETVALUES-STRUCTURE-001', () => { it('errors when SetValues values argument uses class="value" (plain string)', () => { const r = validateTestCase( @@ -1082,6 +1329,75 @@ describe('validateTestCase', () => { }); }); +// ── PDX-509: validity bridge (critical BP → is_valid) ───────────────────────── + +describe('PDX-509 — validity bridge', () => { + const UNKNOWN_API_TC = ` + + + + +`; + + // Empty-but-present : Layer-1 TC_020 does NOT fire (steps element exists), + // and the critical VALID-STEPS-001 BP rule is Layer-1-owned, so it must NOT be bridged. + const EMPTY_STEPS_TC = ` + + +`; + + // A {Var} stored as a plain string is VAR-STRING-LITERAL-001 (major) — must NOT gate is_valid. + const MAJOR_ONLY_TC = ` + + + + + {AccountId} + + + +`; + + it('a critical best-practice violation (unknown apiId) flips is_valid=false and surfaces in issues[]', () => { + const r = validateTestCase(UNKNOWN_API_TC); + assert.equal(r.is_valid, false, 'unknown apiId is load-blocking → is_valid must be false'); + const issue = r.issues.find((i) => i.rule_id === 'API-UNKNOWN-001'); + assert.ok(issue, 'API-UNKNOWN-001 should be bridged into issues[]'); + assert.equal(issue?.severity, 'ERROR'); + // The bridged critical also remains in best_practices_violations[] for scoring parity. + assert.ok( + (r.best_practices_violations ?? []).some((v) => v.rule_id === 'API-UNKNOWN-001'), + 'bridged critical stays in best_practices_violations[] (score parity)' + ); + }); + + it('a major best-practice violation does NOT flip is_valid', () => { + const r = validateTestCase(MAJOR_ONLY_TC); + assert.equal(r.is_valid, true, 'a major (runtime) violation loads — is_valid stays true'); + assert.ok( + (r.best_practices_violations ?? []).some((v) => v.rule_id === 'VAR-STRING-LITERAL-001'), + 'the major violation is still reported' + ); + }); + + it('a Layer-1-owned critical (empty ) is NOT bridged — Layer-1 is authoritative', () => { + const r = validateTestCase(EMPTY_STEPS_TC); + // TC_020 only fires on a MISSING ; an empty-but-present loads. + assert.equal(r.is_valid, true, 'empty-but-present loads — is_valid stays true'); + assert.equal( + r.issues.find((i) => i.rule_id === 'VALID-STEPS-001'), + undefined, + 'VALID-STEPS-001 must not be surfaced into issues[] (Layer-1 owns steps presence)' + ); + }); + + it('bridging does not perturb quality_score (Lambda parity preserved)', () => { + const r = validateTestCase(UNKNOWN_API_TC); + const bp = runBestPractices(UNKNOWN_API_TC); + assert.equal(r.quality_score, bp.quality_score, 'quality_score is computed from the untouched BP list'); + }); +}); + // ── Handler-level tests (registerTestCaseValidate) ──────────────────────────── describe('registerTestCaseValidate handler', () => { @@ -1143,6 +1459,79 @@ describe('registerTestCaseValidate handler', () => { assert.ok(warning.includes('Quality Hub'), 'Warning must mention Quality Hub'); }); + describe('PDX-509 — tri-state status + quality_threshold', () => { + const BAD_TC = ` + + + + +`; + + // Loadable (is_valid true) but carries a major (VAR-STRING-LITERAL-001) so quality_score < 100. + const NEEDS_IMPROVEMENT_TC = ` + + + + + {AccountId} + + + +`; + + async function validate(args: Record): Promise> { + const res = (await capServer.capturedHandler!(args)) as { content: Array<{ text: string }> }; + return JSON.parse(res.content[0].text) as Record; + } + + it('status="invalid" + default quality_threshold 90 when a critical defect blocks loading', async () => { + const r = await validate({ content: BAD_TC }); + assert.equal(r['is_valid'], false); + assert.equal(r['status'], 'invalid'); + assert.equal(r['quality_threshold'], 90, 'default threshold is 90'); + }); + + it('status="valid" when loadable and meets the threshold', async () => { + const r = await validate({ content: VALID_TC, quality_threshold: 0 }); + assert.equal(r['is_valid'], true); + assert.equal(r['status'], 'valid'); + assert.equal(r['meets_quality_threshold'], true); + }); + + it('status="needs_improvement" when loadable but below the threshold', async () => { + // Loadable, but a major violation keeps quality_score below the strict bar of 100. + const r = await validate({ content: NEEDS_IMPROVEMENT_TC, quality_threshold: 100 }); + assert.equal(r['is_valid'], true); + assert.ok((r['quality_score'] as number) < 100, 'fixture must score below 100'); + assert.equal(r['status'], 'needs_improvement'); + assert.equal(r['meets_quality_threshold'], false); + }); + + it('per-call quality_threshold overrides the PROVAR_MCP_QUALITY_THRESHOLD env var', async () => { + const saved = process.env.PROVAR_MCP_QUALITY_THRESHOLD; + process.env.PROVAR_MCP_QUALITY_THRESHOLD = '50'; + try { + const r = await validate({ content: VALID_TC, quality_threshold: 80 }); + assert.equal(r['quality_threshold'], 80, 'per-call arg wins over the env var'); + } finally { + if (saved !== undefined) process.env.PROVAR_MCP_QUALITY_THRESHOLD = saved; + else delete process.env.PROVAR_MCP_QUALITY_THRESHOLD; + } + }); + + it('PROVAR_MCP_QUALITY_THRESHOLD is used when no per-call arg is given', async () => { + const saved = process.env.PROVAR_MCP_QUALITY_THRESHOLD; + process.env.PROVAR_MCP_QUALITY_THRESHOLD = '75'; + try { + const r = await validate({ content: VALID_TC }); + assert.equal(r['quality_threshold'], 75, 'env var is the effective threshold'); + } finally { + if (saved !== undefined) process.env.PROVAR_MCP_QUALITY_THRESHOLD = saved; + else delete process.env.PROVAR_MCP_QUALITY_THRESHOLD; + } + }); + }); + it('key + API success → validation_source "quality_hub" with local metadata', async () => { process.env.PROVAR_API_KEY = 'pv_k_testkey12345'; apiStub = sinon.stub(qualityHubClient, 'validateTestCaseViaApi').resolves({ diff --git a/test/unit/mcp/testPlanValidate.test.ts b/test/unit/mcp/testPlanValidate.test.ts index 2c48e36d..6deef85f 100644 --- a/test/unit/mcp/testPlanValidate.test.ts +++ b/test/unit/mcp/testPlanValidate.test.ts @@ -62,7 +62,7 @@ function makeXml(tcGuid: string, stepGuid: string, id: string): string { '', ``, ' ', - ` `, + ` `, ' ', '', ].join('\n'); diff --git a/test/unit/mcp/testSuiteValidate.test.ts b/test/unit/mcp/testSuiteValidate.test.ts index 8afeeb3e..77ae1731 100644 --- a/test/unit/mcp/testSuiteValidate.test.ts +++ b/test/unit/mcp/testSuiteValidate.test.ts @@ -60,7 +60,7 @@ function makeXml(tcGuid: string, stepGuid: string, id: string): string { '', ``, ' ', - ` `, + ` `, ' ', '', ].join('\n'); @@ -345,7 +345,7 @@ describe('provar_testsuite_validate', () => { }); describe('quality_threshold', () => { - it('uses default threshold of 80 when not specified', () => { + it('uses default threshold of 90 (PDX-509) when not specified', () => { // Just verify no error and score is present const result = server.call('provar_testsuite_validate', { suite_name: 'ThresholdDefault', @@ -363,6 +363,38 @@ describe('provar_testsuite_validate', () => { }); assert.equal(isError(result), false); }); + + it('PDX-509: a loadable but sub-threshold case is "needs_improvement", not "invalid"', () => { + // A {Var} stored as a plain string is a major (VAR-STRING-LITERAL-001): the case + // still loads (is_valid true) but scores below 100. + const TC_MAJOR = { + name: 'Major.testcase', + xml_content: [ + '', + ``, + ' ', + ` `, + ' ', + ' {AccountId}', + ' ', + ' ', + ' ', + '', + ].join('\n'), + }; + const result = server.call('provar_testsuite_validate', { + suite_name: 'NeedsImprovement', + test_cases: [TC_MAJOR], + quality_threshold: 100, + }); + const body = parseText(result); + const cases = body['test_cases'] as Array>; + assert.equal(cases[0]['is_valid'], true, 'a major violation still loads'); + assert.equal(cases[0]['status'], 'needs_improvement'); + const summary = body['summary'] as Record; + assert.equal(summary['test_cases_needs_improvement'], 1); + assert.equal(summary['test_cases_invalid'], 0, 'sub-threshold is no longer collapsed to invalid'); + }); }); describe('PDX-470 — detail level', () => { diff --git a/test/unit/mcp/validationRuleRegistry.test.ts b/test/unit/mcp/validationRuleRegistry.test.ts new file mode 100644 index 00000000..676e0012 --- /dev/null +++ b/test/unit/mcp/validationRuleRegistry.test.ts @@ -0,0 +1,162 @@ +/* + * Copyright (c) 2024 Provar Limited. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.md file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ + +import { strict as assert } from 'node:assert'; +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; +import { dirname, join } from 'node:path'; +import { describe, it } from 'mocha'; +import { LAYER1_OWNED_BP_RULES } from '../../../src/mcp/tools/testCaseValidate.js'; + +const repoRoot = join(dirname(fileURLToPath(import.meta.url)), '..', '..', '..'); +const registryPath = join(repoRoot, 'docs', 'VALIDATION_RULE_REGISTRY.md'); +const bpRulesPath = join(repoRoot, 'src', 'mcp', 'rules', 'provar_best_practices_rules.json'); +const layer1RulesPath = join(repoRoot, 'src', 'mcp', 'rules', 'provar_layer1_rules.json'); +const validatorSrcPath = join(repoRoot, 'src', 'mcp', 'tools', 'testCaseValidate.ts'); + +interface BPRule { + id: string; + severity: string; +} + +interface Layer1Rule { + id: string; + severity: 'ERROR' | 'WARNING'; + applies_to: string; + description: string; + owns_bp_rules?: string[]; +} + +describe('Validation Rule Registry (PDX-508 Tier 6 / PDX-511)', () => { + const registry = readFileSync(registryPath, 'utf-8'); + const bp = JSON.parse(readFileSync(bpRulesPath, 'utf-8')) as { rules: BPRule[] }; + // Layer-1 catalog — the single source of truth (PDX-511). Both the registry + // generator and testCaseValidate.ts read this JSON; nothing is hand-copied. + const layer1 = (JSON.parse(readFileSync(layer1RulesPath, 'utf-8')) as { rules: Layer1Rule[] }).rules; + // Layer-2 criticals a Layer-1 check owns — derived from the catalog, NOT hand-listed. + const ownedFromCatalog = new Set(layer1.flatMap((r) => r.owns_bp_rules ?? [])); + + /** Pull the "Gates is_valid?" cell for a given rule id from its table row. */ + function gatingCell(ruleId: string): string | undefined { + const line = registry.split('\n').find((l) => l.includes(`\`${ruleId}\``)); + if (!line) return undefined; + const cells = line.split('|').map((c) => c.trim()); + return cells.find((c) => c === 'Yes' || c === 'No'); + } + + it('lists every best-practice rule (guards against doc drift)', () => { + const missing = bp.rules.filter((r) => !registry.includes(`\`${r.id}\``)).map((r) => r.id); + assert.deepEqual( + missing, + [], + `Registry is stale — re-run scripts/build-validation-rule-registry.cjs. Missing: ${missing.join(', ')}` + ); + }); + + it('includes the core Layer-1 structural rules', () => { + for (const id of ['TC_001', 'TC_010', 'TC_020', 'TC_035', 'COMPARISON-TYPE-001', 'VAR-REF-001']) { + assert.ok(registry.includes(`\`${id}\``), `Expected Layer-1 rule ${id} in the registry`); + } + }); + + it('marks a bridged critical as gating is_valid', () => { + // API-UNKNOWN-001 is critical and not Layer-1-owned → bridged → gates is_valid. + assert.equal(gatingCell('API-UNKNOWN-001'), 'Yes'); + }); + + it('marks a Layer-1-owned critical as NOT gating (suppressed from the bridge)', () => { + assert.equal(gatingCell('VALID-STEPS-001'), 'No'); + }); + + it('marks a major best-practice rule as NOT gating is_valid', () => { + // VAR-STRING-LITERAL-001 is a runtime (major) defect — quality_score only. + assert.equal(gatingCell('VAR-STRING-LITERAL-001'), 'No'); + }); + + it('the Layer-1-owned criticals never gate in the registry', () => { + for (const id of ownedFromCatalog) { + if (registry.includes(`\`${id}\``)) { + assert.equal(gatingCell(id), 'No', `${id} is Layer-1-owned and must not be bridged`); + } + } + }); + + // ── PDX-511: provar_layer1_rules.json is the single source of truth ────────── + // Each guard below fails CI if the Layer-1 catalog drifts from either consumer + // (the registry generator, or the validator's detection + bridge-suppression). + + it('renders every Layer-1 catalog rule with the gating its severity implies', () => { + for (const r of layer1) { + assert.ok( + registry.includes(`\`${r.id}\``), + `Layer-1 rule ${r.id} missing from the registry — re-run scripts/build-validation-rule-registry.cjs` + ); + // Layer-1 ERROR gates is_valid; WARNING is advisory (quality only). + assert.equal(gatingCell(r.id), r.severity === 'ERROR' ? 'Yes' : 'No', `${r.id} gating column mismatch`); + } + }); + + it('renders the Layer-1 counts line from the catalog', () => { + const gating = layer1.filter((r) => r.severity === 'ERROR').length; + assert.ok( + registry.includes(`Layer 1 — ${layer1.length} rules (${gating} gating)`), + `Layer-1 counts line is stale — expected "Layer 1 — ${layer1.length} rules (${gating} gating)"` + ); + }); + + it('derives the validator bridge-suppression set from the catalog (no hand-duplication)', () => { + assert.deepEqual( + [...LAYER1_OWNED_BP_RULES].sort(), + [...ownedFromCatalog].sort(), + 'testCaseValidate.ts LAYER1_OWNED_BP_RULES drifted from provar_layer1_rules.json owns_bp_rules' + ); + }); + + it('catalog owns_bp_rules reference critical best-practice rules', () => { + const bpById = new Map(bp.rules.map((r) => [r.id, r])); + for (const owned of ownedFromCatalog) { + const rule = bpById.get(owned); + // COMPARISON-TYPE-ENUM-001 is a deferred BP rule id (owned pre-emptively); tolerate absence. + if (rule) { + assert.equal(rule.severity, 'critical', `${owned} is owned by a Layer-1 check but is not a critical BP rule`); + } + } + }); + + it('catalogs every Layer-1 rule the validator emits, with matching severity + applies_to (drift guard)', () => { + const src = readFileSync(validatorSrcPath, 'utf-8'); + // Detection sites read `rule_id: 'ID',` then `severity: 'SEV',` then (after the + // message/suggestion) `applies_to: 'SCOPE'`. The validity bridge uses + // `rule_id: v.rule_id` (no string literal) and is skipped. This guard relies on + // that field order: a push that emits severity before rule_id drops out of the + // set below and trips the deepEqual — a loud, if indirect, failure (see message). + const re = + /rule_id:\s*'([A-Za-z0-9_-]+)',\s*\n\s*severity:\s*'(ERROR|WARNING)',[\s\S]*?applies_to:\s*'([A-Za-z]+)'/g; + const emitted = new Map(); + for (let m = re.exec(src); m !== null; m = re.exec(src)) { + const [, id, severity, appliesTo] = m; + const prior = emitted.get(id); + assert.ok( + prior === undefined || (prior.severity === severity && prior.appliesTo === appliesTo), + `${id} is emitted with conflicting metadata across detection sites in testCaseValidate.ts` + ); + emitted.set(id, { severity, appliesTo }); + } + const catalogById = new Map(layer1.map((r) => [r.id, r])); + assert.deepEqual( + [...emitted.keys()].sort(), + [...catalogById.keys()].sort(), + 'Layer-1 rule ids emitted by testCaseValidate.ts differ from provar_layer1_rules.json ' + + '(or a detection push reordered its rule_id/severity fields, which this guard scans for) — update the catalog' + ); + for (const [id, meta] of emitted) { + const rule = catalogById.get(id); + assert.equal(meta.severity, rule?.severity, `${id} severity in the validator differs from the catalog`); + assert.equal(meta.appliesTo, rule?.applies_to, `${id} applies_to in the validator differs from the catalog`); + } + }); +});