feat: security analysis guidelines

[iamsreec]

Summary

Adds the guidelines-security-analysis skill: a 4-step pipeline (architecture analysis -> policy guidance questionnaire -> threat model -> enforcement mapping) that turns a target MCP server's source and OWASP Top 10 for Agentic AI Security (ASI01–ASI10) catalog into a scoped set of security guidelines for OPA policy.

Closes: #issue-number

Changes

• opa_policy/guidelines-security-analysis/guidelines-security-analysis.md — new orchestrator doc that sequences the 4 steps
• opa_policy/guidelines-security-analysis/owasp/architecture_analysis.md — maps an MCP server's layers, trust boundaries, and OPA enforcement points into architecture.md
• opa_policy/guidelines-security-analysis/owasp/policy_guidance_questionnaire.md — derives a 22-question policy-intent questionnaire from architecture.md + guidance.txt/system_vars.json/tool_definitions.json where present
• opa_policy/guidelines-security-analysis/owasp/threat_model.md — evaluates all 10 ASI categories against the tool using a three-question applicability test, sourced live from the catalog rather than hardcoded
• opa_policy/guidelines-security-analysis/owasp/enforcement_mapping.md — maps each threat instance to an enforcement layer (OPA vs. agent/tool/infra) and produces plain-English policy-rule requirements grounded in the catalog's mitigations
• src/smith/data/owasp_10_ai_catalog.json — new repo-relative data file: the OWASP Top 10 for Agentic AI Security catalog (ASI01–ASI10), single source of truth for category names, descriptions, impact, mitigations, and attack scenarios

All four step docs share a consistent "Authoritative Paths" contract: inputs are read from ONLY the named files under <TARGET_AGENT_PATH>/smith/guidelines-security-analysis/ (or the repo-relative catalog path), with an explicit stop-and-ask instruction if a required file is missing.

Checks

• make ci passes (lint, Rego lint, license headers, build smoke)
• make test passes (policy scorecard — needed if policy behavior changed) — not applicable, no assets/policy.rego changes in this PR
• CHANGELOG.md updated under ## [Unreleased] (if user-facing)
• Commits are signed off for the DCO (git commit -s) — verified via git log -1 --format='%B', Signed-off-by present

Notes (optional)

Base branch should be hl/promptfoo_config_auto, not main — this skill was built on top of unreleased work already on that branch. Confirmed clean: git diff --stat hl/promptfoo_config_auto..feat/guidelines-security-analysis shows exactly these 6 files (1392 insertions, 0 deletions), no policy_build/CLI/example files leaking in.

[araujof]

LGTM

[araujof]

@iamsreec do we want to push docs/session-log.md to the main repo?

[araujof]

Can you please address the following? Then, I think we should be good to merge.

• Failed/generated state committed: The PR contains 663 files, including 561 generated test cases, red-team outputs, an SQLite DB, scorecard artifacts, and unreachable.dot. Can you double check if all of these are indeed needed/useful to be in the PR? @dhl123 thoughts on this?

• Fail-open default policy: assets/policy.rego allows whenever no deny rule matches, without validating the request envelope or tool name. For example, a bogus delete_everything request returns allow: true. Is that intended? I wonder if we even want to have that policy.rego stored there under assets. It should probably go with one of our examples. @dhl123 thoughts?

[dhl123]

I had a pass on the PR, below is my suggestions:

Please delete/empty the following files, they are intermediate files:

1. redteam.yaml (delete)
2. Clean assets/policy.rego for a initial setup, it should be an empty file (empty)
3. Remove scorecard under references (delete)
4. Delete unreachable.dot (delete)

Questions on changes to existing code

1. get_tool_definitions in the CLI: please revert the fallback loop. This function should raise failure, not fall back. If it cannot fetch tool definitions, that means the MCP server did not start correctly, and that deserves a hard error rather than a silent degradation. Falling back to a cached file is worse than failing: the cache may hold tool definitions generated in a different session (cross session population), which will silently generate policies against mismatched tools.
2. src/smith/policy_generation/extract_tool.py changes the input schema. But unsure why?

Questions on the artifacts produced by the security analysis

1. The security analysis only produced extra guidances for both the RagChatbot and HR-agent examples. The RagChatbot output looks reasonable, but the HR-agent guidance seems to have been derived from a one line guidance file used for cpex per tool testing rather than whole_guidance.txt. Please regenerate the HR-agent security requirements from ./examples/hr-agent/whole_guidance.txt.
2. For the other agent examples, the analysis doesn't produce new guidance, yet the output shape varies case by case:

• Some examples write a guidance_updated.txt containing content even though nothing was updated. And the contents are different. We would expect, if there is no suggestion-> the file is empty.
• For the call-for-papers agent, the analysis found no suggestions but still overwrote the existing guidance with comments. It shouldn't modify guidance when there's nothing to add, and the guidance file is consumed as plain text, so it has no comment syntax, so injected comments become part of the guidance content, which will populate the later parts.
• Taken together, the return values and operations of security analysis seems to be non-determinstic and we might need to add regulations to force it follows some hardcoded logics rather than reply on LLM.

General suggestion

The security analysis currently injects 4–5 long skill files into SKILL.md, each includes many steps smith need to follow and analyze. These files almost takes the majority of all skill files smith will read. Loading that much text will overwrite the running context and adds noise to every decision the agent makes in the session as well, which hurts both clarity and determinism.

Since this analysis depends only on guidance.txt and not on the agent's runtime logic, one of the following would be great

• Make it a separate skill, invoked only when needed (such as using /security_analysis). It will have its own skill.md
  or
• wrap it as a CLI call that hides details of analysis, and only write outputs that requires human review. (please see duplication suggestion CLIs for details). It will keep SKILL.md small and the main session's context clean.

[dhl123]

Please also use git commit -s -m to add your signature for each commit.

Do not commit test cases, I will regenerate and reshape them

[iamsreec]

I checked the commits in the PR. All non-merge commits already have the Signed-off-by line. The only one without it is a merge commit which should be excluded from the DCO sign-off check.

[iamsreec]

1. The main issue here is that these generated/intermediate artifacts are not currently excluded by .gitignore, which is why they ended up being tracked in the PR. I cleaned up the current files, but .gitignore should also be updated so the same issue does not recur in future runs.

Addressed:

1. redteam.yaml — deleted
2. assets/policy.rego — cleared for the initial setup
3. references/scorecard/ — deleted
4. unreachable.dot — deleted

Please also update .gitignore to exclude references/scorecard/, /redteam.yaml, and /unreachable.dot. Otherwise, these generated artifacts will be added again in future iterations and become a recurring blocker.

2. get_tool_definitions — done. I removed the fallback and reverted it to raise an error if tool definitions cannot be fetched, so it won’t silently use potentially stale cached definitions from another session.

3. extract_tools.py — this change was added for MCP 1.x/2.x compatibility. MCP 1.x uses Tool.inputSchema, while MCP 2.x uses Tool.input_schema, and pyproject.toml currently allows mcp[cli]>=1.0.0 without restricting the major version. The getattr handling keeps tool extraction compatible with both versions. This is also causing the make lint error now.

4. whole_guidance.txt is currently not an input to the pipeline because it is located at the example root rather than under the smith/ directory. As a result, the security analysis did not consume it during the run.

If we want the full HR-agent guidance to be used end to end, whole_guidance.txt would need to be moved or merged into the guidance file under smith/, and then the pipeline can be rerun to regenerate the security requirements and test cases from that input.

5. Maintained uniformity in updated guidances.

[dhl123]

LGTM