[Tests + Fix]: Added unit tests, Fixed duplication in promptfoo config file generation, changed remaining "arguments" to args for CPEX

[dhl123]

Summary

A cleanup and bug-fix pass over the test-generation pipeline, and test scripts creation, plus the unit and integration test suites.

Closes

#12 #46 #49

Changes

1. args-migration.

• Changed arguments in docs/policy_creation.md, docs/test_generation.md, policy_cross_validation.md, README.md, and the two docs guides to args for cpex compatibility. (problems left in last PR)

2. agent's default value changes in promptfoo config.

• Changed the default agent port from 8000 to 9000 in promptfooconfig.yaml files.

3. made adjusts to cpex hr-agent example

• Delete guidance that cannot be enforced by opa (requires approval etc).
• Added a readme file for running hr-agent example.

4. delete usage of top-k

• We changed all openai client to aws. aws models only take either temp or topk, so we deleted topk.

5. null values handling in convert_test_case.py.

• _convert_var in src/smith/test_generation/convert_test_case.py called int(None) / float(None) when a generated case supplied null for a numeric system variable, raising TypeError. Most often triggered by adversarial Promptfoo cases that omit an integer field (e.g. queries_this_session). Now returns None, leaving the field absent so OPA treats it as unset.

**6. remove any previously-appended tool-parameter block in promptfoo config generation (problems left in last PR)

• Promptfoo config updates weren't idempotent. generate_promptfoo_config.py re-appended the tool-parameter block on every run, so the prompt grew each update. Added a sentinel marker and a strip step so the block is replaced instead of duplicated.

Problem identified when writing unit and integration tests.

7. Missing env setup values

• Added variables the CLI reads but .env_template omitted. Added variables: TEST_OUTPUT_DIR, TEST_PATH, MCP_URL, TOP_P, MAX_LLM_CALLS.

8. Two modules read the environment themselves — attack.py and
attack_promptfoo.py called load_dotenv()/os.getenv inside the function the CLI
invokes. Both now take the value as a parameter, resolved in cli.py with every other
path. Three functions gained required parameters; cli.py is their only caller.

9. AST graph indices inherited across calls — node ids came from the size of two
module-level dicts that were never reset, so the same policy parsed twice produced
different graphs. init_graph now clears them first.

10. convert_test_coverage.py ran its whole pipeline at import — and crashed
without configuration, which broke the policy_testing unit lane. Environment
resolution and I/O moved into main(); the transformation logic extracted into
importable functions. Output byte-identical, so the frozen coverage numbers still hold.

11. apply_cross_validate crashed when the destination didn't exist — shutil.move
doesn't create directories, so promoting a case into an absent allow/ raised
FileNotFoundError mid-loop, leaving the tree partially applied. Added
os.makedirs(..., exist_ok=True) before each move.

12. Whitespace-only lines now dropped in decompose.py — remove_empty_line tested
len(strr) != 0, so a line of spaces survived and became a rule, spending an LLM call
to produce a meaningless record. Now judged with .strip(); line content untouched.

13. Discarded exit code hid Regal failures — subprocess.run ran without
check=True and its result was thrown away, and the shell pipeline reported sed's
status anyway. So a missing regal binary produced an empty report and a successful
flag: the same answer as a clean policy. Regal is now invoked directly and its exit
code classified (0 clean, 3 violations, anything else a failure). check=True
would have been wrong — it raises on 3, the normal result. Output byte-identical
wherever Regal actually runs.

Tests

• To run the unit tests: make unit
• To run the integration test: make integration