WIP: OIE mapping pipeline + OWASP eval harness (not merge-ready)

[northdpole]

Status

Draft. Not ready for merging. Visibility snapshot of the living OIE librarian/harvester pipeline and the eval data the harnesses actually run on. Do not treat CI green/red as a merge signal. Do not write these decisions into the live OpenCRE graph.

B2 gold for AISVS/API/LLM/Top10/K8s loads from application.utils.mapping_fixtures (merged in #1098) on the PR branch. Local dual-index worktree eval uses CRE_B2_GOLD_DIR=scripts/oie_owasp_eval/fixtures/b2_gold/ overrides (Related / AI-taxonomy / Agentic graph-hint remaps) — eval-only, canonical mapping fixtures unchanged until mentors agree.

Summary

• Post-GSoC librarian/harvester/orchestrator work
• OWASP eval harness scripts under scripts/oie_owasp_eval/
• Dual-index + hub/leaf librarian path (flags default OFF): CRE_LIBRARIAN_CRE_SUMMARY, CRE_LIBRARIAN_DUAL_INDEX
• Harness testing data at scripts/oie_owasp_eval/fixtures/:
  • b2_gold/ — agentic stub on PR branch; worktree also holds local remaps for classical/LLM/Agentic scoring experiments
  • b2_sources/ — section text the B2 harness scores against
  • reports/ — latest B1 / B2 / Module C accuracy JSON
• Librarian golden set remains at application/tests/librarian/fixtures/golden_dataset.json

How it is tested today

Vertical	What it measures	Latest snapshot
Librarian golden (evaluate_librarian.py)	C.0 boundary, C.0.5 explicit ids, C.1 recall@20, C.2 top-1, C.3 ECE, C.4 auto-link vs review	Retrieval ~98% recall@20; top-1 ~75%; decision accuracy at τ=0.80 ~57%; review recall 5/5
B1 (score_b1_opencre_api.py)	ASVS chunk / CheatSheet sheet vs live opencre.org CRE gold	33/37 (89%), gate >60% pass
B2 (run_b2_pr_mappings.py)	New-family section grain: ≥1 of union(rerank top-2 ∪ vector top-2) ∈ gold cre_ids	See denominators below

B2 denominators (honest)

Gold / arm	Score	Notes
Mapping-fixtures gold (PR harness reports)	~26–35/67	Canonical classic CRE grain
Local Related-remap + Pro+dual+hub/leaf	43/67 (64%)	Classical Top10 10/10; LLM still 0/10
+ Local LLM AI-taxonomy expand	53/67 (79%)	LLM 10/10 (canonical LLM gold is classic CREs from GSoC fixtures)
+ Local Agentic graph/hint expand	58/67 (87%)	Agentic 5/5; no Agentic hub Links on prodclone — gold expanded from stub + graph/hint, not imported Links
Excl. Agentic	53/62 (85%) after LLM expand	Already ≥80% without Agentic

Full gate ≥54/67 met on local remapped gold (58/67). Classical near-misses remain (API5, AISVS3/8/9, K8s); deferred. Do not read 58/67 as production accuracy on canonical fixtures.

Cold-start (hub-link gold, HUB_LINK_SEED=0): 7/21 — hub Links sparse on prodclone; not a substitute for Related-remap.

Reproduce

# B2 against mapping_fixtures gold + in-repo sources
PYTHONPATH=. python scripts/oie_owasp_eval/run_b2_pr_mappings.py --score-only --run-id <id>

# Local remap gold (worktree)
CRE_B2_GOLD_DIR=scripts/oie_owasp_eval/fixtures/b2_gold \
PYTHONPATH=. python scripts/oie_owasp_eval/run_b2_pr_mappings.py --score-only --run-id <id>

# Librarian golden (hermetic)
python scripts/evaluate_librarian.py \
  --dataset application/tests/librarian/fixtures/golden_dataset.json

Test plan

• Mentors / contributors: read the reports and comment on ranking, corpus (CRE descriptions), and B2 misses — not “please merge”
• B2 gold for AISVS/API/LLM/Top10/K8s comes from mapping_fixtures; agentic stub remains local
• Document local dual-index + gold-remap scores with honest denominators (58/67 local remap; not canonical)
• Before any real merge: split, rebase onto main, drop WIP leftovers, re-run B1/B2

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}