Skip to content

dataset(code-review): add synthetic reaction gold answers from BCApps PR 10255 - #869

Draft
Wenjie Fan (gggdttt) wants to merge 1 commit into
mainfrom
self-improvement/bcapps-10255-synthetic-eval
Draft

dataset(code-review): add synthetic reaction gold answers from BCApps PR 10255#869
Wenjie Fan (gggdttt) wants to merge 1 commit into
mainfrom
self-improvement/bcapps-10255-synthetic-eval

Conversation

@gggdttt

@gggdttt Wenjie Fan (gggdttt) commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds minimal synthetic code-review gold answers from explicit BCApps PR feedback.

Full source PR patches are intentionally excluded. Positive reactions become
isolated expected findings; thumbs-down findings become precision guards by
omission. Existing equivalent gold answers are not duplicated.

Accepted findings

Rejected findings

Knowledge dependency

Validation

See the gate-owned offline evaluation section below.

Generated by the BC-ALAgentsInternal self-improvement workflow.

Offline evaluation: regression

Candidate correctness failed: unexpected or missed findings remain. Existing ignored gold comments retain their neutral scoring semantics.

Preparation attempt 1: validated.

Selection: new. New: synthetic__privacy-activity-log-lifecycle-01. Reused (complete payloads): ``.

  • synthetic__privacy-activity-log-lifecycle-01 / false_positive_guard / Add expense activity history foundation BCApps#10255 (comment): Reproduces the rejected finding's exact shape: an OnDelete trigger on a posted document header cascades deletion of activity-log rows matched by source table number and the document's SystemId, mirroring PostedExpenseReportHeader.OnDelete calling ExpenseActivityLogMgt.DeleteEntriesForSource(Database::"Posted Expense Report Header", Rec.SystemId). The developer reply confirms this cascade is by-design lifecycle ownership to avoid orphaned actor/comment/amount personal data, not an audit-trail defect, so expected_comments is empty by omission. Limitation: this guard only covers the cascade-delete-on-OnDelete boundary for activity/audit-style child records; it does not generalize to cases where the child table holds compliance-mandated retention data with no privacy offset, which could still warrant a finding.
    Coverage references and outcome shapes are validated mechanically. Semantic equivalence, severity calibration and recommendation quality are NOT proved by these checks or a matching F1.

Common engine code: ecf8e31759d6ddd6d78e3a0b7836b40134368009; dataset SHA256: D3F92967ABDE17CF834EFFE7ED69523E42DDDCA8A135B9D8A24A7A0313034A05.
Model: gpt-5.6-luna; judge: gpt-5.3-codex. Exact D: synthetic__privacy-activity-log-lifecycle-01.
Dataset base: 26365d6a41a96fac41df9dcc8339cb71241fcb1f; candidate: c555f9294b31cfe6dc807e43f276a1f5bb9f53af.

Arm Evaluation commit Engine pin Knowledge pin
baseline ac33eb8e634f538ce5b377078ee6f263a548207d 4407f20b44fe92e3693d0444d25cc3430f1d5262 8584217c7506eea7eef27a9db353d78c0cd6a9a1
candidate c83280f7651a03d27e3ddd09b8c676d8eaa001e9 8558750e58fac1624187201ae462383f4d697661 ff91a136a29cc8c49a78c6e3739bcd976dec4f91

Baseline

Run: https://github.com/microsoft/BC-Bench/actions/runs/34349857118; conclusion: success; wall clock: 7.9 minutes.
Raw findings and runtime pin proofs are retained in the self-improvement-feedback artifact and the linked evaluation run.

Entry Expected Generated Missed Unexpected F1 AI credits Agent seconds
synthetic__privacy-activity-log-lifecycle-01 0 3 0 3 0 unavailable 260.790731108

Observed AI credits: unavailable; coverage 0/1 entries. Missing entries are not extrapolated into a total.

Aggregate metric Value
total 1
expected_comment_count 0
generated_comment_count 3
matched_comment_count 0
missed_comment_count 0
incorrect_comment_count 3
precision 0
recall 1
f1 0

Candidate

Run: https://github.com/microsoft/BC-Bench/actions/runs/34350651192; conclusion: success; wall clock: 7.5 minutes.
Raw findings and runtime pin proofs are retained in the self-improvement-feedback artifact and the linked evaluation run.

Entry Expected Generated Missed Unexpected F1 AI credits Agent seconds
synthetic__privacy-activity-log-lifecycle-01 0 2 0 2 0 unavailable 250.606904361

Observed AI credits: unavailable; coverage 0/1 entries. Missing entries are not extrapolated into a total.

Aggregate metric Value
total 1
expected_comment_count 0
generated_comment_count 2
matched_comment_count 0
missed_comment_count 0
incorrect_comment_count 2
precision 0
recall 1
f1 0

Missing telemetry is unavailable, not zero. Evaluation-only metrics exclude candidate generation and are not the full-cycle cost.
Human review must verify source-patch fidelity, gold correctness, and target attribution. No automatic merge or branch-protection claim is made.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant