Skip to content

knowledge: improve review precision from BCApps PR 10010 feedback - #171

Draft
Wenjie Fan (gggdttt) wants to merge 1 commit into
mainfrom
self-improvement/bcapps-10010-quality
Draft

knowledge: improve review precision from BCApps PR 10010 feedback#171
Wenjie Fan (gggdttt) wants to merge 1 commit into
mainfrom
self-improvement/bcapps-10010-quality

Conversation

@gggdttt

@gggdttt Wenjie Fan (gggdttt) commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Improves BCQuality knowledge based on maintainer thumbs-down feedback from explicit BCApps PR review runs.

Source feedback

Validation

  • Frontmatter and article structure validation completed.

Generated by the BC-ALAgentsInternal self-improvement workflow.

Offline evaluation: regression

Candidate correctness failed: unexpected or missed findings remain. Existing ignored gold comments retain their neutral scoring semantics.

Preparation attempt 1: invalid_selection. Coverage requires a patched file and valid lineStart/lineEnd for synthetic__testing-permission-mock-webservice-01.

Preparation attempt 2: validated.

Selection: new. New: synthetic__testing-permission-mock-webservice-01. Reused (complete payloads): ``.

  • synthetic__testing-permission-mock-webservice-01 / false_positive_guard / Add Report Inbox API pages for automated report retrieval BCApps#10010 (comment): Negative (thumbs-down) feedback on the flagged test codeunit, with the maintainer explaining the TestPermissions = Disabled / no-permission-lowering pattern is correct here because the API pages run under the web-service session's identity, not the test codeunit's session -- so lowering permissions on the test session cannot exercise the ReadPermission/WritePermission gates anyway. The synthetic entry reproduces that same web-service E2E shape (Library - Graph Mgt driving the call, TestPermissions = Disabled, no Permissions Mock) and asserts expect_findings=false with no expected or ignored comment in the codeunit-declaration/TestPermissions range, guarding the general 'permission tests must lower the execution context' rule against over-firing on this out-of-process E2E pattern. Limitation: this only covers the web-service E2E boundary; it does not establish that in-process permission tests calling business logic directly are exempt, nor does it validate the API triggers' gates themselves are correct.
    Coverage references and outcome shapes are validated mechanically. Semantic equivalence, severity calibration and recommendation quality are NOT proved by these checks or a matching F1.

Common engine code: ecf8e31759d6ddd6d78e3a0b7836b40134368009; dataset SHA256: DA04922C9B6506D8E6C092A272730988DBC9996DF1E7E9720B923A4D89A2051E.
Model: gpt-5.6-luna; judge: gpt-5.3-codex. Exact D: synthetic__testing-permission-mock-webservice-01.
Dataset base: 26365d6a41a96fac41df9dcc8339cb71241fcb1f; candidate: fbc8fe239114f17deeb12dbfe93b5bc9d7348d6e.

Arm Evaluation commit Engine pin Knowledge pin
baseline 9959dba2cf76b0cf5fac84b4a94ddb48db91ecb7 4407f20b44fe92e3693d0444d25cc3430f1d5262 8584217c7506eea7eef27a9db353d78c0cd6a9a1
candidate b4574666fe6f35bd4e23ab989f610fc2248b0619 6e98a05685d5085ac0a22ef97a406ed23cca0fac e0805a987539c59b83b87342fd62169e7bc4f3f0

Baseline

Run: https://github.com/microsoft/BC-Bench/actions/runs/34352868908; conclusion: success; wall clock: 8.4 minutes.
Raw findings and runtime pin proofs are retained in the self-improvement-feedback artifact and the linked evaluation run.

Entry Expected Generated Missed Unexpected F1 AI credits Agent seconds
synthetic__testing-permission-mock-webservice-01 0 2 0 2 0 unavailable 307.889716136

Observed AI credits: unavailable; coverage 0/1 entries. Missing entries are not extrapolated into a total.

Aggregate metric Value
total 1
expected_comment_count 0
generated_comment_count 2
matched_comment_count 0
missed_comment_count 0
incorrect_comment_count 2
precision 0
recall 1
f1 0

Candidate

Run: https://github.com/microsoft/BC-Bench/actions/runs/34353758347; conclusion: success; wall clock: 8.7 minutes.
Raw findings and runtime pin proofs are retained in the self-improvement-feedback artifact and the linked evaluation run.

Entry Expected Generated Missed Unexpected F1 AI credits Agent seconds
synthetic__testing-permission-mock-webservice-01 0 3 0 3 0 unavailable 325.780951482

Observed AI credits: unavailable; coverage 0/1 entries. Missing entries are not extrapolated into a total.

Aggregate metric Value
total 1
expected_comment_count 0
generated_comment_count 3
matched_comment_count 0
missed_comment_count 0
incorrect_comment_count 3
precision 0
recall 1
f1 0

Missing telemetry is unavailable, not zero. Evaluation-only metrics exclude candidate generation and are not the full-cycle cost.
Human review must verify source-patch fidelity, gold correctness, and target attribution. No automatic merge or branch-protection claim is made.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant