knowledge: improve review precision from BCApps PR 10010 feedback - #171
Draft
Wenjie Fan (gggdttt) wants to merge 1 commit into
Draft
knowledge: improve review precision from BCApps PR 10010 feedback#171Wenjie Fan (gggdttt) wants to merge 1 commit into
Wenjie Fan (gggdttt) wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Improves BCQuality knowledge based on maintainer thumbs-down feedback from explicit BCApps PR review runs.
Source feedback
Validation
Generated by the BC-ALAgentsInternal self-improvement workflow.
Offline evaluation: regression
Candidate correctness failed: unexpected or missed findings remain. Existing ignored gold comments retain their neutral scoring semantics.
Preparation attempt 1: invalid_selection. Coverage requires a patched file and valid lineStart/lineEnd for synthetic__testing-permission-mock-webservice-01.
Preparation attempt 2: validated.
Selection: new. New:
synthetic__testing-permission-mock-webservice-01. Reused (complete payloads): ``.synthetic__testing-permission-mock-webservice-01/false_positive_guard/ Add Report Inbox API pages for automated report retrieval BCApps#10010 (comment): Negative (thumbs-down) feedback on the flagged test codeunit, with the maintainer explaining the TestPermissions = Disabled / no-permission-lowering pattern is correct here because the API pages run under the web-service session's identity, not the test codeunit's session -- so lowering permissions on the test session cannot exercise the ReadPermission/WritePermission gates anyway. The synthetic entry reproduces that same web-service E2E shape (Library - Graph Mgt driving the call, TestPermissions = Disabled, no Permissions Mock) and asserts expect_findings=false with no expected or ignored comment in the codeunit-declaration/TestPermissions range, guarding the general 'permission tests must lower the execution context' rule against over-firing on this out-of-process E2E pattern. Limitation: this only covers the web-service E2E boundary; it does not establish that in-process permission tests calling business logic directly are exempt, nor does it validate the API triggers' gates themselves are correct.Coverage references and outcome shapes are validated mechanically. Semantic equivalence, severity calibration and recommendation quality are NOT proved by these checks or a matching F1.
Common engine code:
ecf8e31759d6ddd6d78e3a0b7836b40134368009; dataset SHA256:DA04922C9B6506D8E6C092A272730988DBC9996DF1E7E9720B923A4D89A2051E.Model:
gpt-5.6-luna; judge:gpt-5.3-codex. Exact D:synthetic__testing-permission-mock-webservice-01.Dataset base:
26365d6a41a96fac41df9dcc8339cb71241fcb1f; candidate:fbc8fe239114f17deeb12dbfe93b5bc9d7348d6e.9959dba2cf76b0cf5fac84b4a94ddb48db91ecb74407f20b44fe92e3693d0444d25cc3430f1d52628584217c7506eea7eef27a9db353d78c0cd6a9a1b4574666fe6f35bd4e23ab989f610fc2248b06196e98a05685d5085ac0a22ef97a406ed23cca0face0805a987539c59b83b87342fd62169e7bc4f3f0Baseline
Run: https://github.com/microsoft/BC-Bench/actions/runs/34352868908; conclusion: success; wall clock: 8.4 minutes.
Raw findings and runtime pin proofs are retained in the self-improvement-feedback artifact and the linked evaluation run.
synthetic__testing-permission-mock-webservice-01Observed AI credits: unavailable; coverage 0/1 entries. Missing entries are not extrapolated into a total.
Candidate
Run: https://github.com/microsoft/BC-Bench/actions/runs/34353758347; conclusion: success; wall clock: 8.7 minutes.
Raw findings and runtime pin proofs are retained in the self-improvement-feedback artifact and the linked evaluation run.
synthetic__testing-permission-mock-webservice-01Observed AI credits: unavailable; coverage 0/1 entries. Missing entries are not extrapolated into a total.
Missing telemetry is unavailable, not zero. Evaluation-only metrics exclude candidate generation and are not the full-cycle cost.
Human review must verify source-patch fidelity, gold correctness, and target attribution. No automatic merge or branch-protection claim is made.