Preserve unknown AI credits in bceval exports

[Wenjie Fan (gggdttt)]

Fix

Preserve ai_credits: null when exporting a result to bceval instead of turning unknown usage into a measured zero. Observed zero and nonzero credits are unchanged. The only production-code change is removal of this export fallback.

Valid nullable usage is distinct from missing or invalid metrics

A missing engine metrics file or a schema-invalid payload already raises AgentError; those failures remain unchanged. Separately, the evaluated engine's Get-CopilotRunMetrics/Save-CopilotRunMetrics can produce a valid schema-v1 payload with unavailable numeric usage:

• No chat spans: usage_complete=false, malformed_records=0, nullable usage fields.
• A valid chat span with token counts but no billing attribute: complete token usage, ai_credits=null.

Both payloads are accepted by the existing strict BC-Bench loader. New regression cases follow that loader through the result and bceval export; they do not merely inject None into the exporter. The exact engine producer functions at 159572aad814d6e1022ca8d54e85a6eab7c4d5c6 were executed offline and their outputs matched the regression payloads.

This establishes the supported nullable path, not the collection root cause of run 34203457236. Its old raw telemetry is unavailable, so why that run lacked usage remains unknown. No engine correction is claimed.

Scope after review

Removed the proposed _run-metrics.json artifact retention and all associated workflow download filtering/mock-artifact naming changes. All workflow files and workflow tests match the PR merge base; intermediate engine telemetry remains abstracted from CI orchestration. No pin, model, package/lockfile, result schema, publication, requeue, or scoring changes.

bc-eval 0.3.14 directly adds prompt/completion tokens before the custom callback and cannot accept null there. Existing numeric token export behavior is intentionally unchanged; there is no new total-token export. Nullable AI credits are accepted as metadata. Gate token accounting must use original per-entry result metrics rather than bceval token fallbacks.

Validation

• 51 focused metrics/reporting/export tests passed, including existing missing-file and invalid-schema failures.
• Ruff format/check and targeted ty passed.
• Actual producer outputs for both nullable cases passed through the real BC-Bench loader/result/export and the exact bc-eval 0.3.14 row method with offline test sinks; no model or storage calls.
• Automatic CI 34221892958 passed at 62c283424eb3cf1514fcbd8e9347df7fdcfbc721, including summarize-results / Results and its bceval step. No evaluation/model dispatch, rerun, or merge was requested.

[Sun Haoran (haoranpb)]

Do we know why this happens?

missing upstream usage

Missing or schema-invalid metrics already appear to fail, maybe additional corn cases we should address instead?

---

Not sure if we want to keep "_run-metrics.json", it should have been an intermediate step, and abstracted away from the overall orchestration

[Wenjie Fan (gggdttt)]

Thanks, agreed on keeping the intermediate artifact out of orchestration. I removed its retention and all related workflow/filter/mock-naming changes in 62c2834; those files now match the PR base.

There are two distinct cases:

• A missing metrics file or schema-invalid payload already fails in _load_run_metrics; that behavior and its tests are unchanged.
• A valid producer payload can still have unavailable numeric usage. I executed the evaluated engine's actual Save-CopilotRunMetrics/Get-CopilotRunMetrics functions offline: no chat spans produces a valid payload with usage_complete=false, malformed_records=0, and null usage; a valid token-bearing chat span without billing produces complete tokens but ai_credits=null. Both pass the existing strict loader. The new regression follows these producer-matched payloads through the loader, result, and export and catches the exporter changing unknown credits to zero.

We still do not know why the old live run lacked usage: its raw telemetry is no longer available. This is an export-correctness fix, not a collection-root-cause claim. The final production change is only preserving null AI credits. Token export remains unchanged because bc-eval 0.3.14 cannot add null token counts; its actual consumer accepts nullable credit metadata. No engine behavior or failure checks were relaxed.

New live evidence (separate from the old run): The completed paired gate run 34221000483 retained valid schema-v1 aggregates from CLI 1.0.82: baseline 34221111997 and candidate 34222250479 both have malformed_records=0, usage_complete=false, and ai_credits=null. Token-bearing chat counts are 23/165 and 26/174; the respective 3,123,685 and 3,903,159 token totals are partial sums, not complete token cost. Executing build_pr_review_metrics from the frozen Bench base 46b6ad94497fcb272417b170b2cbb7b3f96d7209 against both stored aggregates exactly reproduces their saved per-entry metrics: incomplete token usage is intentionally suppressed, while credits were already null independently of the token-completeness flag. Both experiment commits leave that loader unchanged. This explains these new runs' null projection, but not why many chat spans lack usage/billing, nor the old run 34203457236's collection issue. The scoped experiment proof artifacts do not reinstate production retention in this PR.

[Sun Haoran (haoranpb)]

Thanks for the update.

Going forward, I think the ai_credits will probably be a very important metric, we could fail the jobs when is it None, but that is out of scope of this PR