Repository navigation
Improve model usage accounting across providers - #1887
Conversation
Streamed requests had gained stream_options on the default router road, changing its established wire bytes. Only connected services now ask for that option, following the actual endpoint after a billing-road change. The router retains its existing usage.include request. The usage option had also admitted unrelated streamed 400s to learn(), and a connected endpoint refusing it with 422 failed a call dev answered. Keep the old repair eligibility independent of the usage option and retry only an explicit unsupported-option refusal on 400 or 422. Remember that refusal for the endpoint without changing unrelated errors. Cover router and connected wire shapes, fallback and refusal memoization, dev error/request-count parity, raw-stream single settlement, and the separation between token estimates and billing. Update the manual and usage disclosures with the precise roads and refusal statuses. Ollama 0.9.0 reuses generation ids from a small random range. The former receipt key collapsed separate answers and reused event ids with different counts. Include the physical attempt in that key, and carry it through late recovery so estimates still pair with their measured receipt. Cover repeated provider ids and raw-stream recovery without duplicate rows or bills. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Previously a caller's status could disagree with emitted components, unsafe integers could reach the relay, and late receipts could turn absent counts into measured zeroes. Derive status from safe emitted components, omit invalid counts and unsafe sums, and require a receipt identity for estimates. Keep late token presence through a session-bound recorder. Cost-only recovery emits no token update; one-sided recovery stays partial. Extend the shared fixture and test its outputs against the relay's validation rules. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Previously whitespace generation ids split recovery from observation, repeated tool names inflated estimates, opaque replay counted as input text, and a metadata-only cut invented observed zero output. Canonicalize the generation inside the physical-attempt receipt helper and count only honest observations. Compose independent repairs once each without widening unrelated refusal handling. Associate usage-option refusal wording with that option. Preserve late token presence and recovered prices. Keep unmetered direct-service answers off receipt lookup because recovery retains the original bearer. Prove native cache aliases match canonical billing counts and token pricing; no estimate is banked or priced. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Previously the disclosure omitted the total-only exception and overstated measurement and recovery behavior. Describe component-preserving recovery, canonical receipt identities, safe numeric limits, opaque input, empty output, and bounded composed repairs. Record the deliberate native-alias measured-count change and unchanged no-lookup behavior for unmetered direct-service overflow. Estimates remain separate from billing, response usage and spend. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Took this over to merge-ready. Four commits on top of Santosh's, each with tests named for the behaviour they prove:
Verified: CI green on this head; Merge order stays as Santosh set it: this goes in once the relay deploy from #82 is confirmed live. |
AbirAbbas
left a comment
There was a problem hiding this comment.
Verified; see the comment above. Merge waits on the relay deploy confirmation.
Hotfix of #1887 onto v0.8.0, carrying only that change to main. Connected services ask for streamed token counts with stream_options.include_usage (the default router road keeps its existing usage request), an explicit 400/422 refusal of the option retries once and is remembered per endpoint, and missing text counts gain separately labelled estimates that never enter billing. Each physical answer keeps its own hashed receipt so a later provider receipt replaces its estimate in analytics. The relay that accepts the new usage statuses shipped as website-open-infra #82. The settings page keeps v0.8.0's resident practice paragraph; the rest of the change is byte-identical to #1887's head 9445d02 except where dev-only commits had moved the surrounding lines. Co-Authored-By: Abir Abbas <abirabbas1998@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The v0.8.1 hotfix carries #1887 alone on top of v0.8.0, so its section holds that change and the v0.8.0 reconciliation entry (#1825) that was still loose on main. Both entries leave the unreleased folder here, and the merge back into staging and dev removes them there too, so the next roll-up on dev cannot list them a second time. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
main receives #1887 alone as v0.8.1; merging that hotfix here keeps every later staging-to-main promotion a fast-forward and lets the release guard find the rc commit on staging. Where staging already matched dev, the merged files are byte-identical to #1887's head. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
#1887 reached dev as 663dfbf and main as the v0.8.1 hotfix. This merge brings that hotfix's history and changelog roll-up here, so dev keeps containing staging and main. Code is unchanged; the v0.8.1 section lands in CHANGELOG.md, and the #1825 and #1887 entries it rolled up leave the unreleased folder so the next roll-up does not list them again. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Local and custom model endpoints could finish calls without token usage, leaving gaps in usage accounting. This improves compatible-provider handling with requested streaming receipts and clearly separated estimates when measured counts are unavailable.
Companion: https://github.com/Agent-Field/website-open-infra/pull/82. Deploy that relay change first, then release this client change. Existing measured-only charts remain valid.
Validation: focused provider/config/accounting regressions, privacy/allowlist and disclosure checks, the exact shared client-to-relay fixture, a 50-event body-limit check, production build, vet, and formatting pass. The companion relay's 35 route tests pass with PostHog mocked. No synthetic production events were sent.
The broad local test gate was stopped by automatic approval review when it attempted an OpenRouter request with an unestablished payload. Targeted mock-based checks and build/vet completed; CI remains the full changed-package check.