Skip to content

build-catalog: fuzzy matcher can enrich a model with a DIFFERENT model's pricing #387

Description

@wz-gsa

Defect (data-correctness, caught during a live gateway regeneration)

integrations/providers/usai/scripts/build-catalog.mjs's models.dev fuzzy matcher is too permissive: when it cannot find a correct match for a gateway model id, it falls through to a different model's data instead of leaving the fields absent.

Reproduced: with only claude-opus-5 / claude-sonnet-5 present under the anthropic provider in the enrichment subset, claude_4_5_haiku was enriched with Opus numbers (context 1,000,000, $5/$25 — correct is 200,000, $1/$5) and claude_4_5_sonnet picked up Sonnet-5 pricing ($2/$10 vs the correct $3/$15).

This is worse than missing data: the fields look populated and plausible, so a completeness check ("every model has cost/context") passes while the values are wrong. Shipping that would tell users Haiku costs $5/$25.

Fix

Add a precision guard: a candidate is accepted only when it agrees on both

  1. family/tierhaiku must not match opus/sonnet; flash must not match pro; flash-lite must not match plain flash; mini/nano/pro variants must not match the base model; and
  2. major.minor version4.5 must not match 5 or 4.8 (handle 4_5, 4.5, -4-5, bare 5).

If no candidate satisfies both, return no match and emit the model with absent context/output/cost. Absent-but-honest beats populated-but-wrong.

Keep the legitimate normalizations that were verified working: gateway-suffix strips (guardrails.*, latest.*, -default-?v\d+) so gpt_5_5_default_v2 still matches gpt-5.5, and the llama4llama-4 glue-split for the Bedrock meta.llama4-maverick-… form.

Tests to add

Hermetic (inline models.dev-shaped fixtures, not the committed subset):

  • claude_4_5_haiku must not match an opus/sonnet entry; claude_4_5_sonnet must not match claude-sonnet-5.
  • gemini-2.5-flash must not match gemini-2.5-pro; gemini-2.5-flash-lite must not match plain gemini-2.5-flash.
  • Positive controls that must still match: gpt_5_5_default_v2gpt-5.5, llama_4_maverick→bedrock llama4-maverick, claude-opus-5claude-opus-5, gemini-3.5-flashgemini-3.5-flash.

Context

Deferred from the urgent model-refresh PR (which shipped correct values by carrying forward the reviewed numbers for the unchanged ids rather than trusting the matcher). The guard is what makes future live regenerations safe to trust.

AI-assisted (OpenCode).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingvalidationSchema, linting, tests, CI validation

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions