th-1cc9fa: model-output ceiling clamp + raise starvation defaults (TypeScript parity) - #195
th-1cc9fa: model-output ceiling clamp + raise starvation defaults (TypeScript parity)#195brentrager wants to merge 1 commit into
Conversation
… server) TypeScript parity with the merged Rust reference (rust/smooth-operator-server: admin.rs model_output_ceiling/map_model_info + config.rs DEFAULT_MAX_TOKENS/ ITERATIONS). - modelCeiling.ts: best-effort per-model output ceiling from the gateway's /model/info (extractModelCeilings + createGatewayModelCeilingResolver), cached once per process, undefined on any error => engine leaves max_tokens unclamped. The TS analog of admin.rs's fetch_model_costs/model_output_ceiling/map_model_info. - TurnRunner: raise DEFAULT_MAX_TOKENS 512->8192 and DEFAULT_MAX_ITERATIONS 6->20 (chat-widget sizing starved reasoning models), thread the per-turn ceiling into the engine via AgentOptions.modelMaxOutput, and set an explicit DEFAULT_MODEL (= engine default) that both the request and the ceiling lookup share. - Thread model + modelCeiling through FrameDispatcher, ServerOptions, serveLocal; main.ts builds the resolver from SMOOAI_GATEWAY_URL/KEY (undefined on the keyless local path => unclamped, behaviour unchanged). - Bump @smooai/smooth-operator-core pin to ^0.20.1 (the release introducing modelMaxOutput / effectiveMaxTokens). Tests: modelCeiling extract/resolve (caching, best-effort, auth header, single-flight) + TurnRunner clamp-down / passthrough / unclamped. Server suite green (57 tests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
|
Superseded — this exact work is already on main. The TypeScript server's model-output ceiling clamp + raised defaults landed via #199 ( Nothing here is unlanded, so closing rather than rebasing. Triage pearl th-b093c7. |
1 participant
Why
A budget/policy
max_tokenscan exceed what a model can physically emit. When it does, reasoning models burn the whole budget onreasoning_contentand return emptycontent, or the upstream 400s (e.g.groq-compoundcaps output at 8192). Separately, the server's chat-widget-sized defaults (max_tokens=512,max_iterations=6) starve reasoning / multi-step turns.What (TypeScript server)
Mirrors the already-merged Rust reference on this branch —
rust/smooth-operator-server/src/admin.rs(model_output_ceiling/fetch_model_costs/map_model_info),config.rs(DEFAULT_MAX_TOKENS/DEFAULT_MAX_ITERATIONS),handler.rs+runner.rs(thread the ceiling into the turn) — in TS idioms:modelCeiling.ts(new): best-effort per-model output ceiling from the gateway's/model/info.extractModelCeilings(payload)(pure, themap_model_infoanalog, narrowed tomodel_info.max_output_tokens) +createGatewayModelCeilingResolver(url, key, fetch?), fetched at most once per process and cached; any error / unknown model ⇒undefined⇒ the engine leavesmax_tokensunclamped (graceful, no behaviour change).TurnRunner: raiseDEFAULT_MAX_TOKENS512→8192 andDEFAULT_MAX_ITERATIONS6→20 (safe now that the clamp bounds per-model output; a cap doesn't lengthen concise answers). Look up the resolved model's ceiling per turn and thread it into the engine viaAgentOptions.modelMaxOutput. An explicitDEFAULT_MODEL(= the engine's own default) is now set so the request model and the ceiling-lookup model are always the same.model+modelCeilingthroughFrameDispatcher→ServerOptions/serveLocal;main.tsbuilds the resolver fromSMOOAI_GATEWAY_URL/SMOOAI_GATEWAY_KEY(absent on the keyless local path ⇒ unclamped, behaviour unchanged).@smooai/smooth-operator-corepin to^0.20.1— the release that introducesmodelMaxOutput/effectiveMaxTokens(th-1cc9fa: model-output ceiling clamp + raise starvation defaults (TypeScript parity) smooth-operator-core#71).Mirroring Rust
undefined/unset ceiling ⇒ no clamp, identical to the RustNonepassthrough. The ceiling is sourced by the consumer (this server) from/model/infoand passed into the engine, keeping LiteLLM-specific HTTP out of the published engine — same split as Rust.Tests
modelCeiling: payload extraction (skips missing name / no ceiling / non-integer), resolver caching (one fetch),{url}/model/info+ bearer header, no-key path, best-effortundefinedon non-ok / thrown fetch (not cached → retries), single-flight across concurrent first lookups.TurnRunner: raised default budget sent; clamp-down when ceiling < budget; passthrough when ceiling ≥ budget; unclamped on unknown model; ceiling looked up for the model the turn actually uses.tscbuild + typecheck clean (verified against a local link to the modified core).Caveat — release ordering (see companion core PR)
The
modelMaxOutputoption ships in@smooai/smooth-operator-core0.20.1 (SmooAI/smooth-operator-core#71). Until that publishes, a freshpnpm installhere can't resolve the bumped^0.20.1pin (the committed lockfile still points at the stale prior core). The TS server is a private package and is not built/tested in PR CI, so this doesn't gate the PR — but land/publish the core PR first, then refresh the lockfile.🤖 Generated with Claude Code