Skip to content

fix: reduce Captain token consumption by 3x#99

Merged
DenysKuchma merged 2 commits into
mainfrom
fix/captain-token-usage
Jul 20, 2026
Merged

fix: reduce Captain token consumption by 3x#99
DenysKuchma merged 2 commits into
mainfrom
fix/captain-token-usage

Conversation

@DavertMik

Copy link
Copy Markdown
Contributor

Problem

A Langfuse trace of a single read-only Captain request ("whats on this page") consumed 32,821 tokens across two ~16K-input generations. Diagnosis showed three independent sources of waste:

  1. A no-op roundtrip after done() — the SDK loop only stopped on stepCount, so after done() resolved the request, the full 16.4K context was re-sent for a generation that produced 1 token (an empty assistant message). 50% of the trace.
  2. Unconditional full-HTML injectionreinjectContextIfNeeded guarded on page_html (which the initial prompt never contains), so it always fired on the first iteration, adding ~4.5K tokens of raw HTML plus a duplicate copy of the ARIA snapshot the initial prompt had just sent.
  3. runCommand description inlined the entire /help text (3,720 chars ≈ 1K tokens per call) even though isExplicitSlashRequest programmatically restricts the tool to verbatim pass-through of user-typed slash commands.

Fix

  • Pass stopWhen: () => isDone to invokeConversation — the loop halts the moment done() succeeds. A rejected done() (missing details) doesn't flip isDone, so retries still happen within the same invocation.
  • reinjectContextIfNeeded now guards on page_aria and injects URL/title + ARIA only. HTML remains available on demand via the web-mode context() tool.
  • runCommand description trimmed to a one-line rule plus command names.

Impact

For the traced request: one generation instead of two, at ~10K input instead of ~16K — ~10K total vs 32.8K. Savings apply to every Captain roundtrip (no prompt caching on this model tier, so context is repaid in full each step).

Testing

  • bun test tests/unit/captain-artifacts.test.ts tests/unit/captain-mode.test.ts — 19 pass
  • bun test tests/integration/ — 63 pass
  • bun run format clean

🤖 Generated with Claude Code

- stop the tool loop as soon as done() succeeds instead of burning a
  full-context follow-up generation that returns nothing
- reinject ARIA-only page context guarded on page_aria, dropping the
  unconditional full-HTML block (HTML stays available via context())
- trim runCommand tool description to command names; the explicit
  slash-request guard already enforces pass-through behavior

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown

Explorbot Self-Regression

Commit 0762fd0 · run

Scenario Result Attempts Duration
basic (native) PASS 1/3 9m
experience: control OK — failed as expected 1/1 1m
experience: seeded PASS 2/3 3m

Attempt details

  • basic (native) attempt 1 — PASS: login evidence: PASS (post-login plan=true, post-login research=true); research: PASS (files=5, wellFormed=true, keywords=6/3); scenarios: PASS (tests=7/5, features=4/3); tests passed: PASS (5 passed, 0 failed (reporter: 5 passed, 0 failed))
  • experience: control attempt 1 — PASS: control: OK — failed as expected (0 passed, 1 failed)
  • experience: seeded attempt 1 — FAIL: seeded: FAIL (0 passed, 1 failed)
  • experience: seeded attempt 2 — PASS: seeded: PASS (1 passed, 0 failed)

Session analysis — basic (native):

Session Analysis

The Issues feature exploration confirms the core flows work: creating issues, searching by keyword, filtering by status, filtering by label, and handling empty results. All five tests achieved their stated goals. One test had minor automation trouble clearing a filter after verification, but the primary filtering functionality itself is sound.

Coverage

  • Pages: /issues, /issues?new=1, /issues?q=<keyword>&status=<status>, /issues?label=<id>
  • Features: Create issue, Keyword search, Status filtering, Label filtering, Empty state handling

What works

  • Issue creationET-1 Create a new issue using the New Issue button
  • Keyword searchET-2 Search for an existing issue by keyword
  • Status filteringET-3 Filter issues by status
  • Label filteringET-4 Filter issues by label
  • Empty state handlingET-5 Search for a non-existent issue

Execution Issues

  • ET-4 Filter issues by label — click on "All labels" to clear filter failed twice; core filter functionality works but post-verification cleanup unreliable

@DenysKuchma
DenysKuchma merged commit adaaa7d into main Jul 20, 2026
3 checks passed
@DenysKuchma
DenysKuchma deleted the fix/captain-token-usage branch July 20, 2026 10:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants