perf: take llmgate's guard fix; report where Judge time goes (HAL-1708) - #69
Open
hallelx2 wants to merge 2 commits into
Conversation
The Judge client rebuilt its tokenizer on every count, 256 ms a time, once per question, inside the limiter's slot (HAL-1708). Taking the fix, same engine code, first 12 FinanceBench questions, back to back: llmgate v0.5.0 median 50.5 s 12/12 4.2 req $0.00368 llmgate 010e07c median 2.2 s 12/12 4.2 req $0.00369 The provider did identical work. The 37-to-50-second query was our own CPU, and nothing reported it: a round-trip number cannot tell local preparation from the provider. So every Judge request now reports itself. The engine and server log each one at debug, and warn when local preparation passes 250 ms, which is a few milliseconds when healthy. navbench and tocdump print prepare, first-byte and total percentiles at the end of a run. The Judge is warmed at startup so the first query does not open the connection or build the tokenizer. llmgate is pinned to the fix's commit until its PR merges and is tagged.
Measured before and after the guard fix, the ingest fan-out and the retrieval skim, on the FinanceBench corpus: query median 50.5 s to 1.9 s at 36/40 and a lower cost per question, ingest median 66.1 s to 4.1 s per filing at 47/47. Marks the 2026-09-25 latency figures as superseded.
Reviewer's GuideThis PR upgrades llmgate to the tokenizer-guard fix, warms Judge clients before use, and adds tracing from client preparation through provider response so engine logs and benchmark summaries expose the true latency split. It also adds concurrency-safe benchmark statistics, tests for warning and summary behavior, and documentation showing the resulting query and ingest improvements while superseding prior latency figures. Sequence diagram for warmed and traced Judge requestssequenceDiagram
participant Engine
participant Judge as TypeSafeJudge
participant Guard as TokenizerGuard
participant Provider
participant Logger
Engine->>Judge: Warm(ctx)
Judge->>Guard: initialize tokenizer
Judge->>Provider: open connection
Provider-->>Judge: connection ready
Engine->>Judge: Judge request
Judge->>Guard: count context
Guard-->>Judge: preparation complete
Judge->>Provider: send request
Provider-->>Judge: first byte and response
Judge-->>Engine: result
Judge-->>Logger: OnRequest(RequestTrace)
alt prepare exceeds 250 ms
Logger-->>Logger: Warn slow local preparation
else normal preparation
Logger-->>Logger: Debug request timing
end
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Sep 28, 2026
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 2 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="cmd/engine/main.go" line_range="458-464" />
<code_context>
}
+ // Open the connection and build the guard's tokenizer now, so the
+ // first query after boot does not pay for either.
+ go func() {
+ ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+ defer cancel()
+ if err := j.Warm(ctx); err != nil {
+ logger.Warn("judge: warm-up failed; the first request will open its own connection", "err", err)
+ }
</code_context>
<issue_to_address>
**issue (bug_risk):** `Warm` runs asynchronously and the wrapped Judge is returned immediately, so the first request can start before warm-up completes and still pay the connection/tokenizer initialization cost; concurrent initialization also depends on `llmgate` making `Warm` and request preparation safe to run together.
**Triggers:** When a request arrives immediately after engine or server startup.
**Suggested fix:** Run warm-up synchronously before returning the Judge, or expose a readiness barrier that requests await until `Warm` completes.
```suggestion
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if err := j.Warm(ctx); err != nil {
logger.Warn("judge: warm-up failed; the first request will open its own connection", "err", err)
}
```
</issue_to_address>
### Comment 2
<location path="cmd/navbench/main.go" line_range="85-87" />
<code_context>
}
key := os.Getenv(typesafe.EnvAPIKey)
- tj, err := typesafe.New(typesafe.Config{APIKey: key})
+ tj, err := typesafe.New(typesafe.Config{APIKey: key, OnRequest: stats.Observe})
if err != nil {
fmt.Fprintln(os.Stderr, "judge:", err)
</code_context>
<issue_to_address>
**issue (broader_impact):** The benchmark Judge clients register timing hooks but never call `Warm`, so the first navigation question or first document still pays cold connection/tokenizer setup and its timing is included in the run instead of benefiting from the advertised startup warm-up.
**Triggers:** On the first request of every navbench run or the first Judge request of a tocdump run.
**Suggested fix:** Warm these clients before starting the measured run, using a bounded context and handling a warm-up failure explicitly.
</issue_to_address>
Comment on lines
+458
to
+464
| go func() { | ||
| ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) | ||
| defer cancel() | ||
| if err := j.Warm(ctx); err != nil { | ||
| logger.Warn("judge: warm-up failed; the first request will open its own connection", "err", err) | ||
| } | ||
| }() |
There was a problem hiding this comment.
issue (bug_risk): Warm runs asynchronously and the wrapped Judge is returned immediately, so the first request can start before warm-up completes and still pay the connection/tokenizer initialization cost; concurrent initialization also depends on llmgate making Warm and request preparation safe to run together.
Triggers: When a request arrives immediately after engine or server startup.
Suggested fix: Run warm-up synchronously before returning the Judge, or expose a readiness barrier that requests await until Warm completes.
Suggested change
| go func() { | |
| ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) | |
| defer cancel() | |
| if err := j.Warm(ctx); err != nil { | |
| logger.Warn("judge: warm-up failed; the first request will open its own connection", "err", err) | |
| } | |
| }() | |
| ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) | |
| defer cancel() | |
| if err := j.Warm(ctx); err != nil { | |
| logger.Warn("judge: warm-up failed; the first request will open its own connection", "err", err) | |
| } |
Comment on lines
-85
to
87
| tj, err := typesafe.New(typesafe.Config{APIKey: key}) | ||
| tj, err := typesafe.New(typesafe.Config{APIKey: key, OnRequest: stats.Observe}) | ||
| if err != nil { | ||
| fmt.Fprintln(os.Stderr, "judge:", err) |
There was a problem hiding this comment.
issue (broader_impact): The benchmark Judge clients register timing hooks but never call Warm, so the first navigation question or first document still pays cold connection/tokenizer setup and its timing is included in the run instead of benefiting from the advertised startup warm-up.
Triggers: On the first request of every navbench run or the first Judge request of a tocdump run.
Suggested fix: Warm these clients before starting the measured run, using a bounded context and handling a warm-up failure explicitly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The 37–50 s query was our own CPU. llmgate's context guard rebuilt its tokenizer on every count (256 ms), once per question, inside the limiter slot. The provider answered in ~0.6 s the whole time.
What
docs/evaluations/2026-09-28-latency-was-our-cpu.md. It marks the 2026-09-25 latency figures as superseded.Evidence
Same engine code, only llmgate swapped, first 12 FinanceBench questions, back to back: median 50.5 s → 2.2 s. Still 12/12, with identical requests, tokens and $/q.
Ingest, 21 filings: median per filing 66.1 s → 8.8 s with this change alone. Coverage still 47/47.
Locally:
go build ./...,go vet,go test -racegreen. CI cannot run because the GitHub account is locked on billing (HAL-1712). Not merge-ready until it does.Stacked on #68. #70 and #71 stack on this.
Closes HAL-1708
Summary by Sourcery
Reduce Judge latency by adopting the llmgate guard fix, warming clients, and exposing request timing breakdowns in services and benchmarks.
New Features:
Bug Fixes:
Enhancements:
Documentation:
Tests: