perf(ingest): send every TOC batch at once (HAL-1545) - #70
Open
Conversation
Each Judge phase of the TOC stage — detection fan-out, detection, verification, contents confirmation, page resolution and heading splitting — built one batch, sent it, waited for the answer and only then built the next. The batches never depended on each other, so a phase's wall clock was the sum of its requests where it could have been the longest of them. They now build every batch and hand the set to sendJudgeBatches, which sends them together and lets llmgate's adaptive limiter decide how many are in flight. The first failure cancels the rest and fails the phase, never a partial set, which is the contract every caller already had. The leaf splitter walks one generation at a time and splits every leaf of a generation at once; each leaf accounts into its own Usage, folded back in tree order so the ledger and its Degraded list read the same on every run. The resolver now chooses between tied pages in sorted key order instead of map order. The request budget stays at 24k. The 6k this issue first proposed rested on a probe the 2026-09-25 evaluation retracted.
Reviewer's GuideThe PR reduces TOC Judge-stage latency by batching all independent requests concurrently through a shared cancel-on-failure helper and parallelizing leaf splits generation by generation, while preserving phase-level failure behavior, request/result ordering, deterministic usage accounting, and deterministic page-resolution ties. Sequence diagram for concurrent TOC Judge batchessequenceDiagram
participant Phase as TOCPhase
participant Helper as sendJudgeBatches
participant Judge as llmgateJudge
Phase->>Helper: sendJudgeBatches(ctx, Judge, reqs)
par Independent request 1
Helper->>Judge: Judge(ctx, reqs[0])
Judge-->>Helper: Judgment[0]
and Independent request 2
Helper->>Judge: Judge(ctx, reqs[1])
Judge-->>Helper: Judgment[1]
and Additional requests
Helper->>Judge: Judge(ctx, reqs[i])
Judge-->>Helper: Judgment[i]
end
Helper-->>Phase: results in request order
alt First request fails
Judge-->>Helper: error
Helper->>Helper: cancel()
Helper-->>Phase: firstErr, no partial results
end
Flow diagram for generation-by-generation leaf splittingflowchart TD
Start["Collect splittable leaves"] --> Generation["Process one generation"]
Generation --> Parallel["splitLeaf for every leaf concurrently"]
Parallel --> Fold["merge Usage and results in tree order"]
Fold --> Children{Children created?}
Children -->|Yes| Generation
Children -->|No| Done["Return deterministic ledger and tree"]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Six Judge loops in the TOC stage built a batch, sent it, waited, then built the next. The batches were independent, so each phase took the sum of its requests where it could have taken the longest.
What
sendJudgeBatches: builds every batch first, sends them together, and lets the adaptive limiter set concurrency. The first failure cancels the rest and fails the phase, never returning a partial set (the contract every caller already had).Usage, folded back in tree order so the ledger andDegradedlist are deterministic.Evidence
21 FinanceBench filings, one at a time, Judge-only (
docs/evaluations/2026-09-28-latency-was-our-cpu.md§2):Title recall against the original build is 0.944. The tokenizer fix alone, which asks identical questions, reaches 0.949, so that's Jev's run-to-run floor and this sits inside it. The issue's "≥ 0.96" bar can't be met by re-running unchanged code.
Tests: concurrency, order, and cancel-on-first-failure for
sendJudgeBatches; the existing ingest suite.go test -racegreen locally. CI blocked (HAL-1712).Stacked on #69.
Closes HAL-1545
Summary by Sourcery
Reduce TOC ingest latency by concurrently executing independent Judge work while preserving deterministic accounting, ordering, and failure behavior.
Enhancements:
Tests: