Skip to content

Fix flushing causing delays in evals that don't use traces - #220

Merged
David Elner (delner) merged 1 commit into
mainfrom
fix/eval-trace-flush-delays
Sep 23, 2026
Merged

David Elner (delner) merged 1 commit into
mainfrom
fix/eval-trace-flush-delays

Conversation

@delner

Copy link
Copy Markdown
Collaborator

Fixes #210.

What happened

Every eval case called force_flush synchronously between task and scorers. A customer measured 10-25s per call on a 139-case eval — most of a ~24 minute run.

The flush exists so a scorer declaring trace: can BTQL-query that case's spans while they're fresh. It ran unconditionally, including when nothing in the eval touches trace:.

How this fixes it

Scorers and classifiers already expose #call_parameters (used by KeywordFilter to drop undeclared kwargs). Runner reuses it to decide once whether anything can receive trace:, then gates the per-case work:

kase.trace = build_trace(eval_span) if @needs_trace
flush_spans if kase.trace

Gating on kase.trace also skips local-only evals, where build_trace returns nil. **kwargs callables count as consumers since they may forward trace:. No false negatives — KeywordFilter slices kwargs to declared keys, so trace: can't arrive undeclared.

Two supporting changes:

  • Legacy positional scorers now report a real signature. wrap_block rewrote them as ->(**kw), which reads as "accepts anything" — so every legacy scorer would have kept flushing. Wrappers now declare the keywords they forward. Behavior unchanged; KeywordFilter already sliced those kwargs.
  • One flush at the end of Runner#run. Trace.setup only registers the at_exit hook when it owns the provider, so callers passing their own tracer_provider: relied on the per-case flush for delivery. N flushes become 1, and it covers the last case's spans, which the per-case flush never reached.

@delner David Elner (delner) self-assigned this Sep 23, 2026
@delner
David Elner (delner) requested a review from a team as a code owner September 23, 2026 18:28
@delner David Elner (delner) added the bug Something isn't working label Sep 23, 2026
@delner
David Elner (delner) merged commit 122ca58 into main Sep 23, 2026
10 checks passed
@delner
David Elner (delner) deleted the fix/eval-trace-flush-delays branch September 23, 2026 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Eval::Runner's per-case force_flush adds 10-25s+ per case even when no scorer/classifier uses trace:

3 participants