Repository navigation
Conversation
…adata
When gd-eval joins gen-ai's turns into the item's trace, gen-ai's spans
name the experiment through baggage but carry no experiment metadata.
Langfuse then lists most such experiments with empty metadata, although
gd-eval's root span carries all of it. On the staging nightly since the
join shipped, 60 of 68 joined experiments of one dataset list `{}`,
while every unjoined one keeps its metadata. Readers that filter
experiments by github_run_id, branch or workflow drop those runs.
baggage_entries now adds a langfuse_experiment_metadata_<key> member
per run metadata key. The langfuse SDK on the gen-ai side maps each one
to langfuse.experiment.metadata.<key>, so the spans of the experiment
carry the metadata the root carries. A string goes as is and any other
value as JSON, as the root span shows it. A key that is not a plain
token is skipped with one warning, the rule GOODDATA_EVAL_TRACE_LABELS
already applies, and the two now share BAGGAGE_KEY.
eval_k and eval_gate are stamped at scoring time, after the turns ran,
so gen-ai's spans still lack those two keys.
jira: QA-29847
risk: low
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage |  |
Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 9 functions across 3 files. | Write docstrings for the functions missing them to satisfy the coverage threshold. |
-
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title clearly and concisely describes the main change: sending run metadata to gen-ai as experiment metadata. Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches: 💡 1
-
📝 Generate docstrings: 💡
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
A rabbit packs metadata tight,
And quotes each value just right.
String, number, list go in,
Nulls and bad keys stay thin.
The baggage hops into the night.
Comment @coderabbitai help to get the list of available commands.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1897 +/- ##
=======================================
Coverage 85.10% 85.10%
=======================================
Files 337 337
Lines 24343 24358 +15
=======================================
+ Hits 20716 20731 +15
Misses 3627 3627 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GOODDATA_EVAL_JOIN_GENAI_TRACE, on by default since feat(gooddata-eval): put gen-ai turns under the eval item root trace #1863), gen-ai's spans name the experiment through baggage but carry no experiment metadata. Langfuse then lists most such experiments withmetadata: {}, although gd-eval's root span carries all of it.agent_metric_skilllist empty metadata, 0 of 184 unjoined ones (pre-merge, same SDK version) do. Readers that filter experiments bygithub_run_id, branch or workflow (the gdc-nas eval trend and the PR-comment baseline) drop those runs.baggage_entriesnow adds onelangfuse_experiment_metadata_<key>member per run-metadata key. The langfuse SDK on the gen-ai side (checked on 4.14.4, gen-ai's lock, and 4.15.2) maps each one tolangfuse.experiment.metadata.<key>. A string goes as is, any other value as JSON, as the root span shows it. A key that is not a plain token is skipped with one warning; that rule andGOODDATA_EVAL_TRACE_LABELSnow shareBAGGAGE_KEY.eval_kandeval_gateare stamped at scoring time, after the turns ran, so gen-ai's spans still lack those two keys. Nothing in gdc-nas reads them from the experiment listing. The baggage header has no size budget; the run metadata in use today is eight short keys.Test Plan
test_the_run_metadata_rides_along_as_experiment_metadataintests/test_sse_client.py: the metadata members, a bool sent astrue, and keys with a space, a comma or non-ASCII skipped.Risk Assessment
low — adds baggage members to the chat request only when the item joins gen-ai's trace; gen-ai drops caller
langfuse_*baggage unlessacceptCallerBaggageis on. Revert: full.jira: QA-29847
risk: low
🤖 Generated with Claude Code
Summary by CodeRabbit