Merge remote-tracking branch 'origin/master' into fix/adhoc-viz-fallback-missing-id

Author: Tomkess

packages/gooddata-eval/AGENTS.md

@@ -4,15 +4,15 @@
 `gdc-nas`) through a dataset of natural-language questions and scores what comes back,
 including side-by-side comparison across models. Each dataset item is a JSON envelope
 loaded from a local folder or pulled from a Langfuse dataset. Results are aggregated into
-pass@K / pass^K reports and optionally pushed to Langfuse as scored traces tied to a
-dataset run. The newest and most actively developed package in the repo.
+pass@K / pass^K reports and optionally pushed to Langfuse as scored traces tied to an
+experiment. The newest and most actively developed package in the repo.
 
 ## Owns
 
 - The `gd-eval` CLI (`gd-eval run`, `gd-eval models`)
 - Dataset loading and the evaluation run loop
 - Per-capability evaluators and their scoring
-- Result reporting, and pushing runs, scores and trace links to Langfuse
+- Result reporting, and pushing experiments, scores and trace links to Langfuse
 
 ## Does NOT Own
 
@@ -29,7 +29,7 @@ dataset run. The newest and most actively developed package in the repo.
 | `core/summary/` | HTTP client for the dedicated dashboard-summary endpoint — a single-shot chat backend, not reporting |
 | `core/dataset/` | dataset format and loading |
 | `core/evaluators/` | single-shot evaluators and their registry |
-| `core/langfuse/` | `sink.py` only — pushes single-turn scores and dataset-run items |
+| `core/langfuse/` | the whole Langfuse v4 client: `_env` (base URL + credentials), `otlp` (OTLP/JSON encoding), `experiment` (root-span construction, score targets), `observations` (trace reads), `client` (httpx calls), `sink` (single-shot results as experiments) |
 | `core/reporting/` | console and JSON output rendering |
 | `core/scoring.py`, `core/runner.py` | scoring and orchestration |
 | `core/models.py` | `DatasetItem`, `ChatResult`, `ItemReport` and friends |

packages/gooddata-eval/README.md

@@ -113,7 +113,7 @@ gd-eval run \
 | Flag | Description |
 |---|---|
 | `--dataset PATH` | Flat folder of JSON files — one question per file. |
-| `--langfuse-dataset NAME` | Pull items by name from a Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--langfuse-dataset NAME` | Pull items by name from a Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` and `LANGFUSE_BASE_URL` (or the legacy `LANGFUSE_HOST`). |
 | `--kind TEST_KIND` | Fallback `test_kind` for dataset items that do not embed one. Defaults to `visualization`; use e.g. `agentic_metric_skill` for multi-turn agentic evaluation. Items that declare their own `test_kind` ignore this. |
 
 #### Model selection
@@ -152,15 +152,39 @@ interleaves when K > 1, and per-item latencies rise, so they stop being clean si
 
 | Flag | Description |
 |---|---|
-| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Names each experiment run `{dataset_name}_{timestamp}_{model}`, suffixed `_effort-{level}` when `--reasoning-effort` is set (so runs differing only by effort stay separate) and `_run{N}` per run when `--runs` > 1 — e.g. `general_question_2026-09-02-11-13_gpt-5.2_run0`. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Names each experiment run `{dataset_name}_{timestamp}_{model}`, suffixed `_effort-{level}` when `--reasoning-effort` is set (so runs differing only by effort stay separate) and `_run{N}` per run when `--runs` > 1 — e.g. `general_question_2026-09-02-11-13_gpt-5.2_run0`. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` and `LANGFUSE_BASE_URL` (or the legacy `LANGFUSE_HOST`). |
 
-Set `TAVERN_E2E_SKIP_TRACE_LINK=1` to skip trace lookup entirely (scores are then orphaned; the run says so).
+##### Langfuse v4
 
-**A local `--dataset` cannot be attached to a Langfuse run.** `--langfuse` is refused alongside `--dataset`
-because a local folder's item ids are not Langfuse dataset item ids. But trace linking does not depend on that
-flag — each `evaluate_agentic_*` builds its own client whenever `LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are
-exported — so a local run still finds its traces and writes its scores onto them, and only the per-run grouping
-fails, with one `404 from dataset-run-items` reported per run. The run warns about this before it starts. Use
+One run is one Langfuse **experiment**. Each evaluated item becomes its own trace whose root span carries the
+experiment and dataset-item attributes (`langfuse.experiment.name`, `langfuse.experiment.dataset.id`,
+`langfuse.experiment.item.id`), and the four scores attach to that root observation. gd-eval speaks to Langfuse
+over four REST endpoints and uses no Langfuse SDK, so it runs on every Python version the package supports:
+
+| Endpoint | Used for |
+|---|---|
+| `POST /api/public/otel/v1/traces` | exporting the experiment root span as OTLP/HTTP JSON |
+| `POST /api/public/scores` | one score per write, on a trace or on a single observation inside it |
+| `GET /api/public/v2/observations` | finding the agent's gen-ai trace for a conversation |
+| `GET /api/public/dataset-items` | loading `--langfuse-dataset` items, and resolving an item's dataset id |
+
+Two consequences of v4's immutable observations. gd-eval sets `version` only on its own experiment span, never on
+the agent's gen-ai trace — filter on the gd-eval experiment's `langfuse.version` to compare models. And on the
+agentic kinds the latency in `value_score` is the gen-ai trace's root generation latency, read from the
+observations endpoint; the single-shot `--langfuse` sink keeps using the item's own measured average latency.
+
+Langfuse Cloud drops v3 on **2026-11-16**; a self-hosted Langfuse must be on v4 for any of this to work.
+
+Set `TAVERN_E2E_SKIP_TRACE_LINK=1` to turn the whole **agentic** Langfuse write path off — no trace lookup, no
+span export and no scores for `agentic_*` items. The run says so once. It does not reach the `--langfuse` sink,
+which still writes a span and four scores for every single-shot item; drop `--langfuse` to silence that too.
+
+**A local `--dataset` cannot be attached to a Langfuse experiment.** `--langfuse` is refused alongside
+`--dataset` because a local folder's item ids are not Langfuse dataset item ids. But trace linking does not
+depend on that flag — each `evaluate_agentic_*` builds its own client whenever
+`LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are exported — so a local run still finds its traces and writes its
+scores onto them, and only the per-run grouping fails: the dataset-item lookup 404s and the run reports the item
+as one that does not exist in Langfuse, once. The run also warns about this before it starts. Use
 `--langfuse-dataset` when you want runs that are comparable across models, or `TAVERN_E2E_SKIP_TRACE_LINK=1` to
 skip linking altogether.
 
@@ -380,6 +404,10 @@ Without `[llm-judge]`, those items are **skipped**.
 
 ## Scores (in JSON report and Langfuse)
 
+In Langfuse every score is written to the experiment run's root observation — `traceId` plus `observationId` of
+the item's own root span. On the agentic path each score is mirrored onto the agent's gen-ai trace as well
+(`traceId` only), so a score survives even when one of the two traces is missing.
+
 | Score | Description |
 |---|---|
 | `pass_at_k` | 1 if any of the K runs passed strict checks, else 0. |

packages/gooddata-eval/src/gooddata_eval/cli/main.py

@@ -181,15 +181,14 @@ def _apply_timer_flag(enabled: bool) -> None:
 
 
 def _warn_if_local_dataset_cannot_link(config: RunConfig, agentic_items: list) -> None:
-    """Say up front that dataset-run assembly will fail, rather than after the run.
+    """Say up front that experiment assembly will fail, rather than after the run.
 
     --langfuse is refused outright with a local dataset because local item ids cannot be
     linked. But every evaluate_agentic_* falls back to try_make_langfuse_client() when the
-    caller passes none, so with LANGFUSE_* exported the linking runs anyway and each
-    conversation earns a 404 from dataset-run-items -- arriving in a block at the very end
-    of the run, long after the flag that would have prevented it could be changed. The
-    fallback is deliberate (direct library and tavern callers rely on it), so this warns
-    instead of disabling it.
+    caller passes none, so with LANGFUSE_* exported the linking runs anyway and every
+    dataset-item lookup 404s -- arriving in a block at the very end of the run, long after
+    the flag that would have prevented it could be changed. The fallback is deliberate
+    (direct library and tavern callers rely on it), so this warns instead of disabling it.
     """
     from gooddata_eval.core.agentic._langfuse import SKIP_ENV_VAR, langfuse_credentials_present  # noqa: PLC0415
     from gooddata_eval.core.config import env_flag  # noqa: PLC0415
@@ -201,7 +200,7 @@ def _warn_if_local_dataset_cannot_link(config: RunConfig, agentic_items: list) -
     print(
         f"warning: --dataset is a local folder, so its item ids are not Langfuse dataset item ids. "
         f"Traces will be found and scored, but the per-run grouping that makes models comparable "
-        f"cannot be created and each conversation will report a 404 from dataset-run-items. "
+        f"cannot be created and each conversation will report that its item does not exist in Langfuse. "
         f"Use --langfuse-dataset for comparable runs, or set {SKIP_ENV_VAR}=1 to skip trace linking.",
         file=sys.stderr,
     )