[bot] Merge master/1d27d3fa into rel/dev

packages/gooddata-eval/AGENTS.md

@@ -27,6 +27,7 @@ experiment. The newest and most actively developed package in the repo.
 | `core/agentic/` | multi-turn agentic evaluation per capability, **plus** all Langfuse trace polling and linking (`_langfuse.py`, `_trace_linker.py`) |
 | `core/chat/` | SSE client for the agent's streaming chat endpoint |
 | `core/summary/` | HTTP client for the dedicated dashboard-summary endpoint — a single-shot chat backend, not reporting |
+| `core/takeaway/` | HTTP client for the AI Publisher takeaway endpoint — a single-shot backend like `core/summary/`, keeping the whole response on `ChatResult.raw_response` — plus `evaluate_document_takeaway`, the K-run, gate and Langfuse entry point the Tavern shim calls |
 | `core/dataset/` | dataset format, loading, and `from_insights.py` — dataset generation from a workspace's real insights |
 | `core/evaluators/` | single-shot evaluators and their registry |
 | `core/langfuse/` | the whole Langfuse v4 client: `_env` (base URL + credentials), `otlp` (OTLP/JSON encoding), `experiment` (root-span construction, score targets), `item_scope` (an item's experiment scope sent to gen-ai as baggage, and which conversations joined), `observations` (trace reads), `client` (httpx calls), `sink` (single-shot results as experiments) |
@@ -59,7 +60,8 @@ implementation, sometimes under different `test_kind` strings (`search_tool` vs
 `expected_output: Any`. `expected_output` is deliberately untyped — each evaluator parses
 its own shape. `test_kind` on the item is what labels the result, not the evaluator class,
 which is why `knowledge_question` can reuse `GeneralQuestionEvaluator` verbatim.
-`dashboard_summary` items additionally need `summary_input`.
+`dashboard_summary` items additionally need `summary_input`, `document_takeaway` items
+`takeaway_input`.
 
 ## Running the pipeline
 
@@ -155,8 +157,8 @@ Run this before guessing a `--model` string.
 | `GD_EVAL_JUDGE_MODEL` | judge model, same as `--judge-model` |
 | `GD_EVAL_AGENT_ID` | which agent to drive, same as `--agent-id` |
 | `LANGFUSE_HOST`, `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` | `--langfuse`, `--langfuse-dataset` |
-| `GOODDATA_EVAL_CHAT_*` | SSE retry, backoff and timeout knobs |
-| `GOODDATA_EVAL_TRACE_LABELS` | `key=value,...` labels stamped on every observation of a chat conversation via W3C baggage; `model_version` also sets the observation `version`, so set it only for a one-model run: every `--model` in one process gets the same labels. Keys must match `[A-Za-z0-9_-]+`; others are skipped with a warning. Dashboard-summary runs are not labelled |
+| `GOODDATA_EVAL_CHAT_*` | retry and backoff knobs shared by the chat and takeaway clients (`core/_retry.py`), and the SSE turn and item timeouts |
+| `GOODDATA_EVAL_TRACE_LABELS` | `key=value,...` labels stamped on every observation of a chat conversation via W3C baggage; `model_version` also sets the observation `version`, so set it only for a one-model run: every `--model` in one process gets the same labels. Keys must match `[A-Za-z0-9_-]+`; others are skipped with a warning. Dashboard-summary and document-takeaway runs are not labelled |
 | `GOODDATA_EVAL_JOIN_GENAI_TRACE` | on unless set to `0`, `false`, `no`, `off` or empty. An agentic item's chat turns ask gen-ai to open them under gd-eval's experiment root, so each item is one Langfuse trace with its scores written once. A gen-ai that drops caller baggage keeps today's separate traces; see the gotcha below |
 | `GD_EVAL_TIMERS` | same as `--timers` |
 
@@ -181,12 +183,15 @@ That default is correct: agent tool calls create real server-side objects (metri
 Adding a kind to `PARALLEL_SAFE_TEST_KINDS` is a deliberate assertion that it is read-only,
 which nothing in the package can prove for you.
 
-**The SSE client's retry predicate is load-bearing.** `core/chat/sse_client.py` retries
-429/502/503/504 and `httpx.RemoteProtocolError` (a mid-stream disconnect) with exponential
-backoff, and treats a `METADATA_SYNC_IN_PROGRESS` payload as transient. The
-`RemoteProtocolError` case was added after it was confirmed live to contaminate a small
-percentage of visualization runs with a hard fail and no retry. Narrowing that predicate
-reintroduces the problem.
+**The SSE client's retry predicate is load-bearing.** The backoff loop and the retryable
+status set (429/500/502/503/504) live in `core/_retry.py`, shared by the chat and takeaway
+clients and tuned by the same `GOODDATA_EVAL_CHAT_*` knobs. `core/chat/sse_client.py`
+retries that status set and `httpx.RemoteProtocolError` (a mid-stream disconnect), and
+treats a `METADATA_SYNC_IN_PROGRESS` payload as transient. The `RemoteProtocolError` case
+was added after it was confirmed live to contaminate a small percentage of visualization
+runs with a hard fail and no retry. Narrowing that predicate reintroduces the problem. The
+takeaway client retries the same status set plus connection failures, but not a read
+timeout: a slow page would spend the whole timeout again on every attempt.
 
 **Langfuse trace linking is deliberately off the item critical path.** Polling for trace
 ingestion has no pass/fail signal and inflates or misattributes per-item latency, so

packages/gooddata-eval/Makefile

@@ -1,2 +1,6 @@
 # (C) 2026 GoodData Corporation
 include ../../project_common.mk
+
+.PHONY: calibrate-takeaway
+calibrate-takeaway:
+	uv run --package gooddata-eval --extra llm-judge python scripts/calibrate_takeaway_judge.py

packages/gooddata-eval/README.md

@@ -132,8 +132,8 @@ gd-eval run \
 | `--runs K` | `2` | Independent runs per item. |
 | `--gate` | `any` | Which verdict decides an item: `any` = pass@K (a run passing is enough), `power` = pass^K (every run must pass, so the verdict measures stability). Identical at `--runs 1`. Kinds that repeat K runs only — `power` is refused when the dataset also has non-agentic items or `agentic_conversation`, which are always decided on pass@K. |
 | `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests — see *Concurrency and workspace safety* below. |
-| `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail` and `dashboard_summary`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
-| `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` items go through the summary endpoint, which has no such option. |
+| `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail`, `dashboard_summary` and `document_takeaway`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
+| `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` and `document_takeaway` items go through their own endpoints, which have no such option. |
 
 **Concurrency and workspace safety.** Agentic kinds that create workspace objects
 (`agentic_metric_skill`, `agentic_alert_skill`, `agentic_conversation`, `agentic_kda_skill`) always run one at a
@@ -707,7 +707,7 @@ A dataset is a folder of `.json` files, one per question:
 ```
 
 Supported `test_kind` values: `visualization`, `metric_skill`, `alert_skill`,
-`search_tool`, `general_question`, `guardrail`, `dashboard_summary`.
+`search_tool`, `general_question`, `guardrail`, `dashboard_summary`, `document_takeaway`.
 
 ### `dashboard_summary` items
 
@@ -746,14 +746,73 @@ The `expected_output` rubric:
 Each criterion is scored independently by the LLM judge, so `quality_score`
 is the fraction of satisfied criteria.
 
+### `document_takeaway` items
+
+Takeaway items call the AI Publisher takeaway endpoint
+(`POST /api/v1/ai/workspaces/{ws}/takeaway`) with a `takeaway_input` block, authored in
+snake_case and sent as the endpoint's camelCase body. In a Langfuse dataset the block goes
+in the item's `metadata` (or its `input` object), and `input` can stay a plain case label;
+an item carrying `takeaway_input` is a `document_takeaway` item unless it declares another
+`test_kind`. The endpoint needs the `enableGenAiReportTakeaways` feature flag.
+
+```json
+{
+  "id": "takeaway-001",
+  "dataset_name": "agent_document_takeaway",
+  "test_kind": "document_takeaway",
+  "question": "P01 default format, Sales page, no instructions",
+  "takeaway_input": {
+    "visualizations": ["<visualization id>", {"visualization_id": "<id>", "filters": []}],
+    "filter_context": [],
+    "instructions": {"document": null, "slot": null, "format_hint": null},
+    "max_characters": 300
+  },
+  "expected_output": {
+    "no_data": false,
+    "format": {"no_lists": false},
+    "must_include": ["States one finding that connects at least two of the charts"],
+    "must_not_include": [],
+    "rubric": []
+  }
+}
+```
+
+`takeaway_input` requires `visualizations`; unknown keys are rejected. Every
+`expected_output` key is optional:
+
+- `no_data` — whether the endpoint must answer `noData`; when either side says no data,
+  nothing else is graded.
+- `format` — overrides of the deterministic checks, which gate pass/fail: `bold_headline`,
+  `max_headline_words` (10), `max_sentences` (2), `max_words` (50), `single_paragraph`,
+  `no_lists`, `no_headings`. `false` or `null` switches a check off; unknown keys are
+  rejected. `length` (`max_characters`), `no_ids` and `no_meta` always apply.
+- `default_rules` — `false` drops the default judge criteria (one finding; no executive
+  summary, chart walk-through, background opener, figures, dates, causes the data cannot
+  show, chart description).
+- `must_include`, `must_not_include`, `rubric` — the item's own judge criteria, graded like
+  `dashboard_summary`.
+
+`word_count` and `headline_word_count` are reported in the item detail. The default judge
+criteria are calibrated with `make calibrate-takeaway` (needs `OPENAI_API_KEY`); re-run it
+after changing their wording.
+
+The client retries 429/5xx answers and connection failures. A request error that survives
+the retries is handled differently by the two entry points: `gd-eval run` aborts the item
+with that error, whatever `--runs` is, while the CI entry point
+(`evaluate_document_takeaway`) records it as an errored run, keeps the item's other runs,
+and raises only when no run completed. A read timeout is not retried, and the CI entry
+point records it as an errored run too. Any other 4xx and an unsupported URL scheme are
+never retried and fail the item in both.
+
 ### Items asked in a dashboard context (`user_context`)
 
 An item can be asked the way a user asks from an open dashboard or an attached widget.
 Its `user_context` is sent verbatim as `userContext` on every chat message of the item,
 follow-up and clarification messages included — the server treats a message without one
 as a cleared context. Requires the `enableAiContextSetup` feature flag on the target
 organization — without it the server ignores the dashboard view. Applies to chat items
-only; `dashboard_summary` items carry their scope in `summary_input` instead. No
+only; `dashboard_summary` and `document_takeaway` items carry their scope in
+`summary_input` / `takeaway_input` instead. No
 `userContext` key is sent at all when neither the item nor, for `agentic_conversation`,
 the current turn sets one.
 
@@ -806,6 +865,7 @@ from that turn on until another turn changes it, as the attached context does in
 | `general_question` | Text answer judged by LLM | `[llm-judge]` |
 | `guardrail` | Refusal/redirect (visualization response auto-fails) | `[llm-judge]` |
 | `dashboard_summary` | Dashboard summary (via `/summary` endpoint) scored against a rubric by LLM | `[llm-judge]` |
+| `document_takeaway` | One short takeaway for a page (via `/takeaway` endpoint): deterministic format checks plus a rubric scored by LLM | `[llm-judge]` |
 
 ## Optional extras
 

packages/gooddata-eval/scripts/calibrate_takeaway_judge.py

@@ -0,0 +1,75 @@
+# (C) 2026 GoodData Corporation
+"""Does the real judge put each calibration text on the right side of every default criterion?
+
+tests/test_takeaway_evaluator.py pins the wiring and the criterion wording; it cannot pin
+the verdict, because the verdict is an LLM's. This does, against the texts in
+tests/fixtures/takeaway_calibration.json:
+
+- a "pass" text must satisfy every default judge criterion on every run;
+- a "fail" text must violate each criterion it names on every run.
+
+Run it after changing any DEFAULT_MUST_INCLUDE / DEFAULT_MUST_NOT_INCLUDE wording:
+
+    OPENAI_API_KEY=... make calibrate-takeaway
+
+It is a script rather than a test because CI has no OPENAI_API_KEY. Exits 1 on any mismatch.
+"""
+
+import json
+import os
+import sys
+from pathlib import Path
+
+from gooddata_eval.core.evaluators.takeaway import (
+    DEFAULT_MUST_INCLUDE,
+    DEFAULT_MUST_NOT_INCLUDE,
+    DocumentTakeawayEvaluator,
+)
+from gooddata_eval.core.models import ChatResult, DatasetItem, TakeawayInput
+
+RUNS = 3
+FIXTURE = Path(__file__).resolve().parent.parent / "tests" / "fixtures" / "takeaway_calibration.json"
+
+CRITERIA = {f"default_include_{k}": k for k in DEFAULT_MUST_INCLUDE} | {
+    f"default_exclude_{k}": k for k in DEFAULT_MUST_NOT_INCLUDE
+}
+
+
+def main() -> int:
+    if not os.environ.get("OPENAI_API_KEY"):
+        raise SystemExit("OPENAI_API_KEY required")
+    fixture = json.loads(FIXTURE.read_text())
+    evaluator = DocumentTakeawayEvaluator()
+    print(f"judge model: {evaluator._positive_judge.model}, runs: {RUNS}")
+    mismatches = 0
+    for case in fixture["texts"]:
+        item = DatasetItem(
+            id=case["label"],
+            dataset_name="takeaway_calibration",
+            test_kind="document_takeaway",
+            question=fixture["question"],
+            expected_output={},
+            takeaway_input=TakeawayInput(visualizations=["calibration"]),
+        )
+        chat = ChatResult(text_response=case["text"], raw_response={"takeaway": case["text"], "noData": False})
+        satisfied = dict.fromkeys(CRITERIA.values(), 0)
+        for _ in range(RUNS):
+            detail = evaluator.evaluate(item, chat).detail
+            for key, name in CRITERIA.items():
+                satisfied[name] += detail.get(key) is True
+        want_pass = case["want"] == "pass"
+        must = dict.fromkeys(satisfied, RUNS) if want_pass else dict.fromkeys(case["criteria"], 0)
+        print(f"\n{case['label']} (want {case['want']})")
+        for name, count in satisfied.items():
+            flag = ""
+            if name in must:
+                ok = count == must[name]
+                mismatches += not ok
+                flag = "OK" if ok else "MISMATCH"
+            print(f"  {name:20} satisfied {count}/{RUNS}  {flag}")
+    print(f"\n{'all calibration texts on the right side' if not mismatches else f'{mismatches} mismatch(es)'}")
+    return 1 if mismatches else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

packages/gooddata-eval/src/gooddata_eval/cli/main.py

@@ -7,7 +7,7 @@
 import threading
 from datetime import datetime, timezone
 from pathlib import Path
-from typing import get_args
+from typing import Any, get_args
 
 import httpx
 from gooddata_api_client.exceptions import ApiException, ApiTypeError
@@ -36,32 +36,31 @@
 from gooddata_eval.core.reporting.json_report import build_multi_model_report, write_multi_model_report
 from gooddata_eval.core.runner import ItemReport, run_items
 from gooddata_eval.core.summary.http_client import SummaryClient
+from gooddata_eval.core.takeaway.http_client import TakeawayClient
 from gooddata_eval.core.timing import TIMERS_ENV_VAR
 from gooddata_eval.core.workspace import ModelResolutionError, WorkspaceModelController
 
 _EXIT_OK = 0
 _EXIT_OPERATIONAL_ERROR = 2
-_SUMMARY_TEST_KIND = "dashboard_summary"
 
 
 class _RoutingBackend:
     """Dispatch each item to the right backend by test_kind.
 
-    `dashboard_summary` items go to the dedicated summary endpoint; everything
-    else uses the conversational chat endpoint.
+    Kinds in `by_kind` go to their dedicated endpoint client (`dashboard_summary` to the
+    summary endpoint, `document_takeaway` to the takeaway endpoint); everything else uses
+    the conversational chat endpoint.
     """
 
-    def __init__(self, chat: ChatClient, summary: SummaryClient):
+    def __init__(self, chat: ChatClient, by_kind: dict[str, Any]):
         self._chat = chat
-        self._summary = summary
+        self._by_kind = by_kind
 
     def ask(self, item: DatasetItem) -> ChatResult:
-        if item.test_kind == _SUMMARY_TEST_KIND:
-            return self._summary.ask(item)
-        return self._chat.ask(item)
+        return self._by_kind.get(item.test_kind, self._chat).ask(item)
 
     def close(self) -> None:
-        for backend in (self._chat, self._summary):
+        for backend in (self._chat, *self._by_kind.values()):
             if hasattr(backend, "close"):
                 backend.close()
 
@@ -614,7 +613,14 @@ def on_langfuse_item_done(
                     turn_timeout_s=config.turn_timeout_s,
                     item_timeout_s=config.item_timeout_s,
                 ),
-                SummaryClient(host=config.host, token=config.token, workspace_id=config.workspace_id),
+                {
+                    "dashboard_summary": SummaryClient(
+                        host=config.host, token=config.token, workspace_id=config.workspace_id
+                    ),
+                    "document_takeaway": TakeawayClient(
+                        host=config.host, token=config.token, workspace_id=config.workspace_id
+                    ),
+                },
             )
             try:
                 single_report = run_items(