[romrak]

The Publisher copilot evaluation now reads and scores v2 documents. v1 documents score exactly as before. The rerun of the evaluation on v2 belongs to LX-3342.

LX-3336, epic GDP-3491.

What this implements

The document-skill evaluator (core/agentic/document_skill.py; report_skill.py is only the deprecated re-export) recognises a document with top-level version: "2" and walks the v2 model:

{
  "id": "sales_overview", "type": "document", "version": "2", "title": "Sales overview",
  "period": {"type": "absolute", "granularity": "GDC.time.date", "from": "2026-01-01", "to": "2026-06-30"},
  "content": {
    "sections": [{"id": "section_1", "pages": [
      {"id": "page1", "kind": "cover", "layout": {"type": "column", "children": [{"type": "title", "id": "t", "markdown": "{documentName}"}]}},
      {"id": "page2", "kind": "content", "visible": true, "layout": {"type": "column", "children": [
        {"type": "title", "id": "t", "markdown": "Revenue"},
        {"type": "row", "children": [{"type": "visualization", "id": "widget1", "visualization": {"identifier": "revenue_trend", "type": "insight"}}]},
        {"type": "takeaway", "id": "takeaway", "markdown": "Revenue grew 12 % ..."}
      ]}}
    ]}]
  }
}

The text the judge reads is the same for a v1 and a v2 version of the same document:

Document: Sales overview
Period: 2026-01-01 to 2026-06-30

Page 2: Revenue
Charts: revenue_trend
Summary: Revenue grew 12 % ...

• Pages come from content.pages, or from content.sections[*].pages in order.
• The layout walk follows the v2 children lists and the v1 row/column lists.
• A visualization id is read from a bare string or from {identifier}.
• title and heading widgets give the page heading.
• takeaway widgets are the page summaries.

Decisions

1. v2 is detected by top-level version == "2" (a string), not by the part's format.
   gen-ai still sends format: "aac-v1", and the spec puts version on the entity.

• A v2 document with an integer version: 2 is read as v1 and fails every check.

2. Hidden pages (visible: false) are excluded from page_count and from every other check.
   page_count is what the user is told ("a document with 3 pages"), and a hidden page is out of the viewer, the exports and the page numbering.

• The part's and the tool's page_count must equal the number of visible pages.
• Hidden pages are also left out of the cover/content rule, the chart check, the summary check and the page numbers the judge reads.

3. A relative period always fails document_period_correct.
   Fixtures state fixed start/end dates, and a relative range cannot be compared with them. The failure shows the range, e.g. the relative range GDC.time.month -6 to -1.

• A copilot that rightly picks "last 6 months" as a relative period is marked wrong. Scoring that needs a fixture shape for relative expectations, which is follow-up work.

4. In v2, only takeaway widgets count as summaries, and every one on a content page must be written.
   The copilot writes generated text into takeaways. The default page templates ship an empty {"type": "text", "id": "summary"} widget, which is static text.

• A text widget with id summary is ignored: it is not counted, it does not fail as unwritten, and it is not shown to the judge.
• A page with an empty takeaway fails, even if another takeaway on the page is written.

5. Content is read from content, falling back to the top level, and sections win over pages if both are present.
   Only content is in the spec. The fallback is lenient, and metadata-api rejects both pages and sections together anyway.
6. The part's document must still carry type: "document".

• A v2 part carrying the REST attributes form, which has no type, fails document_part_present.

What comes next

• LX-3342: gen-ai drafts v2 documents and the evaluation reruns on v2. Its part and tool result must match decisions 1, 2, 4 and 6. The v2 part carries its own format (e.g. aac-v2); detection does not depend on it.
• LX-3244: removes the v1 parsing.

Test plan

• make -C packages/gooddata-eval format lint type-check: clean.
• TEST_ENVS=py314 make -C packages/gooddata-eval test: 1757 passed, 1 skipped.
• The v1 tests are parametrized over v1 and v2 with unchanged v1 inputs. origin/master's 81 v1-only document tests pass against the new code.
• The new v2 tests fail on the unmodified code with assertion failures (not import errors), and the key ones go red under sabotage.

risk: nonprod

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Document evaluation supports v1 and v2 documents, including section-based pages, nested layouts, and visualization references in either string or object form.
  • V2 takeaway widgets provide narrative content; headings and date ranges are read from the appropriate v2 fields.
• Bug Fixes
  • Hidden pages are excluded from page counts, reader-facing checks, visualization discovery, and evaluation text.
  • Page-count checks use visible pages, and date-range comparisons match the corresponding start and end bounds.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Essentials
• Run ID: dcbbdaef-1f84-4fa6-8719-3252693a2713

📥 Commits

Reviewing files that changed from the base of the PR and between 2962735 and 7fcae3a.

📒 Files selected for processing (2)

• packages/gooddata-eval/src/gooddata_eval/core/agentic/document_skill.py
• packages/gooddata-eval/tests/test_agentic_document_skill.py

Included review availability: This review used your included allowance. 3 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 5 reviews per hour.

📝 Walkthrough

Document evaluation now supports v1 and v2 document shapes for page retrieval, content extraction, judge rendering, and scoring. V2 handling includes section-based pages, hidden-page filtering, widget summaries, visualization references, and relative periods.

Changes: Document evaluation

Layer / File(s)	Summary
Page shapes and layout traversal packages/gooddata-eval/src/gooddata_eval/core/agentic/document_skill.py, packages/gooddata-eval/tests/test_agentic_document_skill.py	Page retrieval supports v1 and v2 containers, including sections. Layout traversal follows children, row, and column values. Fixtures and tests cover section ordering and page layouts.
Content extraction and judge rendering packages/gooddata-eval/src/gooddata_eval/core/agentic/document_skill.py, packages/gooddata-eval/tests/test_agentic_document_skill.py	Rendering handles v1 and v2 headings, visualization references, summary widgets, and period descriptions. Tests cover absolute and relative periods, summary widgets, and rendered document text.
Scoring and validation packages/gooddata-eval/src/gooddata_eval/core/agentic/document_skill.py, packages/gooddata-eval/tests/test_agentic_document_skill.py	Scoring validates page counts using visible pages, version-specific periods, visualizations, and narrative slots. Tests cover both document versions, including hidden-page behavior.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Merge Risk: ⚪ Minimal · up to 7fcae

The v2 evaluator appears ready to merge after normal checks. No actionable issue is established for the reviewed head.

Pre-merge checks · 4 · 1

Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 25.71% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 70 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: adding scoring support for v2 Publisher documents in gooddata-eval.

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{A rabbit checks each page in view,
Then finds the headings, charts, and text.
It counts the pages that readers see,
And checks the dates in both formats.
The v1 and v2 paths agree,
So off it hops, content in its nest.}

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.69%. Comparing base (69269ec) to head (7fcae3a).
⚠️ Report is 10 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1874      +/-   ##
==========================================
+ Coverage   84.65%   84.69%   +0.03%     
==========================================
  Files         335      335              
  Lines       23696    23745      +49     
==========================================
+ Hits        20061    20110      +49     
  Misses       3635     3635              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.