You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
gooddata-eval: read set_skills' result rather than its arguments to determine active skills #1779
Split out of #1762 discussion (agreed there as separable).
_set_skills_declarations() / _final_skill_declaration() in packages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.py derive the active skill set from the set_skills tool call's arguments — i.e. what the agent asked for. The same is true of _check_visualization_skill_activated in core/evaluators/visualization.py.
That is not always what became active. The service:
drops requested names it does not recognise, and
pulls in skills declared as dependencies of the ones requested.
The tool's own result echoes back the authoritative post-replacement set, so reading the result would be strictly truer to what actually became active than re-deriving it from the request.
Consequence today: a fixture requesting a misspelled or retired skill name is credited with activating it, and a skill activated only as a dependency is not credited at all — so both skill_routing and full_skill_coverage can be wrong in either direction.
Left out of #1762 because it is a behaviour change to how activation is measured, independent of the replace-semantics fix that PR is about, and it needs its own fixtures.
Scope:
Parse the set_skills tool result and prefer it over the arguments, falling back to arguments when the result is absent or unparseable (older captured traces, and any trace where the call errored).
Same treatment in core/evaluators/visualization.py.
Tests covering: unrecognised name requested, dependency pulled in implicitly, and result missing (fallback path).
Activity