fix(evaluation): skip invocations without user events to prevent ValidationError#5287
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
|
Response from ADK Triaging Agent Hello @Koushik-Salammagari, thank you for your contribution! It looks like the Contributor License Agreement (CLA) has not been signed yet. Could you please sign it? The PR cannot be merged without it. Thanks! |
|
@googlebot I signed it! |
…ts_to_eval_invocations Sessions can contain invocation_ids whose events are all authored by agents or tools (e.g. internal/background turns with no corresponding user message). Previously, convert_events_to_eval_invocations left user_content as an empty Content(parts=[]) for such invocations, and earlier versions used an empty string, which caused a Pydantic ValidationError because Invocation.user_content requires a genai_types.Content object. Invocations without a user-authored event are not meaningful for evaluation, so skip them instead of constructing an Invocation with a placeholder user_content. A debug log line is emitted for each skipped invocation to aid troubleshooting. Fixes google#3760
550a771 to
bc8727d
Compare
|
@googlebot I signed it! |
|
Hi @Koushik-Salammagari , Thank you for your contribution! We appreciate you taking the time to submit this pull request. Please fix formatting errors by running autoformat.sh |
…entity feature PiperOrigin-RevId: 898786239
After an agent gets deployed to agent engine, include a link to try out the agent. Co-authored-by: Kathy Wu <wukathy@google.com> PiperOrigin-RevId: 899093067
Co-authored-by: Kathy Wu <wukathy@google.com> PiperOrigin-RevId: 899104727
Closes google#5110 Co-authored-by: George Weale <gweale@google.com> PiperOrigin-RevId: 899108631
5ba2830 to
4fea84d
Compare
|
Hi @rohityan — formatting has been fixed via |
Link to Issue or Description of Change
Closes #3760
Description
EvaluationGenerator.convert_events_to_eval_invocationsiterates over events grouped byinvocation_id. For invocations that contain no user-authored event (e.g. internal background turns, system-driven tasks where all events haveauthor != "user"), the code previously leftuser_contentas an emptyContent(parts=[]).Invocation.user_contentis typed as a requiredgenai_types.Content. PassingContent(parts=[])silently creates a semantically incorrect eval case — an invocation with no user turn is not meaningful for evaluation.The fix: initialise
user_content = Noneand skip the invocation if no user event is found. ADEBUGlog line is emitted so operators can diagnose unexpected skips without producing noisy warnings.Changes
src/google/adk/evaluation/evaluation_generator.py: initialiseuser_content = None; skip invocations where no user event was found.tests/unittests/evaluation/test_evaluation_generator.py: add two regression tests.Testing Plan
test_invocation_without_user_event_is_skipped— single agent-only invocation returns empty list (previously produced an incorrect eval case).test_mixed_invocations_skips_only_agent_only_ones— verifies that valid user+agent invocations are retained and only agent-only ones are filtered out.TestConvertEventsToEvalInvocationtests pass.