fix(evaluation): skip invocations without user events to prevent ValidationError#5287
Open
Koushik-Salammagari wants to merge 1 commit intogoogle:mainfrom
Open
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Collaborator
|
Response from ADK Triaging Agent Hello @Koushik-Salammagari, thank you for your contribution! It looks like the Contributor License Agreement (CLA) has not been signed yet. Could you please sign it? The PR cannot be merged without it. Thanks! |
Author
|
@googlebot I signed it! |
…ts_to_eval_invocations Sessions can contain invocation_ids whose events are all authored by agents or tools (e.g. internal/background turns with no corresponding user message). Previously, convert_events_to_eval_invocations left user_content as an empty Content(parts=[]) for such invocations, and earlier versions used an empty string, which caused a Pydantic ValidationError because Invocation.user_content requires a genai_types.Content object. Invocations without a user-authored event are not meaningful for evaluation, so skip them instead of constructing an Invocation with a placeholder user_content. A debug log line is emitted for each skipped invocation to aid troubleshooting. Fixes google#3760
550a771 to
bc8727d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to Issue or Description of Change
Closes #3760
Description
EvaluationGenerator.convert_events_to_eval_invocationsiterates over events grouped byinvocation_id. For invocations that contain no user-authored event (e.g. internal background turns, system-driven tasks where all events haveauthor != "user"), the code previously leftuser_contentas an empty string""(versions ≤ 1.19) or an emptyContent(parts=[])(currentmain).Invocation.user_contentis typed as a requiredgenai_types.Content. Passing an empty string caused an immediatepydantic.ValidationError; passingContent(parts=[])silently creates a semantically incorrect eval case — an invocation with no user turn is not meaningful for evaluation.The fix: initialise
user_content = Noneand skip the invocation if no user event is found. ADEBUGlog line is emitted so operators can diagnose unexpected skips without producing noisy warnings.Changes
src/google/adk/evaluation/evaluation_generator.py: initialiseuser_content = None; skip invocations where no user event was found.tests/unittests/evaluation/test_evaluation_generator.py: add two regression tests.Testing Plan
test_invocation_without_user_event_is_skipped— single agent-only invocation returns empty list (previously raisedValidationErroror produced incorrect data).test_mixed_invocations_skips_only_agent_only_ones— verifies that valid user+agent invocations are retained and only agent-only ones are filtered out.TestConvertEventsToEvalInvocationtests pass.🤖 Generated with Claude Code