Skip to content

fix(evaluation): skip invocations without user events to prevent ValidationError#5287

Open
Koushik-Salammagari wants to merge 1 commit intogoogle:mainfrom
Koushik-Salammagari:fix/eval-skip-agent-only-invocations
Open

fix(evaluation): skip invocations without user events to prevent ValidationError#5287
Koushik-Salammagari wants to merge 1 commit intogoogle:mainfrom
Koushik-Salammagari:fix/eval-skip-agent-only-invocations

Conversation

@Koushik-Salammagari
Copy link
Copy Markdown

Link to Issue or Description of Change

Closes #3760

Description

EvaluationGenerator.convert_events_to_eval_invocations iterates over events grouped by invocation_id. For invocations that contain no user-authored event (e.g. internal background turns, system-driven tasks where all events have author != "user"), the code previously left user_content as an empty string "" (versions ≤ 1.19) or an empty Content(parts=[]) (current main).

Invocation.user_content is typed as a required genai_types.Content. Passing an empty string caused an immediate pydantic.ValidationError; passing Content(parts=[]) silently creates a semantically incorrect eval case — an invocation with no user turn is not meaningful for evaluation.

The fix: initialise user_content = None and skip the invocation if no user event is found. A DEBUG log line is emitted so operators can diagnose unexpected skips without producing noisy warnings.

# Before
user_content = Content(parts=[])   # always appended, even with no user event
...
invocations.append(Invocation(user_content=user_content, ...))  # may crash or be wrong
# After
user_content = None
...
if user_content is None:
    logger.debug('Skipping invocation %s: no user-authored event found.', invocation_id)
    continue
invocations.append(Invocation(user_content=user_content, ...))

Changes

  • src/google/adk/evaluation/evaluation_generator.py: initialise user_content = None; skip invocations where no user event was found.
  • tests/unittests/evaluation/test_evaluation_generator.py: add two regression tests.

Testing Plan

  • test_invocation_without_user_event_is_skipped — single agent-only invocation returns empty list (previously raised ValidationError or produced incorrect data).
  • test_mixed_invocations_skips_only_agent_only_ones — verifies that valid user+agent invocations are retained and only agent-only ones are filtered out.
  • All 9 TestConvertEventsToEvalInvocation tests pass.

🤖 Generated with Claude Code

@google-cla
Copy link
Copy Markdown

google-cla bot commented Apr 11, 2026

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@adk-bot adk-bot added the eval [Component] This issue is related to evaluation label Apr 11, 2026
@adk-bot
Copy link
Copy Markdown
Collaborator

adk-bot commented Apr 11, 2026

Response from ADK Triaging Agent

Hello @Koushik-Salammagari, thank you for your contribution!

It looks like the Contributor License Agreement (CLA) has not been signed yet. Could you please sign it? The PR cannot be merged without it.

Thanks!

@Koushik-Salammagari
Copy link
Copy Markdown
Author

@googlebot I signed it!

2 similar comments
@Koushik-Salammagari
Copy link
Copy Markdown
Author

@googlebot I signed it!

@Koushik-Salammagari
Copy link
Copy Markdown
Author

@googlebot I signed it!

…ts_to_eval_invocations

Sessions can contain invocation_ids whose events are all authored by agents
or tools (e.g. internal/background turns with no corresponding user message).
Previously, convert_events_to_eval_invocations left user_content as an empty
Content(parts=[]) for such invocations, and earlier versions used an empty
string, which caused a Pydantic ValidationError because Invocation.user_content
requires a genai_types.Content object.

Invocations without a user-authored event are not meaningful for evaluation,
so skip them instead of constructing an Invocation with a placeholder
user_content. A debug log line is emitted for each skipped invocation to aid
troubleshooting.

Fixes google#3760
@Koushik-Salammagari Koushik-Salammagari force-pushed the fix/eval-skip-agent-only-invocations branch from 550a771 to bc8727d Compare April 11, 2026 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

eval [Component] This issue is related to evaluation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evaluation Invocation.user_content ValidationError when session has invocations without user events

2 participants