references/dataset_schema.md
GitHubEvaluation Dataset Schema
Canonical formats for evaluation datasets in the Agent Platform Evaluation
SDK. The summary below covers the type tree as of the version this skill
targets — for the live, authoritative definitions see the public SDK source:
types/evals.py
and types/common.py.
Core Types
EvaluationDataset
└── eval_cases: list[EvalCase] # List of evaluation cases
EvalCase
├── prompt: Content # Single-turn: the user query
├── responses: list[ResponseCandidate] # Single-turn: model response(s); list to support multi-candidate eval
├── reference: ResponseCandidate # Ground truth (for reference-based metrics)
├── agent_data: AgentData # Multi-turn: full conversation trajectory
├── rubric_groups: dict[str, RubricGroup] # Per-case rubrics; key is referenced from LLMMetric.rubric_group_name
└── (extra fields allowed) # Custom fields for custom metrics
ResponseCandidate
└── response: Content # The actual Content (role + parts)
AgentData
├── agents: dict[str, AgentConfig] # Agent definitions
└── turns: list[ConversationTurn] # Ordered conversation turns
ConversationTurn
├── turn_index: int # 0-based turn number
└── events: list[AgentEvent] # Events within this turn
AgentEvent
├── author: str # "user", agent_id, or "tool"
└── content: Content # Content with role and parts
Note on
responsesandreference. Both wrap aContentinside aResponseCandidateobject. So a single-turn case writes"responses": [{"response": {"role": "model", "parts": [...]}}]and"reference": {"response": {"role": "model", "parts": [...]}}— NOT a bareContent.promptandagent_data.turns[].events[].contentare bareContent(not wrapped).
Single-Turn Dataset
For simple prompt-response evaluation (e.g., QA, summarization).
{
"eval_cases": [
{
"eval_case_id": "capital_of_france",
"prompt": {
"role": "user",
"parts": [{"text": "What is the capital of France?"}]
},
"responses": [
{
"response": {
"role": "model",
"parts": [{"text": "The capital of France is Paris."}]
}
}
],
"reference": {
"response": {
"role": "model",
"parts": [{"text": "Paris"}]
}
}
},
{
"eval_case_id": "summarize_article",
"prompt": {
"role": "user",
"parts": [{"text": "Summarize this article: ..."}]
},
"responses": [
{
"response": {
"role": "model",
"parts": [{"text": "The article discusses..."}]
}
}
]
}
]
}
Required fields by metric type
| Metric category | Required fields |
|---|---|
| Predefined (single-turn) | prompt, responses |
| Computation-based | responses, reference |
| Translation | prompt (source), responses, reference |
| Custom LLM/code | Fields referenced in your template/function |
Multi-Turn / Multi-Agent Dataset
For evaluating multi-turn agent conversations, including systems with
multiple collaborating agents and tool calls. The agents map declares
all participating agents; turns is the chronological conversation,
where each event author is "user", an agent ID from the agents
map, or "tool".
{
"eval_cases": [
{
"eval_case_id": "flight_booking_via_specialist",
"agent_data": {
"agents": {
"router": {
"agent_id": "router",
"agent_type": "RouterAgent",
"instruction": "Route requests to the appropriate specialist."
},
"flight_bot": {
"agent_id": "flight_bot",
"agent_type": "SpecialistAgent",
"instruction": "Search and book flights.",
"tools": [{
"function_declarations": [{
"name": "search_flights",
"description": "Search flights by destination",
"parameters": {
"type": "OBJECT",
"properties": {
"destination": {"type": "STRING"}
}
}
}]
}]
}
},
"turns": [
{
"turn_index": 0,
"events": [
{
"author": "user",
"content": {
"parts": [{"text": "Book a flight to NYC"}]
}
},
{
"author": "router",
"content": {
"parts": [{"text": "Routing to flight_bot."}]
}
}
]
},
{
"turn_index": 1,
"events": [
{
"author": "flight_bot",
"content": {
"parts": [{
"function_call": {
"name": "search_flights",
"args": {"destination": "NYC"}
}
}]
}
},
{
"author": "flight_bot",
"content": {
"parts": [{
"function_response": {
"name": "search_flights",
"response": {"flights": [{"id": "AA123", "price": 320}]}
}
}]
}
},
{
"author": "flight_bot",
"content": {
"parts": [{"text": "Found AA123 to NYC for $320."}]
}
}
]
}
]
}
}
]
}
For a single-agent multi-turn case, omit the extra agent definitions
and use one entry in agents.
Per-Case Rubrics (rubric_groups)
EvalCase.rubric_groups is a dict of named rubric groups attached to a specific eval case. Use this when you want a rubric-based LLMMetric to evaluate against case-specific criteria rather than a globally-defined rubric. The metric references a group by name via LLMMetric.rubric_group_name; the name on the metric must match a key under rubric_groups in the matching EvalCase.
{
"eval_cases": [
{
"eval_case_id": "booking_confirmation",
"prompt": {"role": "user", "parts": [{"text": "Book my flight to Paris."}]},
"rubric_groups": {
"booking_rubrics": {
"rubrics": [
{"rubric_id": "confirmation_check", "content": {"property": {"description": "The model must confirm the booking and provide a reference number."}}},
{"rubric_id": "no_speculation", "content": {"property": {"description": "The response must not speculate about prices or availability beyond what tools returned."}}}
]
}
}
}
]
}
In eval_config.yaml, an LLMMetric then references the group by name:
custom_metrics:
- name: booking_quality
rubric_group_name: booking_rubrics # must match the key above
prompt_template: |
...
If rubric_group_name is omitted on the metric, the metric runs without per-case rubrics. If the name is set but doesn't match any key in the case's rubric_groups, the rubrics for that case are simply not applied (no error).
Common Mistakes
| Mistake | Fix |
|---|---|
Using role="assistant" |
Use role="model" (Vertex convention) |
Missing turn_index |
Always set sequential 0-based indices |
Tool response without function_response |
Wrap in a function_response part |
Using prompt field for multi-turn |
Use agent_data with the full trajectory |
Mixing prompt and agent_data in one case |
Use one or the other per EvalCase |