If your team uses an AI system for a recurring task—such as support-ticket classification, document extraction, meeting summaries, or proposal drafting—occasional manual reviews may not be enough on their own to reveal quality regressions. This article’s recommendation is to create a versioned evaluation set that represents real work and run it consistently when changes are compared. Anthropic describes an evaluation as inputs paired with grading logic and distinguishes tasks, trials, graders, and complete transcripts or trajectories, especially for multi-step workflows in its explanation of evaluations.
Start with a small seed of real work
Begin with approximately 20–50 simple tasks drawn from real work, giving particular attention to previous failures, and expand the suite as new failure patterns appear as recommended in Anthropic’s starting guidance. This is not a claim that the size is sufficient for every task or domain; it is a practical starting point that a team can adjust.
Apply data-protection controls before collecting examples. Minimize the data to what the evaluation needs, remove or redact personal information, define access permissions, set a retention and deletion policy, and review privacy or compliance requirements when storing full text or tool calls. Redaction alone should not be treated as a guarantee of complete anonymization; Google’s People + AI Guidebook warns about this limitation in its guidance on data collection and evaluation.
Ask a domain expert to help define the desired outcome before labels are created. OpenAI presents a workflow that specifies success, measures performance, and improves the system, with domain experts helping define desired outcomes in its discussion of evaluation workflows. Then collect acceptable and unacceptable examples, recording the reason for each judgment rather than only a general score.
What should each test case contain?
The schema below is an editorial composite proposal, not a standard schema published by one source. It draws generally on guidance to document test sets, metrics, tools, and evaluation methods; on representative data, maintenance, and labeler disagreement; and on the distinction between inputs, grading logic, and execution trajectories described by NIST, described by Google, and described by Anthropic.
| Field | Suggested purpose |
|---|---|
| case_id | A stable identifier for the case. |
| task_version | The task definition version being evaluated. |
| input and context | The information needed to rerun the test. |
| expected_behavior | The expected behavior or acceptable output properties. |
| reference_answer_or_assertions | A reference answer or assertions that can be checked. |
| risk_tags and slice_tags | Risk labels and analysis slices, such as language or task type. |
| grader_type and grader_rubric | The grader type and the rules it uses. |
| provenance | Where the case came from and how it was labeled. |
These fields are not a binding published specification; they are this article’s practical template. Also record the data version, labeling rules, evaluator version, and known limitations. NIST recommends documenting test sets, metrics, tools, and evaluation methods; involving domain experts or independent assessors; and testing systems before deployment and during operation in the Measure section of the AI RMF.
Match the grader to the definition of success
For deterministic requirements, use code-based checks such as exact matches, structured-output validation, regex checks, unit tests, or outcome verification. These are implementation examples proposed by this article, not a claim that one source requires every one of them.
For relevance, tone, completeness, and factual grounding, a rubric-based model grader may be useful, with the team periodically reviewing a sample of its decisions. Anthropic describes combining code-based, model-based, and human graders for agent evaluations in its discussion of grader types. Do not treat a model-grader score as final truth: record disagreements and use human review to revise the rubric when needed. Google’s guide also emphasizes documenting labeler disagreement, maintaining the dataset, and continuing to test against real-world conditions in its data-collection and evaluation guidance.
For agentic or multi-step tasks, retain the full conversation transcript, tool calls, intermediate actions, and final state—not only the final answer consistent with Anthropic’s distinction between complete trajectories and final outputs. When outputs vary, run multiple trials and record the pass rate; one trial may not describe that variation.
Report results by slice
Do not rely only on one overall score. Report pass rates by language, customer segment, input length, difficulty, ambiguity, task type, and known edge cases. These slices are an analytical proposal from this article; a team can add slices tied to its risks or actual usage. The purpose is to show where performance changes, rather than produce one number that hides the pattern.
As an optional editorial safeguard against overfitting, the team may keep a small set that is not used for prompt or system development. This article does not attribute that suggestion to Anthropic. Use it later as an independent check under the team’s own policy, documenting who can access it and how it is updated.
Make evaluation part of the release cycle
As a practical release rule, run the set after meaningful changes that could affect behavior—such as changes to the prompt, model, retrieval, tools, or workflow—according to your team’s policy, then compare it with the previous version. This is a proposed operating rule, not a general requirement from OpenAI. OpenAI’s Evals API supports evaluation data sources, configurable graders, and repeated runs across models or parameters according to the OpenAI Evals API reference.
Set release thresholds before approving a change, and decide what should trigger an investigation or block a release. These thresholds and operating choices are editorial recommendations, not universal limits for every team. After launch, add confirmed production failures and user feedback to the set, and monitor for data drift and changing business requirements. This is consistent with the focus on dataset maintenance and ongoing testing under real-world conditions in Google’s guide and with NIST’s recommendation to measure before deployment and during operation in the AI RMF.
A short implementation checklist
- Assign a domain expert to define success.
- Collect 20–50 real cases, starting with known failures.
- Minimize sensitive data, redact it, and define access, retention, and deletion.
- Document expected behavior and acceptance criteria for each case.
- Add risk tags, slice tags, and provenance.
- Separate deterministic checks from qualitative judgments.
- Have a human audit a sample of model-grader decisions.
- Store the complete trajectory for multi-step tasks.
- Report results by slice, not only by the overall average.
- Compare versions, investigate regressions, and add confirmed production failures.
This approach turns an evaluation set into an accumulating regression-test record. The schema, thresholds, and run frequency remain editorial and operational decisions for your team, based on the task and its risks.