Agent Evaluation Readiness Checklist

This practical checklist outlines how to evaluate AI agents, starting with foundational steps like manual trace review and defining clear success criteria. It distinguishes between capability evals for new features and regression evals to prevent backsliding, emphasizing the importance of error analysis before building infrastructure. The guide details different evaluation levels (single-step, full-turn, multi-turn), recommending teams start with full-turn (trace-level) assessments. Finally, it provides in-depth guidance on dataset construction, covering sourcing examples, testing positive and negative cases, and tailoring data to specific agent types.