How we build evals for Deep Agents

Effective agent evaluations are crucial for shaping agent behavior and should be designed to measure specific, desired outcomes. Instead of simply maximizing the number of tests, the focus is on creating targeted evals that reflect production needs. Data for these evaluations is sourced from internal dogfooding, adapting external benchmarks, and writing custom tests for key behaviors. Metrics extend beyond simple correctness to include efficiency measures like step ratio, tool call ratio, and solve rate, which compare an agent's performance to an ideal trajectory.