How to scale agentic evaluation: Lessons from 200,000 SWE-bench runs

Evaluating agentic systems at scale presents significant infrastructure challenges, as demonstrated by running over 200,000 evaluations on the SWE-bench benchmark. Traditional evaluation pipelines fail due to the stateful, long-running nature of agents, leading to bottlenecks in throughput, state collision between runs, and a lack of resumability. Initial attempts to adapt local evaluation code for a Kubernetes cloud environment were slow and unreliable, struggling with resource contention and the overhead of creating thousands of isolated container instances. The core problem lies in managing the massive compute and orchestration required to ensure each agent's run is isolated and reliable without compromising performance.