Introducing Factory Benchmarks

Warp has launched Benchmarks for its Warp Factories platform, a feature for creating model benchmarks based on a user's own coding tasks. This system allows developers to test, measure, and optimize the performance of different AI model and skill configurations. Benchmarks are built from a set of agent tasks, various factory configurations, and "Scorers" that use an LLM-as-a-judge loop to evaluate runs on dimensions like cost and quality. The infrastructure replays tasks across a matrix of configurations and provides visualizations to identify the most effective setup for specific software development needs.