WarpBench: How we reduced our cost-per-PR 75%

Warp utilized its internal tool, WarpBench, to create custom benchmarks for evaluating different AI coding agents on its own codebase and workflows. By testing a range of models including Opus 5, GPT-5 Sol, and Grok 4.6, the company aimed to find the optimal balance between performance and cost. The results showed that switching to a more cost-effective model reduced the internal cost-per-pull-request from $80 to $30. This change not only saved money but also increased the task compliance score, demonstrating the value of private benchmarks over public ones. The company plans to further refine this process with more specialized benchmarks and A/B testing for model routing.