The hidden mismatch: Evaluating coding agents on the work developers actually do

Current public benchmarks for AI coding agents fail to represent how developers actually use them in practice. An analysis of internal developer traffic reveals that most interactions are conversational, involving clarifying questions, analysis, and reviews, rather than pure code generation. Existing benchmarks over-index on complex, single-turn code generation tasks and miss the messy, context-rich reality of multi-turn developer sessions. To address this gap, a new benchmark was created that mirrors real-world usage by setting task quotas based on traffic analysis, incorporating internal code, and simulating mid-session interactions. This approach aims to provide a more accurate evaluation of an agent's performance on practical, everyday coding tasks.