Using LLM-as-a-judge scoring to measure your software factory

Using Large Language Models as judges provides a direct method for scoring and measuring the performance of AI coding agents within a software factory. This approach goes beyond traditional DORA metrics by examining a complete digital record of an agent's work, including all inputs and outputs. The process involves building a record of agent traces, defining 'scoring agents' to grade performance on dimensions like efficiency and code quality, and automating the evaluation. Ultimately, these scores can feed into a self-improvement loop where observer agents suggest changes to improve the overall system and benchmark different model configurations.