One Capital Letter Was Silently Breaking My AI Support Bot, and It Wasn't in the New Model

An AI support bot designed to triage banking messages was silently failing due to a single capitalized letter in its JSON output. This issue was discovered not through overall accuracy metrics, which were improving, but through a detailed regression test comparing different OpenAI model versions. The project uses Weave to trace model calls, version prompts, and create a test dataset from real customer inquiries. This approach highlights how aggregate scores can mask critical formatting errors, demonstrating the need for strict, rule-based checkers alongside qualitative AI judges for robust evaluation.