LLM Evals Are Based on Vibes — I Built the Missing Layer That Decides What Ships

Standard evaluations for large language models are flawed because they rely on subjective "vibe checks" or single scores that fail to catch confident-sounding hallucinations. A more effective system splits the concept of faithfulness into two distinct metrics: attribution, to check if an answer is grounded in context, and specificity, to measure its detail. This separation is critical because the signature of a dangerous hallucination is high specificity combined with low attribution—a pattern a single score would miss. By implementing this logic in a lightweight decision engine, developers can automatically accept, flag, or reject responses with reproducible, explainable results, rather than just generating a number.