Water Cooler Small Talk, Ep. 11: Overfitting in RAG evaluation

Evaluating a Retrieval-Augmented Generation (RAG) system by repeatedly identifying and fixing issues on the same test set leads to overfitting. This process effectively turns the evaluation set into a training set, causing the model to memorize answers rather than learn to generalize. This phenomenon is analogous to overfitting in classical machine learning and is an example of Goodhart's Law, where a measure ceases to be useful once it becomes a target. To avoid this, practitioners must use a genuinely held-out test set that is touched as rarely as possible to get a true sense of the model's performance on unseen data.