How to Do Evals on a Bloated RAG Pipeline

Evaluating a complex Retrieval-Augmented Generation (RAG) pipeline is crucial to determine if advanced techniques, like expanding text chunks to include neighboring content, actually improve answers or just add noise. The system's performance is pressure-tested using metrics such as faithfulness, answer relevancy, and hallucination rate across different datasets, including clean questions, messy queries, and random inquiries. An "LLM-as-a-judge" is used to score the outputs, specifically comparing how well answers are grounded in the initial retrieved text versus the final expanded context. This analysis reveals whether the model effectively uses the additional context from neighboring chunks to provide more accurate and complete responses.