RAG Retrieval Lost 40 Points When Distracting Passages Grew to 2.68M

An experiment tested the search component of Retrieval-Augmented Generation (RAG) systems at different scales. On a small collection of 2,000 research papers, both BM25 and embedding-based search maintained high recall, dropping by less than 4 percentage points as the collection grew. However, when the test was repeated on a much larger corpus of 2.68 million Wikipedia passages, retrieval performance degraded substantially. The study highlights the challenge of maintaining search effectiveness as the number of distractor documents grows, showing that evaluations on small datasets can be misleading. This analysis also considered different retrieval strategies like using flat chunks versus parent documents.