Does Agentic RAG Actually Retrieve Better Evidence? I Tested It on 298 Questions

An experiment tested whether agentic Retrieval-Augmented Generation (RAG) improves evidence retrieval over simpler methods. Four search designs were evaluated against 298 questions and 2,000 arXiv papers: plain search, a reranker, a question rewriter, and an agentic system. The agentic approach, which could perform iterative searches, did not produce a statistically significant improvement over the plain search baseline. While the agent found some papers that plain search missed, it also lost others that the simpler method had found. The most notable and cost-effective improvement came from a simple reranker model that reordered the initial search results.