What the Bits-over-Random Metric Changed in How I Think About RAG and Agents

The Bits-over-Random (BoR) metric provides a new perspective on evaluating Retrieval-Augmented Generation (RAG) and agentic systems. Traditional retrieval metrics like recall can be misleading, as increasing the number of retrieved documents can also introduce irrelevant noise, or "context pollution," which degrades Large Language Model performance. BoR measures whether a retrieval system is genuinely selective compared to random chance, helping to distinguish meaningful retrieval from simply stuffing the context window. This concept is especially critical for AI agents selecting from a library of tools, where providing too many options can lead to confusion and worse outcomes despite high recall. Ultimately, this reframes retrieval as a reasoning-budget allocation problem, prioritizing the quality and purity of the context provided to the model.