Rerankers Aren’t Magic Either: When the Cross-Encoder Layer Is Worth the Cost

Adding a reranker to a Retrieval-Augmented Generation (RAG) system is not the magic solution for fixing poor retrieval that many teams expect. While rerankers, known as cross-encoders, can analyze query and document interactions more deeply than standard embeddings, they often fail to solve fundamental problems like negation and introduce significant latency. The conventional "funnel" architecture uses these models to narrow down a large pool of candidates, but this adds complexity that can be difficult to audit. Empirical tests reveal that a powerful, modern embedding model alone can sometimes outperform a weaker one paired with a reranker, challenging this common architectural pattern.