Introducing RTEB: A New Standard for Retrieval Evaluation

A new benchmark, the Retrieval Embedding Benchmark (RTEB), has been introduced to reliably evaluate the retrieval accuracy of embedding models for real-world applications. It addresses the shortcomings of existing benchmarks, where models can overfit to public test data, creating a gap between reported scores and real-world performance. RTEB uses a hybrid strategy of open and private datasets to provide a more accurate measure of a model's ability to generalize to unseen data. The benchmark is designed with a focus on real-world enterprise domains, is multilingual, and aims to create a fair, application-focused standard for evaluation.