Why MAP and MRR Fail for Search Ranking (and What to Use Instead)

Common search ranking evaluation metrics like Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR) have significant flaws that can be misleading. These metrics often fail because they treat relevance as a binary concept rather than a graded scale and do not accurately model user behavior, such as how attention decays with rank. For instance, MRR only considers the first relevant result, while MAP overemphasizes recall and applies a weak linear position penalty. Better alternatives include Normalized Discounted Cumulative Gain (NDCG) and Expected Reciprocal Rank (ERR), which incorporate graded relevance and apply more realistic positional discounts to better reflect search quality.