Is the #1 AI Model Really #1?

The ranking of AI models on leaderboards can be misleading due to statistical noise from the specific questions used in a benchmark. An analysis of eight models on a 2,000-question MMLU subset shows that the top two performers could not be statistically told apart, with their rankings depending heavily on the specific questions included. The stability of these rankings diminishes significantly with smaller test sets, as a 200-question test could easily yield a different winner compared to a 2,000-question one. The piece advocates for using paired statistical tests and considering performance in clusters rather than relying on precise ranks, especially when score differences are small.