The Open ASR Leaderboard Adds Its First Global South Language

The Open ASR Leaderboard is expanding to include Hindi and Indian English, marking the first language from the Global South to be added. This initiative, a partnership between Voice Arena and Hugging Face, introduces two new evaluation datasets designed to address known biases in Automatic Speech Recognition (ASR). The datasets, named Monsoon, were specifically constructed to vary across nine axes, including geography, age, gender, and acoustic environments, to ensure models are evaluated on more diverse and realistic data. By including detailed speaker metadata and private test splits, the effort aims to create more trustworthy benchmarks and drive the development of ASR systems that perform equitably across different populations. The collection includes thousands of speakers from hundreds of districts across India to capture regional and orthographic variations.