The Open ASR Leaderboard Adds Its First Global South Language

Details

The Open ASR Leaderboard, maintained by Hugging Face, now includes Monsoon en-IN and Monsoon hi-IN — evaluation sets for Indian English and Hindi developed in partnership with Voice Arena. Hindi, spoken by over 500 million people, becomes the first Indic language on a leaderboard that previously covered only European languages. Existing ASR benchmarks collapse performance into a single Word Error Rate, masking systematic disparities. Prior research found commercial systems are roughly twice as bad for Black speakers compared to white speakers, with further gaps by gender, age, and accent. Monsoon addresses this by recording 12 speaker attributes per contributor and designing the corpus to vary across 9 axes: geography, age, gender, vocabulary, recording device, acoustic environment, speech type, speech rate, and transcription ambiguity. The 4,888-speaker dataset was built by recruiting across hundreds of districts — not just major cities — to achieve genuine geographic diversity. Each language includes a public split for self-scoring and a private split withheld to prevent benchmark gaming. Evaluation also includes benchmark-fitting analysis to detect when models reproduce training transcripts rather than actually transcribing audio. For practitioners building voice products for South Asian markets, this is the first open evaluation framework designed to expose failure modes specific to Hindi and Indian English speakers. The leaderboard now applies its WER infrastructure to a population segment underserved by virtually all public ASR benchmarks.