Comparative Analysis of Multilingual Pre-trained Models for Nepali Automatic Speech Recognition
| Source: arXiv AI
Tags: ASR, Nepali, Whisper, IndicWav2Vec, multilingual, speech recognition, low-resource languages
A controlled benchmark of six ASR models on 165 hours of Nepali finds Whisper-Large-v3-Turbo and IndicWav2Vec tie at the top (14.76% vs 14.89% WER) despite a 9x parameter gap — while CTC decoders run up to 29x faster at equivalent accuracy, flipping the deployment preference for any latency-constrained application.
Details
Nepali is nominally supported by multilingual ASR models, but no controlled comparison had been published before this work. Researchers from Tribhuvan University fine-tuned six models — XLSR-53, IndicWav2Vec, MMS-1B, Whisper-Medium, Whisper-Large-v3-Turbo, and Conformer-Hi — on the OpenSLR SLR54 Nepali corpus (~165 hours) using identical preprocessing, splits, optimizer, and family-matched learning-rate schedules. Results are reported across three independent test sets. The headline finding: Whisper-Large-v3-Turbo (14.76% WER) and IndicWav2Vec (14.89% WER) achieve comparable performance despite IndicWav2Vec having 9x fewer parameters and 40x less pretraining data. Language-family proximity in pretraining — IndicWav2Vec was trained on Indic languages including related language families — substitutes for raw scale for in-domain Nepali recognition. CTC-based decoders run up to 29x faster than autoregressive Whisper at comparable accuracy, representing a decisive latency advantage for production deployment. MMS-1B shows the smallest out-of-domain degradation on FLEURS (+12.55 percentage points), suggesting that massive multilingual scale buys cross-domain robustness rather than peak in-domain accuracy. Code and models are open-sourced. The benchmark provides the first standardized, multi-model, efficiency-aware reference numbers for Nepali ASR, directly useful for developers building Nepali voice applications.