NepTTS-Bench: Native-Speaker Benchmarking and NepaliMOS for Nepali Text-to-Speech
| Source: arXiv AI
Tags: NepTTS-Bench, NepaliMOS, ElevenLabs, Piper, SCOREQ, text-to-speech
NepTTS-Bench, a Nepali text-to-speech benchmark rated by 180 native speakers, finds automated scoring favors proprietary ElevenLabs while native listeners prefer open-source Piper, because many systems carry Hindi phonology.
Details
Several Nepali TTS systems exist but there was no shared way to compare them. NepTTS-Bench offers 205 sentences, including 77 minimal pairs targeting aspiration, retroflex and dental contrasts, nasalization, gemination and schwa deletion. 180 native speakers gave 6,962 ratings across 10 TTS configurations and two human references. Seven ASR systems were also tested for round-trip character error rate, with SCOREQ as an automated baseline. SCOREQ preferred proprietary systems such as ElevenLabs, while native listeners preferred Piper. The authors attribute this to systems not tuned for Nepali showing Hindi phonological behavior. The NepaliMOS predictor reaches system-level rho of 0.90 but only 0.43 at utterance level on unseen systems, so human calibration is still recommended. A Python evaluation package is released.