Re-Sonance: A Dysarthric Asynchronous Real-Time Speech Conversion System Based on a Three-Stage Cascaded ASR-LLM-TTS Architecture
| Source: arXiv AI
Tags: speech, dysarthria, ASR, TTS, Whisper, accessibility, Qwen, CosyVoice
Re-Sonance chains Whisper ASR, Qwen LLM, and CosyVoice TTS to reconstruct natural-sounding speech from dysarthric input in real time, demonstrating improved intelligibility and naturalness for mild-to-moderate dysarthria in Mandarin while acknowledging limitations with severe cases.
Details
Dysarthria is a motor speech disorder that impairs pronunciation without affecting cognition, making professional speaking scenarios — conferences, presentations, meetings — particularly challenging. Existing AAC systems typically suffer from high latency and unnatural output. Re-Sonance takes a cascade approach with three off-the-shelf components: Whisper transcribes dysarthric speech, Qwen LLM normalizes and reconstructs the utterance while preserving semantic content, and CosyVoice synthesizes natural-sounding speech.\n\nBoth subjective and objective evaluations on a Mandarin dysarthric speech dataset show improved intelligibility and naturalness while maintaining semantic coherence for mild-to-moderate dysarthria. The LLM stage corrects transcription errors and improves fluency without changing meaning. The system accepted at NCMMSC 2025.\n\nLimitations are clearly stated: severe dysarthria cases remain challenging, and the evaluation is Mandarin-specific, limiting generalizability to other languages. The approach's strength is that no custom dysarthria-specific model training is required — the pipeline is the innovation, not the individual components.