Language Family Matters: Evaluating LLM-Based ASR Across Linguistic Boundaries
| Source: arXiv AI
Tags: ASR, multilingual, speech recognition, LLM, EACL
Training one ASR connector per language family rather than per language reduces parameter count while improving cross-domain generalization in LLM-based speech recognition — validated on two multilingual LLMs and two speech corpora, accepted at EACL 2026.
Details
LLM-based automatic speech recognition links a frozen speech encoder to a pretrained LLM via a trainable connector. Current practice trains a separate connector per language, ignoring structural similarities between related languages in the same family.\n\nThis EACL 2026 paper proposes a connector-sharing strategy based on linguistic family membership — one connector per family (e.g., Romance, Germanic) rather than per individual language. The approach reduces total parameter count proportionally to how many languages share a family while improving generalization across speech domains.\n\nExperiments cover two multilingual LLMs and two real-world corpora: one curated, one crowd-sourced. Family-based connectors consistently improve cross-domain generalization compared to per-language connectors, suggesting that linguistic relatedness encodes shared acoustic-linguistic patterns useful for domain transfer. The result offers a practical and scalable multilingual ASR strategy for teams building cross-lingual voice products.