Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges

Details

Real conversations require more than understanding isolated prompts: users clarify, revise, interrupt, and introduce new context while expecting the system to remember what happened earlier. This survey maps the current state of multi-turn conversational AI across text-only dialogue, AudioLLMs, multimodal and omni-modal systems, and tool-augmented agents.\n\nThe central finding is a capability asymmetry: models have improved dramatically at perceiving and generating across modalities (text, audio, images, actions) but lag in maintaining coherent context across a session. The survey identifies specific persistent gaps: memory that doesn't span full conversations, cross-turn grounding failures when information must be connected across many turns, lack of full-duplex interaction (simultaneously speaking and listening), weak multi-turn evaluation benchmarks, and limited cultural alignment.\n\nEvaluation benchmarks are specifically flagged as inadequate — current benchmarks systematically under-measure multi-turn coherence, meaning published leaderboard scores overstate real conversational capability. For product teams, this survey provides a clear map of where current systems break down in production.\n\nThe survey covers datasets, modeling paradigms, training strategies, and evaluation setups in a unified framework, making it a useful reference document for researchers working across these overlapping subfields.