Towards Expert-level Medical AI for Real-time Video Consultations

Details

AMIE (Articulate Medical Intelligence Explorer) in its video configuration is a Gemini-based multi-agent system that integrates low-latency dialogue, clinical reasoning, and real-time audio-visual perception. The team built a taxonomy and automated evaluations specifically for clinical audio-visual cues in telehealth settings to guide development.\n\nThe evaluation used a randomized Objective Structured Clinical Examination (OSCE) with 30 primary care physicians, 15 patient actors, and 100 clinical scenarios — a rigorous study design. Clinical evaluators rated AMIE (Video) on par or better than PCPs in history-taking, diagnosis, management planning, and physical observation and examination. Patient actors preferred AMIE's approach to assessing and explaining conditions; PCPs were preferred for rapport and partnership building.\n\nIn modality ablation, patient actors preferred AMIE (Video) over text-only AMIE for communicative effectiveness, convenience, and feeling understood — demonstrating that audio-visual modality matters for patient experience, not just clinical accuracy.\n\nKnown limitations include fine anatomical precision, subtle affective nuances, and high-frequency movements. The authors appropriately note that further research is needed before real-world deployment. But reaching clinician-level performance across history-taking, diagnosis, and management in real-time video consultations is a landmark result for medical AI.