Advancing AMIE towards expert-level audio-visual clinical consultations

Details

Google Research published a paper describing AMIE (Video), an extension of its medical AI system that conducts synchronous clinical video consultations. Unlike prior text-only versions, AMIE (Video) processes continuous audio-visual streams—observing gait, breathing, visible symptoms, and guiding physical examination maneuvers. Built on Gemini and Project Astra, it achieved expert-level performance in a randomized controlled study using simulated patient-actor consultations—a stronger evidence standard than benchmark-only evaluations.\n\nThis builds on substantial prior work: text-based AMIE demonstrated expert-level diagnostic dialogue; specialist-level evaluations were completed in oncology, cardiology, and ophthalmology; and real-world clinical studies are underway, including a feasibility study with Beth Israel Deaconess Medical Center and an ongoing nationwide trial with Included Health.\n\nThe shift to video addresses a fundamental gap: clinical diagnosis depends heavily on non-verbal cues (gait, breathing, skin) that patients struggle to describe in text. AMIE (Video) can now observe these directly and guide physical examinations—capabilities that significantly expand diagnostic accuracy. No regulatory clearance has been obtained; AMIE (Video) remains a research model not approved for clinical use.