Why health AI interfaces must adapt to user expertise

Details

Published in Nature Medicine and led by MIT professor Marzyeh Ghassemi, the study tested multiple explainability approaches on skin disease diagnosis. Non-experts improved accuracy with all explainability formats tested — heatmaps, similar-image retrieval, and LLM-generated explanations — mainly by deferring to the AI model. Primary care providers showed the opposite effect: they performed best with AI predictions but no explanation, and explanations degraded their accuracy, likely because they anchored on the AI output rather than applying clinical judgment. This creates a genuine design dilemma: transparency tools intended to build trust can paradoxically reduce accuracy when users defer too heavily to them. The paper also tested a fairness-constrained model designed to reduce diagnostic disparities for darker skin tones, which improved equity but introduced new automation bias risks for non-expert users. MIT's Ghassemi concluded that interface design must account for user expertise level, and that universal explainability mandates may not improve — and could harm — clinical performance.