GH-ESD: Grounded Hypothesis-Driven Error Slice Discovery for Instance-Level Vision Tasks

Details

Apple ML Research has published GH-ESD (Grounded Hypothesis-Driven Error Slice Discovery) at ECCV 2026, addressing a gap in vision model evaluation. Existing methods for finding 'error slices' — systematic failure patterns — rely on clustering representations or predefined attributes, which works for image classification but breaks down for instance-level tasks like object detection and segmentation, where failures often arise from complex spatial and contextual relationships. GH-ESD reframes the problem as a generate-and-verify loop: large language models produce relational failure hypotheses (e.g., a model failing on partially occluded small objects), Vision Language Models find matching instances in the data, and statistical analysis confirms whether those instances are genuinely associated with higher error rates. The approach produces interpretable, actionable failure descriptions — not just opaque cluster labels that engineers must manually decode. To benchmark the method, the authors introduce GESD (Grounded Error Slice Dataset), providing expert-defined and spatially grounded failure slices from detection and segmentation tasks. On GESD, GH-ESD achieves Precision@10 of 0.73 versus 0.63 for baselines on detection — a 0.10 improvement. Segmentation scenarios are also supported. For teams deploying vision models in production (autonomous driving, medical imaging, retail), the ability to systematically surface and characterize failure modes without manual inspection is directly useful for model debugging and targeted data collection.