DeepAmbigQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness
| Source: Apple ML Research
Tags: Apple, benchmarks, question answering, RAG, LLM evaluation, GPT-5, disambiguation
Apple researchers introduce DeepAmbigQA — 3,600 multi-hop questions where half require name disambiguation; even GPT-5 scores only 0.13 exact match on ambiguous questions, exposing a persistent gap in LLM answer completeness for knowledge-intensive tasks.
Details
DeepAmbigQA targets a specific failure mode in LLM-powered search: questions that require both distinguishing between entities sharing the same name and aggregating evidence across many steps. A question like 'Which actor from the film Heat won at least one Academy Award?' demands disambiguation across multiple films named 'Heat' before reasoning across a large actor set. The benchmark includes 3,600 questions generated by DEEPAMBIGQAGEN, a pipeline grounded in text corpora and linked knowledge graphs. Half of the questions embed name ambiguity; all require multi-hop reasoning. Ground truth is human-verified. Results show GPT-5 achieves 0.21 exact match on unambiguous multi-hop questions and only 0.13 on ambiguous ones — indicating even the best available model struggles significantly with complete answer sets when entity names overlap. The benchmark and generation pipeline are released for community use.