They Know Where It Is, But Not What It Says: A Bilingual Hallucination Benchmark of Hadith Recall in Open LLMs

| Source: arXiv AI

Tags: Hadith, Jais-7B, ALLaM-7B, Mistral-7B, hallucination benchmark, retrieval-augmented generation

Across 30,000 responses, five open-weight LLMs never reproduced an exact Hadith text from its reference, showing parametric memory is unsafe for verbatim religious-text recall without retrieval.

Details

People increasingly ask LLMs about religious texts. This bilingual English/Arabic benchmark gives five open-weight models (Mistral-7B, Gemma3-4B, Jais-7B, ALLaM-7B and Phi4-mini-3.8B) only a bibliographic reference from the six canonical Sunni Hadith collections and asks for the exact text, with no retrieval. With 500 Hadith per collection, 30,000 responses in total, none achieved both high lexical (BLEU) and high semantic fidelity. Arabic responses were uniformly worse than English, and refusals were tracked separately. The authors conclude that deployments for Islamic scholarship should require retrieval-augmented generation over a verified corpus. The lesson applies to any product promising exact quotation of scripture, law or other fixed texts.