An Experimental Study of Hybrid Retrieval, NLI and Open-source LLMs for Scientific Claim Verification
| Source: arXiv AI
Tags: claim verification, SciFact, CliniFact, Qwen3, Llama 3.1, BM25
On a 2,035-claim scientific fact-checking benchmark, small open LLMs like Qwen3 8B far outperform an NLI model, and requiring both to agree actually hurts accuracy by overturning correct LLM answers.
Details
Claim verification decides whether text supports, contradicts, or lacks evidence for a claim. NLI models judge single sentences well but struggle across sentences, while LLMs reason more broadly but can overreach. The authors built a benchmark of 2,035 claims from SciFact and CliniFact and retrieved the top five sentences per claim with hybrid semantic plus BM25 retrieval, which slightly beat either alone. Qwen3 8B reached Macro-F1 0.7937, Llama 3.1 8B 0.7204, Mistral 7B 0.6388, and a DeBERTa-v3 NLI baseline only 0.3157. A conservative fusion rule that only accepts agreed labels lowered scores to 0.7392, 0.6730, and 0.5991, because disagreements often overturned correct LLM predictions.