Confident Fools or Careful Searchers? A Behavioural Study of When LLMs Search Beyond Their Static Internal Memory
| Source: arXiv AI
Tags: Web Search, Tool Use, GPT-5 mini, Grok 4 Fast, Mistral Small, Benchmarks
A 400-question benchmark audits when commercial LLMs choose web search, finding GPT-5 mini and Grok 4 Fast search on over 90% of post-cutoff questions while Mistral Small searches on just 20%.
Details
Agentic assistants must decide whether to answer from memory or call search; unnecessary searches add cost and latency, missing ones produce unsupported answers. The benchmark mixes widely known questions with questions about news after the models' training cut-offs, each with an abstain option, scored by two action-level metrics, NAS and KIS. Across five commercial LLMs, GPT-5 mini and Grok 4 Fast searched on over 90% of outside-knowledge questions and only 1% of inside-knowledge ones. Mistral Small searched on 20% and often abstained. Without explicit temporal cues, NAS for the two selective models fell from about 0.72 to 0.57-0.58.