EMBERS: Empirically Motivated Robustness Testing of Deep Research Systems via Defect-Guided Fuzzing

| Source: arXiv AI

Tags: EMBERS, Deep Research Agents, Fuzzing, Robustness Testing, LLM Agents

EMBERS is a defect-guided fuzzing framework for deep research agents, built on a study of 1,932 execution trajectories from three systems that surfaced nine recurring defect categories.

Details

Deep research systems turn LLMs into agents that split questions, search the web repeatedly, and synthesize answers. Their multi-stage execution means small changes in a question or search input can carry through to wrong answers. The authors studied 1,932 execution trajectories from three representative deep research systems and found nine recurring defect categories spread across subquestion formulation, online search, and final reasoning. Using those findings, they built EMBERS, a fuzzing framework that starts from correctly answered questions and generates variations aimed at the observed defect types to test robustness. For teams shipping research agents, the defect taxonomy offers a checklist of where these pipelines tend to break beyond simple task accuracy.