From Logs to Faults: Reference-Free Fault Localization for Retrieval-Augmented Generation Pipelines

| Source: arXiv AI

Tags: RAGLog, RAG, Fault Localization, Claude Haiku, Observability

RAGLog, an open-source tool, localizes faults across RAG pipeline stages from execution logs alone and scores a pooled F1 of 0.73 on natural faults, ahead of Claude Code (0.58) and Codex (0.52), at about $0.005 per run.

Details

Debugging retrieval-augmented generation is slow because faults in one stage can surface downstream or stay hidden even when the answer looks right. Existing approaches usually need reference answers or code instrumentation. RAGLog converts heterogeneous API logs into a common RAG schema and combines rule-based, embedding-based and LLM-based checks to flag the faulty stage. The authors also release a live fault injector and RAGLog Faults, a dataset of 3,134 runs from five pipelines with stage-level labels. With Claude Haiku 5.5, RAGLog reaches a pooled F1 of 0.73 on naturally occurring faults versus 0.58 for Claude Code and 0.52 for Codex, and 0.84 on injected faults. Each run takes about five seconds and US$0.005.