AgenticFinBench: Benchmarking Financial Agents on Diverse Tasks from Real-World Requests
| Source: arXiv AI
Tags: AgenticFinBench, financial agents, hallucination detection, agent harness, LLM benchmark
AgenticFinBench tests financial agents on 230 real-world tasks with a hallucination audit that drops the baseline leader from first to fourth, and finds harness design matters more than model choice for evidence grounding.
Details
Financial requests vary widely and are often open-ended, and existing benchmarks neither cover that range nor separate grounding from correctness. AgenticFinBench contains 230 real-world financial tasks across 8 categories, 2 languages and 4 complexity levels. Its reference-free evaluation pairs question-specific rubrics with a two-stage claim audit that flags hallucinations: claims missing from the agent's trace and refuted by external evidence. Across 18 agent configurations, hallucination auditing cut the baseline leader's score by 7.58 points and moved it to fourth. Harness differences explained 28.6% of variation in evidence grounding versus 8.8% for model differences. Quality fell from 89.44 on direct retrieval to 79.18 on the hardest tasks. Case studies show agents struggle to explain how events affect profits or prices and to compare financial data on a consistent basis.