Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development
| Source: arXiv AI
Tags: AI agents, benchmark evaluation, long-horizon tasks, autonomous research, frontier models, agent evaluation
A systematic evaluation of 7 frontier AI models on 36 long-horizon R&D tasks finds current agents operate as engineering optimizers — implementing established techniques reliably but with high run-to-run variance and without genuine methodological novelty.
Details
As AI agents are deployed for technical work — running experiments, implementing improvements, debugging systems — understanding what they actually do beyond headline scores becomes critical. This paper evaluates 7 frontier models on 36 long-horizon tasks using a new framework with rule-based metrics tracking three phases: Solution Framing (how agents define problems), Execution (how they implement solutions), and Feedback Control (how they incorporate results from prior attempts). Results reveal a gap between perceived and actual capability. Agents can formulate and implement practical solutions, but performance varies substantially across runs on identical tasks — a consistency problem that final scores obscure. Their strongest solutions adapt or combine established techniques; genuine methodological novelty is rare. Experience reuse (applying knowledge from earlier tasks to later ones) can help but also mislead subsequent decisions, depending on how relevant the prior experience is. A key finding for system designers: harness design affects performance stability as much as the underlying model — context management, tool availability, and scaffolding matter enormously. For teams evaluating AI agents for research automation, this paper provides both a sobering calibration and a structured vocabulary for diagnosing where capable agents actually fail.