Artifact-to-Capability: Diagnosing Breaks in Capability Formation of Self-Evolving Agents in a Controlled Environment

| Source: arXiv AI

Tags: self-evolving agents, Reflexion, Voyager, Darwin Godel Machine, Gomoku, agent evaluation

A controlled Gomoku study shows Reflexion, Voyager and DGM all display the usual signs of self-improvement while each fails to actually form a new capability, undercutting signal-based evaluation of self-evolving agents.

Details

Self-evolving agents save experience as artifacts and reuse them. Most evaluations rely on three proxy signals: artifacts pile up, behavior changes when they load, and metrics rise. The authors argue it was never defined what those signals should indicate. They define capability formation as five sequentially dependent properties, two of which act as a checkpoint, and judge them at state, game and lineage levels. In freestyle Gomoku with the same base model, all three methods show the three signals, yet each breaks differently. Reflexion's pass rate rises only because more of its moves are legal. Voyager combines individually sound components that together miss all 20 available one-move wins. DGM passes the checkpoint but later versions only lose passing states. The takeaway for agent builders is that rising metrics alone do not prove an agent has learned anything durable.