When AI Designs AI: Innovation or Imitation?
| Source: arXiv AI
Tags: AI research automation, algorithmic design, LLM agents, automated AI, benchmark, meta-learning
A large empirical study finds 96.8% of AI-designed AI methods recombine existing human algorithmic choices — nearly half exactly replicate a human design — and current LLM agents match human SOTA in only 10 of 72 tested configurations without reliable generalization.
Details
As automated AI research (AutoML, AI scientists) grows more practical, a key question is whether LLM agents actually invent new approaches or just shuffle known components. This paper provides rigorous measurements: it derives task-specific algorithmic design spaces from human-designed methods, maps both human and agent solutions into these spaces, and quantifies differences at the module level. Results across widely-used LLM agents on representative open-ended AI tasks spanning multiple modalities: agents can occasionally match or exceed human SOTA performance (10 of 72 tested configurations), but success does not generalize reliably across tasks or agents. More tellingly, 96.8% of agent-designed methods stay within human-derived algorithmic design spaces, and nearly half exactly match an existing human design. This does not dismiss AI-designed AI — 10/72 successes is non-trivial — but it constrains the narrative. Current agents are sophisticated combinatorial searchers over known algorithms, not creative inventors of new paradigms. The paper provides a principled methodology for measuring this gap. For teams evaluating automated research tools or planning AI scientist investments, these benchmarks offer a grounded picture of where the capability ceiling currently sits.