The Evolution Strategies Paradox: Better Models from Worse Perturbations

| Source: arXiv AI

Tags: Evolution Strategies, LLM Fine-Tuning, Zeroth-Order Optimization

Researchers explain why Evolution Strategies can fine-tune billion-parameter LLMs even when every perturbed offspring scores worse than the parent, tracing the paradox to noise and curvature in high dimensions.

Details

When Evolution Strategies (ES) are used to fine-tune LLMs, all offspring models can perform worse than the parent while the parent still improves each step. The authors observe this across models, tasks and modalities. They attribute it to the local reward landscape and the ES estimator: in high-dimensional spaces exploration lowers performance but still points toward good directions, and aggregating those directions yields local improvement. The paper frames this as a distinct optimization behavior and offers guidance for using perturbation-based methods in LLM fine-tuning and other high-dimensional problems.