Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

August 19, 2026 | Source: arXiv AI

Tags: reinforcement learning, LLM reasoning, self-rewarding AI, multi-agent training, VLM, unsupervised learning

Co-RL achieves 3–8.6% reasoning gains on LLM benchmarks and 2.3–7.2% on multimodal tasks without any ground-truth labels — by having multiple decoupled models reward each other during RL training, with cohort diversity preventing collapse.

Details

The dominant approach to improving LLM reasoning requires verifiable ground-truth rewards — which are expensive to create and increasingly scarce as models reach capabilities beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependency but suffers from a critical failure mode: models trained only on their own outputs reinforce existing biases and gradually collapse toward homogenized responses. Co-RL (Cooperative Reinforcement Learning) sidesteps both problems by training multiple decoupled models simultaneously, each using rewards derived from peer completions rather than self-feedback. No parameters are shared between models. The paper shows that diversity within the cohort is essential — using heterogeneous model families, sizes, and rephrased training samples reduces correlated errors that cause the feedback loops driving collapse. Measured across seven text-only benchmarks, Co-RL delivers 3.0–8.6% average gains for LLMs; on four multimodal benchmarks it yields 2.3–7.2% gains for VLMs. These results match or surpass supervised methods while requiring zero ground-truth labels. Code is publicly available. This work is particularly relevant as the AI field pushes toward self-improving systems that can advance capabilities beyond the ceiling of human annotation.