On the Primacy Bias in RLVR Training

| Source: arXiv AI

Tags: RLVR, primacy bias, reinforcement learning, LLM reasoning, catastrophic forgetting

A new study finds RLVR training mostly sharpens problems a model already solves, while hard problems drive real reasoning gains but can cause forgetting on easy ones with prolonged training.

Details

It is still debated whether reinforcement learning with verifiable rewards (RLVR) expands reasoning beyond what the base model already encodes. The authors study this through primacy bias, the tendency to over-rely on learned representations. In controlled experiments and large-scale analysis, RLVR on mixed data mostly improved easier problems and gave marginal gains on harder ones. Hard problems produced modest overall gains but did the most to expand reasoning, though long training on them could cause forgetting on easy problems. The findings offer an explanation for conflicting prior results: gains on weak out-of-domain areas versus masked gains on well-trained ones.