Efficient RLVR Scheduling via Graph-Structured Online Difficulty Estimation

Details

Reinforcement learning with verifiable rewards (RLVR) is a key technique for improving LLM reasoning, but training efficiency is bottlenecked by rollout exploration costs. Assigning equal rollout budgets to all samples is wasteful: easy samples receive redundant exploration while genuinely learnable-but-hard samples receive too little. Existing adaptive schedulers address this through curriculum selection or non-uniform rollout allocation, but online difficulty estimation remains difficult. History-based estimators face cold starts and stale feedback; dedicated probing adds generation overhead; and most approaches ignore relations among samples. The proposed framework builds a difficulty-aware sample graph from semantic and reasoning similarities, then uses a Potts prior to encourage neighboring samples to share latent difficulty states. A state-level Beta-Binomial model aggregates rollout outcomes per state, updated online via mean-field variational inference. This is plug-and-play — it integrates into any existing sample-selection or rollout-allocation scheduler. Experiments across multiple base models, RL schedulers, and benchmarks show consistent performance gains. For labs training reasoning models with RLVR pipelines, this approach could meaningfully reduce training compute while improving final model performance.