Data selection is a critical aspect of Reinforcement Learning with Verifiable Rewards (RLVR) for enhancing the reasoning capabilities of large language models (LLMs). Current data selection methods are largely heuristic-based, lacking theoretical guarantees and generalizability. This work proposes a theoretically-grounded approach using influence functions to estimate the contribution of each data point to the learning objective. To overcome the prohibitive computational cost of policy rollouts required for online influence estimation, we introduce an off-policy influence estimation method that efficiently approximates data influence using pre-collected offline trajectories. Furthermore, to manage the high-dimensional gradients of LLMs, we employ sparse random projection to reduce dimensionality and improve storage and computation efficiency. Leveraging these techniques, we develop \textbf{C}urriculum \textbf{R}L with \textbf{O}ff-\textbf{P}olicy \text{I}nfluence guidance (\textbf{CROPI}), a multi-stage RL framework that iteratively selects the most influential data for the current policy. Experiments on models up to 7B parameters demonstrate that CROPI significantly accelerates training. On a 1.5B model, it achieves a 2.66x step-level acceleration while using only 10\% of the data per stage compared to full-dataset training. Our results highlight the substantial potential of influence-based data selection for efficient RLVR.
Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVR
School of Informatics, Xiamen University(厦门大学信息学院);LLM Team, Shopee Pte. Ltd.(Shopee 股份有限公司语言模型团队);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院)
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
Beyond Forgetting: Diagnosing and Harnessing Shared Reasoning in Continual RLVR
State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室(BIGAI));Beijing Institute of Technology(北京理工大学);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);State Key Lab of General AI, School of Intelligence Science and Technology, Peking University(通用人工智能国家重点实验室、北京大学智能科学与技术学院)
Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR
New York University(纽约大学);University of Chicago(芝加哥大学);University of Waterloo(滑铁卢大学);University of Alberta(阿尔伯塔大学);NYU Shanghai(纽约大学上海分校);Alberta Machine Intelligence Institute (Amii)(阿尔伯塔机器智能研究所)
Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling
South China University of Technology(华南理工大学);Tsinghua University(清华大学);The Hong Kong University of Science and Technology(香港科技大学);Huawei 2012 Laboratories(华为2012实验室);Shenzhen Technology University(深圳技术大学)
RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Southern University of Science and Technology(南方科技大学)
When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVR
National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University(国家多媒体软件工程研究中心,武汉大学计算机学院);Meituan Longcat Team(美团Longcat团队);The University of Sydney(悉尼大学);University of Science and Technology of China(中国科学技术大学)
Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence
School of Computer Science and Engineering, Northeastern University, China(东北大学计算机科学与工程学院);Tsinghua University(清华大学);Meituan(美团);Meta AI;NiuTrans Research, Shenyang, China(新译研究院,沈阳,中国)