arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.26491cs.LG
收藏

通过离策略影响指导实现数据高效的RLVR

Data-Efficient RLVR via Off-Policy Influence Guidance

CoAI Group, Tsinghua University(联合人工智能组,清华大学)

Erle Zhu, Dazhi Jiang, Yuan Wang, Xujun Li, Jiale Cheng, Yuxian Gu, Yilin Niu, Aohan Zeng, Jie Tang, Minlie Huang, Hongning Wang

AI总结:

本文提出基于影响函数的理论方法,通过离策略影响估计和稀疏随机投影提升RLVR的数据效率,实验显示CROPI框架在训练加速方面表现优异。

摘要:

数据选择是增强大型语言模型(LLMs)推理能力的关键方面,对于可验证奖励强化学习(RLVR)来说尤为重要。当前的数据选择方法大多基于启发式方法,缺乏理论保证和通用性。本文提出了一种基于影响函数的理论方法,通过离策略影响估计和稀疏随机投影提升RLVR的数据效率。为克服在线影响估计所需的策略回放带来的计算成本,我们引入了离策略影响估计方法,利用预收集的离线轨迹高效近似数据影响。此外,为管理LLMs的高维梯度,我们采用稀疏随机投影来减少维度,提高存储和计算效率。利用这些技术,我们开发了基于离策略影响指导的多阶段RL框架(CROPI),该框架迭代选择当前策略最具有影响的数据。实验表明,CROPI显著加速了训练。在7B参数模型上,实验结果表明,CROPI在训练加速方面表现优异。在1.5B模型上,它实现了2.66倍的步骤级加速,同时仅使用完整数据集训练的10%数据。我们的结果突显了基于影响的数据选择在高效RLVR中的巨大潜力。

英文摘要:

Data selection is a critical aspect of Reinforcement Learning with Verifiable Rewards (RLVR) for enhancing the reasoning capabilities of large language models (LLMs). Current data selection methods are largely heuristic-based, lacking theoretical guarantees and generalizability. This work proposes a theoretically-grounded approach using influence functions to estimate the contribution of each data point to the learning objective. To overcome the prohibitive computational cost of policy rollouts required for online influence estimation, we introduce an off-policy influence estimation method that efficiently approximates data influence using pre-collected offline trajectories. Furthermore, to manage the high-dimensional gradients of LLMs, we employ sparse random projection to reduce dimensionality and improve storage and computation efficiency. Leveraging these techniques, we develop \textbf{C}urriculum \textbf{R}L with \textbf{O}ff-\textbf{P}olicy \text{I}nfluence guidance (\textbf{CROPI}), a multi-stage RL framework that iteratively selects the most influential data for the current policy. Experiments on models up to 7B parameters demonstrate that CROPI significantly accelerates training. On a 1.5B model, it achieves a 2.66x step-level acceleration while using only 10\% of the data per stage compared to full-dataset training. Our results highlight the substantial potential of influence-based data selection for efficient RLVR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑