arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.01459cs.LGcs.CL
收藏

LSPO: 面向大语言模型推理中策略优化的长度感知动态采样

LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning

University of Southern California(南加州大学);University of California, Irvine(加州大学尔湾分校)

Weizhe Chen, Sven Koenig, Bistra Dilkina

AI总结:

针对LLM推理任务的RLVR训练优化问题,本文提出新型元RLVR算法LSPO,基于平均响应长度动态选择训练数据,经多模型和数据集验证可提升学习效果,消融实验也为后续研究提供了方向。

摘要:

自Deepseek-R1发布以来,带可验证奖励的强化学习(RLVR)已成为在推理任务上训练大语言模型(LLM)的核心方法。近期研究大多聚焦于修改损失函数,以提升RLVR的效率与效果。本文受LLM过度思考相关研究的启发,提出了长度感知策略优化采样(LSPO)——一种新型元RLVR算法,它在每一步训练中都基于平均响应长度动态选择训练数据。我们在多个基础模型和数据集上评估了LSPO,证明其能持续提升学习效果。此外,我们还开展了详细的消融实验,探究将长度信号融入动态采样的其他替代方式,为后续研究提供了更深入的见解,并指明了有前景的研究方向。

英文摘要:

Since the release of Deepseek-R1, reinforcement learning with verifiable rewards (RLVR) has become a central approach for training large language models (LLMs) on reasoning tasks. Recent work has largely focused on modifying loss functions to make RLVR more efficient and effective. In this paper, motivated by studies of overthinking in LLMs, we propose Length-aware Sampling for Policy Optimization (LSPO), a novel meta-RLVR algorithm that dynamically selects training data at each step based on the average response length. We evaluate LSPO across multiple base models and datasets, demonstrating that it consistently improves learning effectiveness. In addition, we conduct a detailed ablation study to examine alternative ways of incorporating length signals into dynamic sampling, offering further insights and highlighting promising directions for future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑