arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2505.02142cs.CL
收藏

探索离线强化学习在大型语言模型推理中的潜力:一项初步研究

Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study

a-m-team(a-m团队)

Xiaoyu Tian, Sitong Zhao, Haotian Wang, Shuaiting Chen, Yiping Peng, Yunjie Ji, Han Zhao, Xiangang Li

AI总结:

本文初步探索了离线强化学习方法DPO及其变体LD-DPO在提升LLMs推理能力中的潜力,实验显示平均性能提升3.3%,在Arena-Hard上提升10.1%,并强调推理长度应与语义丰富性匹配。

摘要:

尽管大型语言模型(LLMs)在长上下文推理方面取得了显著进展,且主要依赖于在线强化学习(Online RL)方法,但这些方法带来了巨大的计算成本和复杂性。相比之下,更简单、更经济的离线强化学习(Offline RL)方法仍未得到充分探索。为弥补这一空白,我们研究了离线强化学习方法,特别是直接偏好优化(Direct Preference Optimization,DPO)及其长度不敏感变体 LD-DPO,在增强LLMs推理能力方面的有效性。在多个推理基准上的大量实验表明,这些更简单的离线强化学习方法能够显著提升模型性能,平均提升幅度为3.3%,其中在具有挑战性的Arena-Hard基准上提升尤为显著,达到10.1%。此外,我们分析了DPO对输出长度的敏感性,强调推理长度的增加应与语义丰富性保持一致,因为不加区分地延长输出可能会对模型性能产生不利影响。我们全面描述了数据处理和训练方法,为开发更具成本效益的离线强化学习方法提供了经验证据和实践见解。

英文摘要:

Despite significant advances in long-context reasoning by large language models (LLMs), primarily through Online Reinforcement Learning (RL) methods, these approaches incur substantial computational costs and complexity. In contrast, simpler and more economical Offline RL methods remain underexplored. To address this gap, we investigate the effectiveness of Offline RL methods, specifically Direct Preference Optimization (DPO) and its length-desensitized variant LD-DPO, in enhancing the reasoning capabilities of LLMs. Extensive experiments across multiple reasoning benchmarks demonstrate that these simpler Offline RL methods substantially improve model performance, achieving an average enhancement of 3.3\%, with a particularly notable increase of 10.1\% on the challenging Arena-Hard benchmark. Furthermore, we analyze DPO's sensitivity to output length, emphasizing that increasing reasoning length should align with semantic richness, as indiscriminate lengthening may adversely affect model performance. We provide comprehensive descriptions of our data processing and training methodologies, offering empirical evidence and practical insights for developing more cost-effective Offline RL approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑