arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.04066cs.CLcs.ETcs.LG
收藏

适应与繁荣!面向改进大语言模型推理的自适应幂均策略优化

Adapt to Thrive! Adaptive Power-Mean Policy Optimization for Improved LLM Reasoning

Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室);The Chinese University of Hong Kong(香港中文大学)

Yiming Huang, Zhenbo Shi, Shuzheng Gao, Cuiyun Gao, Peiyi Han, Chuanyi Liu

AI总结:

本文提出自适应幂均策略优化方法,通过引入幂均目标和反馈自适应裁剪,提升大语言模型的推理性能,实验显示其在多个任务中表现优于现有方法。

摘要:

可验证奖励的强化学习(RLVR)是增强大语言模型(LLMs)推理能力的重要范式。然而,现有方法通常依赖静态策略优化方案,与模型不断演变的推理能力不匹配。为了解决这一问题,我们提出了自适应幂均策略优化(APMPO),其包含两个主要创新:幂均策略优化(PMPO)和反馈自适应裁剪(FAC)。具体而言,PMPO引入了一个广义的幂均目标,使模型能够自适应地从算术均值的信号放大行为过渡到几何均值的一致性强制行为。FAC根据实时奖励统计自适应调整裁剪界限,以克服静态机制的限制。借助这些创新,APMPO提升了学习动态和推理性能。在三个推理任务的九个数据集上的广泛实验显示,APMPO在数学推理基准上的平均Pass@1分数比GRPO提高了3.0分(使用Qwen2.5-3B-Instruct)。

英文摘要:

Reinforcement Learning with Verifiable Rewards (RLVR) is an essential paradigm that enhances the reasoning capabilities of Large Language Models (LLMs). However, existing methods typically rely on static policy optimization schemes that misalign with the model's evolving reasoning capabilities. To address this issue, we propose Adaptive Power-Mean Policy Optimization (APMPO), which comprises two main innovations: Power-Mean Policy Optimization (PMPO) and Feedback-Adaptive Clipping (FAC). Specifically, PMPO introduces a generalized power-mean objective. This enables the model to adaptively transition from the signal-amplifying behavior of the arithmetic mean to the consistency-enforcing behavior of the geometric mean. FAC adaptively adjusts clipping bounds based on real-time reward statistics to overcome the limitations of static mechanisms. Capitalizing on these innovations, APMPO improves learning dynamics and reasoning performance. Extensive experiments on nine datasets across three reasoning tasks showcase the superiority of APMPO over state-of-the-art RLVR-based baselines. For instance, APMPO boosts the average Pass@1 score on mathematical reasoning benchmarks by 3.0 points compared to GRPO when using Qwen2.5-3B-Instruct.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑