arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.01062cs.AI
收藏

SetPO:基于集级别的策略优化以保持LLM推理的多样性

SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning

Peking University(北京大学);JD Explore Academy, China(京东探索研究院);Beijing Institute of Technology(北京理工大学)

Chenyi Li, Yuan Zhang, Bo Wang, Guoqing Ma, Wei Tang, Haoyang Huang, Nan Duan

AI总结:

SetPO通过集级别策略优化提升LLM推理多样性,通过核化相似性定义目标,验证稀有轨迹对多样性的贡献,实验显示优于基线。

摘要:

可验证奖励的强化学习在增强大语言模型(LLM)推理性能方面表现出显著效果,特别是在数学任务中。然而,此类改进往往伴随着结果多样性的减少,其中模型会将概率质量集中在狭窄的解决方案集上。受边际收益原理的启发,我们引入了一个基于采样轨迹的集级别多样性目标,使用核化相似性来定义。我们的方法为每个采样轨迹推导出留一法边际贡献,并将此目标作为插件优势塑造项整合到策略优化中。我们进一步研究了单个轨迹对语言模型多样性在分布扰动框架内的贡献。这一分析在理论上确认了单调性属性,证明了稀有轨迹对全局多样性的边际贡献始终更高。在各种模型规模上的广泛实验表明,我们提出的方法在Pass@1和Pass@K等不同基准上均优于强大的基线。

英文摘要:

Reinforcement learning with verifiable rewards has shown notable effectiveness in enhancing large language models (LLMs) reasoning performance, especially in mathematics tasks. However, such improvements often come with reduced outcome diversity, where the model concentrates probability mass on a narrow set of solutions. Motivated by diminishing-returns principles, we introduce a set level diversity objective defined over sampled trajectories using kernelized similarity. Our approach derives a leave-one-out marginal contribution for each sampled trajectory and integrates this objective as a plug-in advantage shaping term for policy optimization. We further investigate the contribution of a single trajectory to language model diversity within a distribution perturbation framework. This analysis theoretically confirms a monotonicity property, proving that rarer trajectories yield consistently higher marginal contributions to the global diversity. Extensive experiments across a range of model scales demonstrate the effectiveness of our proposed algorithm, consistently outperforming strong baselines in both Pass@1 and Pass@K across various benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑