arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.19345cs.LGcs.AI
收藏

用于软优势策略优化的平滑门函数

Smooth Gate Functions for Soft Advantage Policy Optimization

Institute for Artificial Intelligence, Lomonosov Moscow State University(莫斯科罗蒙诺索夫莫斯科国立大学人工智能研究所);Lomonosov Moscow State University(莫斯科罗蒙诺索夫莫斯科国立大学)

Egor Denisov, Svetlana Glazyrina, Maksim Kryzhanovskiy, Roman Ischenko

AI总结:

本文探讨了通过平滑门函数提升策略优化稳定性和模型性能,分析了不同门函数在数学推理任务中的实验结果,为大语言模型训练提供指导。

摘要:

组相对策略优化(GRPO)在大语言模型训练中取得了显著进展,但因使用硬裁剪而易不稳定。软自适应策略优化(SAPO)通过将裁剪替换为平滑的sigmoid基门函数,提高了更新稳定性。本文进一步研究不同门函数对训练稳定性和最终模型性能的影响,明确了可行门函数应满足的关键属性,并识别了几类函数进行实验证明。本文基于Qwen2.5-7B-Instruct模型在数学推理任务中的实验结果,分析了发现并为设计更平滑和稳健的策略优化目标提供实践指导。

英文摘要:

Group Relative Policy Optimization (GRPO) has significantly advanced the training of large language models and enhanced their reasoning capabilities, while it remains susceptible to instability due to the use of hard clipping. Soft Adaptive Policy Optimization (SAPO) addresses this limitation by replacing clipping with a smooth sigmoid-based gate function, which leads to more stable updates. We have decided to push this theory further and investigate the impact of different gate functions on both training stability and final model performance. We formalize the key properties that admissible gates should satisfy and identify several families of such functions for empirical evaluation. This paper presents an analysis of our findings based on experiments conducted with the Qwen2.5-7B-Instruct model on mathematical reasoning tasks. These results provide practical guidance for designing smoother and more robust policy optimization objectives for large language model training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑