arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.11682cs.LGcs.AI
收藏

熵保持强化学习

Entropy-Preserving Reinforcement Learning

Aleksei Petrenko, Ben Lipkin, Kevin Chen, Erik Wijmans, Marco Cusumano-Towner, Raja Giryes, Philipp Krähenbühl

AI总结:

本文提出熵保持强化学习方法,通过监控和控制熵来维持探索多样性,提升策略性能和可训练性。

摘要:

策略梯度算法推动了语言模型推理的许多近期进展。一个吸引人的特性是它们能够通过自身轨迹上的探索来学习,这一过程对于促进多样性和创造性解决方案至关重要。正如本文所示,许多策略梯度算法在训练过程中自然地减少熵——从而减少探索轨迹的多样性——导致策略在探索能力上越来越受限。本文认为,熵应在整个训练过程中被主动监控和控制。我们正式分析了主流策略梯度目标对熵动态的贡献,识别了显著影响熵行为的实证因素(如数值精度),并提出熵控制的显式机制。这些包括REPO,一组修改优势函数以调节熵的算法,以及ADAPO,一种自适应非对称截断方法。使用我们熵保持方法训练的模型在训练过程中保持多样性,最终策略在性能上更优,并在新环境中保持序列学习的可训练性。

英文摘要:

Policy gradient algorithms have driven many recent advancements in language model reasoning. An appealing property is their ability to learn from exploration on their own trajectories, a process crucial for fostering diverse and creative solutions. As we show in this paper, many policy gradient algorithms naturally reduce the entropy -- and thus the diversity of explored trajectories -- as part of training, yielding a policy increasingly limited in its ability to explore. In this paper, we argue that entropy should be actively monitored and controlled throughout training. We formally analyze the contributions of leading policy gradient objectives on entropy dynamics, identify empirical factors (such as numerical precision) that significantly impact entropy behavior, and propose explicit mechanisms for entropy control. These include REPO, a family of algorithms that modify the advantage function to regulate entropy, and ADAPO, an adaptive asymmetric clipping approach. Models trained with our entropy-preserving methods maintain diversity throughout training, yielding final policies that are more performant and retain their trainability for sequential learning in new environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑