arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.03392cs.LGcs.AI
收藏

在大型语言模型强化微调中的熵动态研究

On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models

University of Science(科学大学);Tongyi Lab, Alibaba Group(通义实验室,阿里巴巴集团)

Shumin Wang, Yuexiang Xie, Wenhao Zhang, Yuchang Sun, Yanxi Chen, Yaliang Li, Yanyong Zhang

AI总结:

本文研究了大型语言模型强化微调中熵动态的理论框架,提出了熵变化的判别表达式,并设计了熵控制方法以优化探索与利用的平衡。

摘要:

熵作为衡量大型语言模型(LLMs)生成输出多样性的关键指标,为了解其探索能力提供了有价值的见解。尽管最近的研究越来越多地关注监控和调整熵以更好地在强化微调(RFT)中平衡探索与利用,但对这一过程中熵动态的系统理解仍待深入研究。在本文中,我们建立了分析RFT过程中熵动态的理论框架,该框架从一个判别表达式开始,量化了单个logit更新下的熵变化。这一基础使我们能够推导出熵变化的一阶表达式,进而可以扩展到群相对策略优化(GRPO)的更新公式。从理论分析中得出的推论和见解启发了熵控制方法的设计,同时也为现有研究中各种基于熵的方法提供了统一的解释视角。我们提供了实证证据来支持我们分析的主要结论,并展示了推导出的熵判别器裁剪方法的有效性。本研究为RFT训练动态提供了新的见解,为在LLM微调过程中优化探索-利用平衡提供了理论支持和实践策略。

英文摘要:

Entropy serves as a critical metric for measuring the diversity of outputs generated by large language models (LLMs), providing valuable insights into their exploration capabilities. While recent studies increasingly focus on monitoring and adjusting entropy to better balance exploration and exploitation in reinforcement fine-tuning (RFT), a principled understanding of entropy dynamics during this process is yet to be thoroughly investigated. In this paper, we establish a theoretical framework for analyzing the entropy dynamics during the RFT process, which begins with a discriminant expression that quantifies entropy change under a single logit update. This foundation enables the derivation of a first-order expression for entropy change, which can be further extended to the update formula of Group Relative Policy Optimization (GRPO). The corollaries and insights drawn from the theoretical analysis inspire the design of entropy control methods, and also offer a unified lens for interpreting various entropy-based methods in existing studies. We provide empirical evidence to support the main conclusions of our analysis and demonstrate the effectiveness of the derived entropy-discriminator clipping methods. This study yields novel insights into RFT training dynamics, providing theoretical support and practical strategies for optimizing the exploration-exploitation balance during LLM fine-tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑