arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.08892cs.CLcs.AI
收藏

探索 RLVR 中用于 Token 级和 Rollout 级控制的多温度策略

Exploring Multi-Temperature Strategies for Token- and Rollout-Level Control in RLVR

University of Notre Dame(诺丁汉大学);ByteDance(字节跳动)

Haomin Zhuang, Yujun Zhou, Taicheng Guo, Yue Huang, Fangxu Liu, Kai Song, Xiangliang Zhang

AI总结:

本文提出一种多温度采样策略,通过为高熵推理 token 设定高温、低熵知识 token 设定低温,在强化学习中显式促进探索并保持事实准确,显著提升了 LLMs 的推理性能。

摘要:

强化学习在提升大型语言模型(LLMs)的推理能力方面展现出显著进步,并在各个领域表现出重要的适用性。近期研究指出,LLMs 内部的 token 在推理任务中扮演不同角色,可将其分为高熵推理 token 和低熵知识 token。先前的方法通常侧重于限制更新来间接鼓励探索,但并未在 token 生成阶段本身显式促进探索行为。在本研究中,我们提出一种互补方法,通过对不同 token 类型应用不同的温度设置,在采样期间显式促进探索。具体而言,我们的方法对推理 token 采用较高温度以积极鼓励探索,同时对知识 token 保持较低温度以维持事实准确性。此外,我们系统地研究了多种多温度调度策略及其在强化学习环境中的影响。在多个推理基准上的实证评估表明,我们的方法显著提升了 LLMs 的推理性能。代码可在 https://github.com/zhmzm/Multi_Temperature_Verl.git 获取。

英文摘要:

Reinforcement Learning has demonstrated substantial improvements in the reasoning abilities of Large Language Models (LLMs), exhibiting significant applicability across various domains. Recent research has identified that tokens within LLMs play distinct roles during reasoning tasks, categorizing them into high-entropy reasoning tokens and low-entropy knowledge tokens. Prior approaches have typically focused on restricting updates to indirectly encourage exploration, yet they do not explicitly facilitate exploratory behavior during the token generation stage itself. In this work, we introduce a complementary approach that explicitly promotes exploration during sampling by applying distinct temperature settings for different token types. Specifically, our method employs higher temperatures for reasoning tokens to actively encourage exploration, while retaining lower temperatures for knowledge tokens to maintain factual correctness. Furthermore, we systematically investigate various multi-temperature scheduling strategies and their impacts within reinforcement learning contexts. Empirical evaluations on several reasoning benchmarks demonstrate that our approach significantly enhances the reasoning performance of LLMs. The code is available at https://github.com/zhmzm/Multi_Temperature_Verl.git.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑