Reinforcement Learning has demonstrated substantial improvements in the reasoning abilities of Large Language Models (LLMs), exhibiting significant applicability across various domains. Recent research has identified that tokens within LLMs play distinct roles during reasoning tasks, categorizing them into high-entropy reasoning tokens and low-entropy knowledge tokens. Prior approaches have typically focused on restricting updates to indirectly encourage exploration, yet they do not explicitly facilitate exploratory behavior during the token generation stage itself. In this work, we introduce a complementary approach that explicitly promotes exploration during sampling by applying distinct temperature settings for different token types. Specifically, our method employs higher temperatures for reasoning tokens to actively encourage exploration, while retaining lower temperatures for knowledge tokens to maintain factual correctness. Furthermore, we systematically investigate various multi-temperature scheduling strategies and their impacts within reinforcement learning contexts. Empirical evaluations on several reasoning benchmarks demonstrate that our approach significantly enhances the reasoning performance of LLMs. The code is available at https://github.com/zhmzm/Multi_Temperature_Verl.git.
CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR
Qwen Large Model Application Team, Alibaba Institute of Automation, Chinese Academy of Sciences(文勤大模型应用团队,阿里巴巴自动化研究所,中国科学院);Qwen Large Model Application Team, Alibaba(文勤大模型应用团队,阿里巴巴)
Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);Mila - Quebec AI Institute(Mila - 魁北克人工智能研究所);University of Waterloo(滑铁卢大学);The Chinese University of Hong Kong(香港中文大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)
TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs
University of Chinese Academy of Sciences(中国科学院大学);Meituan(美团);MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所模式识别国家重点实验室与智能信息处理重点实验室)
When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVR
National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University(国家多媒体软件工程研究中心,武汉大学计算机学院);Meituan Longcat Team(美团Longcat团队);The University of Sydney(悉尼大学);University of Science and Technology of China(中国科学技术大学)
Auditing Data Membership in Reinforcement Learning With Verifiable Rewards
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Shanghai Jiao Tong University(上海交通大学);Shandong University(山东大学);Xidian University(西安电子科技大学);Wuhan University(武汉大学)
Understanding Diversity Collapse in RLVR via the Lens of Overtraining
Sydney AI Centre, The University of Sydney(悉尼大学悉尼人工智能中心);Southeast University(东南大学);Microsoft(微软);Data61, CSIRO(澳大利亚联邦科学与工业研究组织Data61);Chongqing University(重庆大学);Nanyang Technological University(南洋理工大学)
Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization
Tianjin University(天津大学);Beijing Institute of Technology(北京理工大学);Singapore Management University(新加坡国立大学);University of Chinese Academy of Sciences(中国科学院大学);Xiaomi Inc(小米公司)
Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidance
School of Artificial Intelligence, Beihang University, Beijing, China(北京航空航天大学人工智能学院);Hangzhou International Innovation Institute, Beihang University, Hangzhou, China(北京航空航天大学杭州国际创新研究院);University of Leicester, Leicester, United Kingdom(莱斯特大学);Nanyang Technological University, Singapore(南洋理工大学);The Hong Kong Polytechnic University, Hong Kong SAR, China(香港理工大学)
EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
The Chinese University of Hong Kong(香港中文大学);Microsoft Research Asia(微软亚洲研究院);Hong Kong Baptist University(香港浸会大学);LMU Munich(慕尼黑大学);MoE Key Laboratory of High Confidence Software Technologies(教育部高可信软件技术重点实验室)
ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward
LMU Munich(慕尼黑大学);Harvard University(哈佛大学);University of Cambridge(剑桥大学);Mina AI;Konrad Zuse School of Excellence in Reliable AI (relAI)(康拉德·楚泽可靠人工智能卓越学校(relAI))