arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2505.20101cs.CL
收藏

自适应深度推理:在需要时触发深度思考

Adaptive Deep Reasoning: Triggering Deep Thinking When Needed

Tencent Hunyuan Team(腾讯混元团队)

Yunhao Wang, Yuhao Zhang, Tinghao Yu, Can Xu, Feng Zhang, Fengzong Lian

AI总结:

提出一种基于问题复杂度自主切换长短思维链的强化学习方法,通过分组奖励和模式切换损失优化,在数学数据集上实现高效推理且保持性能。

摘要:

大型语言模型(LLMs)在处理复杂任务时通过长链推理展现了令人印象深刻的能力。然而,涉及的大量推理步骤会显著增加计算成本,给实际部署带来挑战。近期研究聚焦于通过多种方法优化推理效率,例如基于长度的提示工程、在变长CoT数据上进行监督微调,以及带有长度惩罚的强化学习,以缩短思维链(CoT)推理过程。尽管这些方法有效减少了推理长度,它们仍然需要初始的推理阶段。更新的方法尝试将长链和短链推理能力整合到单一模型中,但仍依赖手动控制来切换短链和长链CoT。在本工作中,我们提出了一种新颖的方法,可根据问题复杂度自主在短链和长链推理之间切换。我们的方法首先对基础模型进行监督微调,使其同时具备长链和短链推理能力。随后,我们采用强化学习进一步平衡短链和长链CoT的生成,同时通过两个关键策略保持准确性:第一,将强化学习与长-短自适应分组奖励策略相结合,以评估提示复杂度并提供相应奖励;第二,实现基于logit的推理模式切换损失,以优化模型的初始词元选择,从而引导推理类型的选择。在数学数据集上的评估表明,我们的模型能够动态切换长链和短链推理模式,而不会显著牺牲性能。这一进展增强了大型语言模型推理在实际应用中的实用性。

英文摘要:

Large language models (LLMs) have shown impressive capabilities in handling complex tasks through long-chain reasoning. However, the extensive reasoning steps involved can significantly increase computational costs, posing challenges for real-world deployment. Recent efforts have focused on optimizing reasoning efficiency by shortening the Chain-of-Thought (CoT) reasoning processes through various approaches, such as length-aware prompt engineering, supervised fine-tuning on CoT data with variable lengths, and reinforcement learning with length penalties. Although these methods effectively reduce reasoning length, they still necessitate an initial reasoning phase. More recent approaches have attempted to integrate long-chain and short-chain reasoning abilities into a single model, yet they still rely on manual control to toggle between short and long CoT. In this work, we propose a novel approach that autonomously switches between short and long reasoning chains based on problem complexity. Our method begins with supervised fine-tuning of the base model to equip both long-chain and short-chain reasoning abilities. We then employ reinforcement learning to further balance short and long CoT generation while maintaining accuracy through two key strategies: first, integrating reinforcement learning with a long-short adaptive group-wise reward strategy to assess prompt complexity and provide corresponding rewards; second, implementing a logit-based reasoning mode switching loss to optimize the model's initial token choice, thereby guiding the selection of the reasoning type. Evaluations on mathematical datasets demonstrate that our model can dynamically switch between long-chain and short-chain reasoning modes without substantially sacrificing performance. This advancement enhances the practicality of reasoning in large language models for real-world applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑