arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.13348cs.AI
收藏

AutoTool: 通过解耦熵约束实现强化学习中工具使用能力的自动扩展

AutoTool: Automatic Scaling of Tool-Use Capabilities in RL via Decoupled Entropy Constraints

Harbin Institute of Technology, SCIR Lib(哈尔滨工业大学,SCIR实验室);Peking University(北京大学);Huawei Technologies Ltd(华为技术有限公司)

Yirong Zeng, Xiao Ding, Yufei Liu, Yuxian Wang, Qunyao Du, Yutai Hou, Wu Ning, Haonan Song, Duyu Tang, Dandan Tu, Bing Qin, Ting Liu

AI总结:

本文提出AutoTool方法,通过解耦熵约束实现强化学习中工具使用能力的自动扩展,提升模型在复杂问题上的表现,同时减少计算开销。

摘要:

工具使用是AI代理的关键能力,近期进展集中在利用强化学习(RL)扩展显式推理过程以提高性能。然而,当前基于RL的扩展方法面临关键挑战:(a)直接RL训练难以充分扩展推理长度以解决复杂问题;(b)扩展模型倾向于过度思考简单问题,导致显著的token效率低下。为解决这些问题,我们提出了一种新的训练范式,首先使用预热监督微调帮助模型区分简单和复杂问题,随后通过RL使模型能够自动确定适当的推理轨迹。为进一步解决自动推理长度扩展问题,我们发现基于熵的优化目标有效维持模型多样性并成功解锁模型的扩展能力。基于这一见解,我们引入了一种基于熵的长短期推理融合RL策略。在三个基准测试中的实验表明,模型成功实现了高效的工具使用自动扩展,实现了显著的9.8%准确率提升,同时将计算开销减少了约81%。

英文摘要:

Tool use represents a critical capability for AI agents, with recent advances focusing on leveraging reinforcement learning (RL) to scale up the explicit reasoning process to achieve better performance. However, there are some key challenges for tool use in current RL-based scaling approaches: (a) direct RL training often struggles to scale up thinking length sufficiently to solve complex problems, and (b) scaled-up models tend to overthink simpler problems, resulting in substantial token inefficiency. To address these challenges, we propose a novel training paradigm that first employs warm-up supervised fine-tuning to help models distinguish between simple and complex problems, followed by RL that enable models to automatically determine appropriate reasoning trajectories. Furthermore, to tackle the issue of automatic thinking-length scaling, we discover that entropy-based optimization objectives effectively maintain model diversity while successfully unlocking the model's scaling capabilities. Based on this insight, we introduce an entropy-based long-short reasoning fusion RL strategy. Our experiments on three benchmarks demonstrate that model successfully achieves auto-scaling for efficient tool use, achieving significant 9.8\% accuracy improvements while reducing computational overhead by \textasciitilde81\%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑