arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00994cs.AI
收藏

推理与工具使用在智能体强化学习中的竞争:从量化干扰到解耦调优

Reasoning and Tool-use Compete in Agentic RL:From Quantifying Interference to Disentangled Tuning

School of Information, Renmin University of China(中国人民大学信息学院);Bytedance Inc.(字节跳动公司)

Yu Li, Mingyang Yi, Xiuyu Li, Ju Fan, Fuxin Jiang, Binbin Chen, Peng Li, Jie Song, Tieying Zhang

AI总结:

本文通过引入能力效应归因(CEA)量化推理与工具使用行为之间的干扰,并提出解耦动作-推理调优(DART)框架,通过分离参数更新来提升智能体强化学习的性能。

摘要:

智能体强化学习(ARL)训练大型语言模型将推理与外部工具执行交错进行,以解决复杂任务。大多数现有ARL方法训练一组参数来支持推理和工具使用行为,隐含假设联合训练能提升整体智能体性能。尽管被广泛采用,这一假设很少得到实证检验。本文通过引入能力效应归因(CEA)系统性地检验这一假设,提供了推理与工具使用行为之间干扰的定量证据。通过深入分析,我们表明这两种能力常常导致不一致的梯度方向,产生训练干扰,削弱联合优化的有效性,并挑战了主流的ARL范式。为解决此问题,我们提出解耦动作-推理调优(DART),一个简单高效的框架,通过独立的低秩适应模块显式解耦推理和工具使用的参数更新。仅凭这一简单改变,DART在检索增强问答和NL2SQL的十三个基准上超越了所有联合优化基线,并接近2-智能体上界,进一步支持了我们在共享优化下能力干扰的发现。

英文摘要:

Agentic Reinforcement Learning (ARL) trains large language models to interleave reasoning with external tool execution to solve complex tasks. Most existing ARL methods train a single set of parameters to support both reasoning and tool-use behaviors, implicitly assuming that joint training leads to improved overall agent performance. Despite its widespread adoption, this assumption has rarely been examined empirically. In this paper, we systematically examine this assumption by introducing Capability Effect Attribution (CEA), which provides quantitative evidence of interference between reasoning and tool-use behaviors. Through an in-depth analysis, we show that these two capabilities often induce misaligned gradient directions, leading to training interference that undermines the effectiveness of joint optimization and challenges the prevailing ARL paradigm. To address this issue, we propose Disentangled Action--Reasoning Tuning (DART), a simple and efficient framework that explicitly decouples parameter updates for reasoning and tool use via separate low-rank adaptation modules. With this simple change alone, DART outperforms all joint-optimization baselines and approaches the 2-Agent upper bound across thirteen benchmarks on retrieval-augmented QA and NL2SQL, further supporting our finding of capability interference under shared optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑