arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.09455cs.AI
收藏

E3-TIR: 增强经验利用以实现工具集成推理

E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoning

Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学(深圳));Huawei Technologies Co., Ltd.(华为技术有限公司);Hong Kong University of Science and Technology(香港科技大学)

Weiyang Guo, Zesheng Shi, Liye Zhao, Jiayuan Ma, Zeen Zhu, Junxian He, Min Zhang, Jing Li

AI总结:

本文提出E3-TIR方法,通过整合专家前缀、专家引导和自探索三种经验类型,提升工具集成推理的效率与性能,实验显示在工具使用任务中性能提升6%,数据成本降低。

摘要:

尽管大型语言模型(LLMs)在工具集成推理(TIR)中展现出显著潜力,但现有训练范式面临显著限制:零样本强化学习(Zero-RL)因缺乏先验指导导致探索效率低下和模式退化,而先训练后强化学习(SFT-then-RL)则受限于高数据成本和能力 plateau 问题。为解决这些挑战,我们提出E3-TIR(增强经验利用),一种用于代理训练早期阶段的预热范式。具体而言,我们将训练过程视为三种经验类型的动态整合:专家前缀、专家引导和自探索。通过在专家

英文摘要:

While Large Language Models (LLMs) have demonstrated significant potential in Tool-Integrated Reasoning (TIR), existing training paradigms face significant limitations: Zero-RL suffers from inefficient exploration and mode degradation due to a lack of prior guidance, while SFT-then-RL is limited by high data costs and capability plateaus caused by low-entropy collapse. To address these challenges, we propose E3-TIR (Enhanced Experience Exploitation), a warm-up paradigm for the early stages of agent training. Specifically, we formulate training as the dynamic integration of three experience types: Expert Prefixes, Expert Guided, and Self-Exploration. By executing diverse branching exploration around expert "anchors" and employing a mix policy optimization mechanism, we effectively mitigate distribution shifts and resolve optimization conflicts arising from shared prefixes. Our method dynamically adapts the model's knowledge boundaries, effectively balancing exploration diversity with training efficiency.Experimental results demonstrate that E3-TIR achieves a 6 performance improvement over traditional paradigms on tool-use tasks, while requiring less than 10 of the synthetic data. Furthermore, in terms of ROI, a comprehensive metric integrating performance, data cost, and training efficiency we achieve a 1.46x gain compared to baselines. Code is available at https://github.com/yuki-younai/E3-TIR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑