arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.01204cs.CL
收藏

ASTER: 基于工具集成扩展推理的代理扩展

ASTER: Agentic Scaling with Tool-integrated Extended Reasoning

National Key Laboratory for Novel Software Technology, Nanjing University, China(新型软件技术国家实验室);School of Artificial Intelligence, Nanjing University, China(人工智能学院);Department of Foundation Model, 2012 Labs, Huawei(基础模型部门);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))

Xuqin Zhang, Quan He, Zhenrui Zheng, Zongzhang Zhang, Xu He, Dong Li

AI总结:

ASTER通过冷启动策略提升工具集成推理性能,实现长周期推理的高效扩展。

摘要:

强化学习(RL)已逐渐成为激发大语言模型(LLMs)长周期推理的主导范式。然而,通过RL扩展工具集成推理(TIR)仍然具有挑战性,因为交互崩溃:一种病理状态,其中模型无法维持多轮工具使用,反而退化为大量内部推理,仅进行事后代码验证。我们系统地研究了三个问题:(i)冷启动SFT如何诱导代理、工具使用的行为先验;(ii)冷启动轨迹的交互密度如何影响探索和下游RL结果;(iii)RL交互预算如何影响学习动态和在不同推理时间预算下的泛化能力。我们随后引入ASTER(基于工具集成扩展推理的代理扩展),通过针对性的冷启动策略优先选择交互密集的轨迹来克服这种崩溃。我们发现,仅4K交互密集轨迹的小型专家冷启动集能够产生最强的下游性能,建立了一个稳健的先验,使在扩展RL训练期间实现更优的探索。广泛的评估显示,ASTER-4B在竞争性的数学基准上取得了最先进的结果,达到90.0%的AIME 2025成绩,超越了领先的前沿开源模型,包括DeepSeek-V3.2-Exp。

英文摘要:

Reinforcement learning (RL) has emerged as a dominant paradigm for eliciting long-horizon reasoning in Large Language Models (LLMs). However, scaling Tool-Integrated Reasoning (TIR) via RL remains challenging due to interaction collapse: a pathological state where models fail to sustain multi-turn tool usage, instead degenerating into heavy internal reasoning with only trivial, post-hoc code verification. We systematically study three questions: (i) how cold-start SFT induces an agentic, tool-using behavioral prior, (ii) how the interaction density of cold-start trajectories shapes exploration and downstream RL outcomes, and (iii) how the RL interaction budget affects learning dynamics and generalization under varying inference-time budgets. We then introduce ASTER (Agentic Scaling with Tool-integrated Extended Reasoning), a framework that circumvents this collapse through a targeted cold-start strategy prioritizing interaction-dense trajectories. We find that a small expert cold-start set of just 4K interaction-dense trajectories yields the strongest downstream performance, establishing a robust prior that enables superior exploration during extended RL training. Extensive evaluations demonstrate that ASTER-4B achieves state-of-the-art results on competitive mathematical benchmarks, reaching 90.0% on AIME 2025, surpassing leading frontier open-source models, including DeepSeek-V3.2-Exp.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑