ASTER: 基于工具集成扩展推理的代理扩展
ASTER: Agentic Scaling with Tool-integrated Extended Reasoning
AI总结:
ASTER通过冷启动策略提升工具集成推理性能,实现长周期推理的高效扩展。
摘要:
强化学习(RL)已逐渐成为激发大语言模型(LLMs)长周期推理的主导范式。然而,通过RL扩展工具集成推理(TIR)仍然具有挑战性,因为交互崩溃:一种病理状态,其中模型无法维持多轮工具使用,反而退化为大量内部推理,仅进行事后代码验证。我们系统地研究了三个问题:(i)冷启动SFT如何诱导代理、工具使用的行为先验;(ii)冷启动轨迹的交互密度如何影响探索和下游RL结果;(iii)RL交互预算如何影响学习动态和在不同推理时间预算下的泛化能力。我们随后引入ASTER(基于工具集成扩展推理的代理扩展),通过针对性的冷启动策略优先选择交互密集的轨迹来克服这种崩溃。我们发现,仅4K交互密集轨迹的小型专家冷启动集能够产生最强的下游性能,建立了一个稳健的先验,使在扩展RL训练期间实现更优的探索。广泛的评估显示,ASTER-4B在竞争性的数学基准上取得了最先进的结果,达到90.0%的AIME 2025成绩,超越了领先的前沿开源模型,包括DeepSeek-V3.2-Exp。
英文摘要:
Reinforcement learning (RL) has emerged as a dominant paradigm for eliciting long-horizon reasoning in Large Language Models (LLMs). However, scaling Tool-Integrated Reasoning (TIR) via RL remains challenging due to interaction collapse: a pathological state where models fail to sustain multi-turn tool usage, instead degenerating into heavy internal reasoning with only trivial, post-hoc code verification. We systematically study three questions: (i) how cold-start SFT induces an agentic, tool-using behavioral prior, (ii) how the interaction density of cold-start trajectories shapes exploration and downstream RL outcomes, and (iii) how the RL interaction budget affects learning dynamics and generalization under varying inference-time budgets. We then introduce ASTER (Agentic Scaling with Tool-integrated Extended Reasoning), a framework that circumvents this collapse through a targeted cold-start strategy prioritizing interaction-dense trajectories. We find that a small expert cold-start set of just 4K interaction-dense trajectories yields the strongest downstream performance, establishing a robust prior that enables superior exploration during extended RL training. Extensive evaluations demonstrate that ASTER-4B achieves state-of-the-art results on competitive mathematical benchmarks, reaching 90.0% on AIME 2025, surpassing leading frontier open-source models, including DeepSeek-V3.2-Exp.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。