arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2506.07976cs.LGcs.AI
收藏

思考与行动:通过扩展测试时交互进行推理的智能体

Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction

Carnegie Mellon University(卡内基梅隆大学);Scribe(Scribe公司);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Toronto(多伦多大学);University of California, Berkeley(加州大学伯克利分校);The AGI Company(AGI公司);New York University(纽约大学)

Junhong Shen, Hao Bai, Lunjun Zhang, Yifei Zhou, Amrith Setlur, Shengbang Tong, Diego Caples, Nan Jiang, Tong Zhang, Ameet Talwalkar, Aviral Kumar

AI总结:

针对网络智能体难以从环境获取信息并动态调整的问题,提出扩展测试时交互及基于课程在线强化学习的 TTI,在 WebVoyager 和 WebArena 上取得开源最优表现。

摘要:

当前的测试时扩展范式依赖于在给出响应之前生成冗长的推理轨迹(即更多“思考”)。在需要交互的智能体问题中,可以通过在现实世界中行动之前生成思考轨迹来实现这一点。然而,这一过程不允许智能体从环境中获取新信息,也不允许其随时间调整行为。在本研究中,我们提出扩展测试时交互,这是测试时扩展中一个尚未开发的维度,通过增加智能体的交互视野,使其能够在单次轨迹执行中运行探索、回溯和动态重新规划等丰富行为。为证明该扩展维度的前景,我们研究了网络智能体领域。我们首先表明,即使是未经任何训练、基于提示的交互扩展,也能显著提升网络基准测试中的任务成功率。在此基础上,我们提出 TTI(Test-Time Interaction,测试时交互),这是一种基于课程的在线强化学习(RL)方法,通过自适应调整智能体的轨迹长度来训练智能体。使用 Gemma 3 12B 模型,TTI 在 WebVoyager 和 WebArena 基准测试上打造出了最先进的开源、开放数据网络智能体。我们进一步表明,TTI 使智能体能够自适应地平衡探索与利用。我们的结果证明,交互扩展是与逐步增加计算扩展相辅相成的强大轴线,为训练自适应智能体提供了新途径。

英文摘要:

The current paradigm of test-time scaling relies on generating long reasoning traces ("thinking" more) before producing a response. In agent problems that require interaction, this can be done by generating thinking traces before acting in the world. However, this process does not allow agents to acquire new information from the environment or adapt their behavior over time. In this work, we propose to scale test-time interaction, an untapped dimension of test-time scaling that increases the agent's interaction horizon to enable running rich behaviors such as exploration, backtracking, and dynamic re-planning within a single rollout. To demonstrate the promise of this scaling dimension, we study the domain of web agents. We first show that even prompting-based interaction scaling without any training can improve task success on web benchmarks non-trivially. Building on this, we introduce TTI (Test-Time Interaction), a curriculum-based online reinforcement learning (RL) approach that trains agents by adaptively adjusting their rollout lengths. Using a Gemma 3 12B model, TTI produces state-of-the-art open-source, open-data web agents on WebVoyager and WebArena benchmarks. We further show that TTI enables agents to balance exploration and exploitation adaptively. Our results establish interaction scaling as a powerful, complementary axis to scaling per-step compute, offering new avenues for training adaptive agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑