arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.24828cs.AI
收藏

测试时深度思考以探索隐式规则

Test-Time Deep Thinking to Explore Implicit Rules

Renmin University of China(中国人民大学);Department of Statistics and Data Science, Tsinghua University(清华大学统计与数据科学系);School of Computer Science and Engineering, UESTC(UESTC计算机科学与工程学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);School of Mathematical Sciences, Nankai University(南开大学数学科学学院);Whiting School, Johns Hopkins University(约翰斯·霍普金斯大学惠特林学院);School of Artificial Intelligence, Shanghai Jiaotong University(上海交通大学人工智能学院)

Wentong Chen, Xin Cong, Zhong Zhang, Yaxi Lu, Siyuan Zhao, Yesai Wu, Qinyu Luo, Haotian Chen, Yankai Lin, Zhiyuan Liu, Maosong Sun

AI总结:

针对智能体在隐式规则环境中失败的问题,提出TTExplore框架,通过训练专用模型Exp-Thinker进行测试时推理,平均提升基线性能14-19点。

摘要:

随着大型语言模型(LLMs)的不断进步,智能体变得越来越重要。然而,这些智能体在由隐式规则——无法直接观察、必须通过交互推断的隐藏约束——支配的环境中常常失败。这导致智能体陷入重复的试错循环,最终导致任务失败。为了应对这一挑战,我们提出了测试时探索(TTExplore)框架,其中思考者组件分析交互历史以推断这些隐式规则并指导行动者。在此设置中,有效的探索关键取决于思考者的推理能力。然而,评估深度推理轨迹本质上不稳定且困难,这对有效训练构成了主要障碍。为了解决这个问题,我们引入了一种新颖且稳定的强化学习流程。核心思想是使用准确的任务级分数作为间接奖励,以绕过评估中间推理的困难,并仅保留每个轨迹的单个思考节点以缓解奖励稀疏性。使用此流程,我们训练了一个专门的7B模型Exp-Thinker。在五个基于文本的具体任务上的实验表明,配备Exp-Thinker的TTExplore将基线智能体性能平均提升了14-19个点,证明了显式推理隐式规则的有效性。

英文摘要:

With the continuous advancement of Large Language Models (LLMs), intelligent agents are becoming increasingly vital. However, these agents often fail in environments governed by implicit rules--hidden constraints that cannot be observed directly and must be inferred through interaction. This causes agents to fall into repetitive trial-and-error loops, ultimately leading to task failure. To address this challenge, we propose Test-Time Exploration (TTExplore), a framework where a thinker component analyzes interaction history to infer these implicit rules and guide an actor. Effective exploration in this setting critically depends on the reasoning ability of the thinker. However, evaluating deep reasoning trajectories is inherently unstable and difficult, which poses a major obstacle to effective training. To overcome this issue, we introduce a novel and stable reinforcement learning pipeline. The core idea is to use accurate task-level scores as indirect rewards to bypass the difficulty of evaluating intermediate reasoning, and to retain only a single thinking node per trajectory to alleviate reward sparsity. Using this pipeline, we train a specialized 7B model, Exp-Thinker. Experiments on five text-based embodied tasks show that TTExplore equipped with Exp-Thinker improves baseline agent performance by an average of $14$-$19$ points, demonstrating the effectiveness of explicitly reasoning about implicit rules.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑