GSM-Agent: 利用可控环境理解智能体推理
GSM-Agent: Understanding Agentic Reasoning Using Controllable Environments
AI总结：
该研究提出用于评估智能体推理的基准GSM-Agent，发现前沿模型在该基准上准确率仅67%，进而提出智能体推理图概念并发现模型普遍缺失重访节点能力，最终提出工具增强测试时缩放方法以提升性能。
摘要：
随着大语言模型（LLM）越来越多地被部署为智能体，智能体推理——即结合工具使用（尤其是搜索）与推理的能力——已成为一项关键技能。然而，在复杂环境与任务中进行评估时，很难将智能体推理与其他能力分离开来。当前的智能体基准通常将智能体推理与高难度数学推理、专家级知识及其他高级能力混杂在一起。为填补这一空白，我们构建了一个全新的基准GSM-Agent：在该基准中，大语言模型智能体需要解决小学阶段的推理问题，但提示中仅向其呈现问题本身，不包含解决任务所需的前提信息，智能体需要主动使用工具收集这些信息。尽管原始任务是小学数学题，我们观察到即使是GPT-5这类前沿模型也仅能达到67%的准确率。为理解并分析智能体推理模式，我们提出了智能体推理图的概念：将环境的文档嵌入聚类为节点，并将每次工具调用映射到其最近的节点，以此构建推理路径。令人意外的是，我们发现在静态推理中被普遍视为关键模式的“重访先前访问过的节点”的能力，在许多模型的智能体推理中往往是缺失的。基于这一洞见，我们提出了一种工具增强的测试时缩放方法，通过添加工具鼓励模型进行重访，以提升大语言模型的智能体推理性能。我们期望该基准与智能体推理框架能够助力未来关于理解并拓展智能体推理边界的研究。
英文摘要：
As LLMs are increasingly deployed as agents, agentic reasoning - the ability to combine tool use, especially search, and reasoning - becomes a critical skill. However, it is hard to disentangle agentic reasoning when evaluated in complex environments and tasks. Current agent benchmarks often mix agentic reasoning with challenging math reasoning, expert-level knowledge, and other advanced capabilities. To fill this gap, we build a novel benchmark, GSM-Agent, where an LLM agent is required to solve grade-school-level reasoning problems, but is only presented with the question in the prompt without the premises that contain the necessary information to solve the task, and needs to proactively collect that information using tools. Although the original tasks are grade-school math problems, we observe that even frontier models like GPT-5 only achieve 67% accuracy. To understand and analyze the agentic reasoning patterns, we propose the concept of agentic reasoning graph: cluster the environment's document embeddings into nodes, and map each tool call to its nearest node to build a reasoning path. Surprisingly, we identify that the ability to revisit a previously visited node, widely taken as a crucial pattern in static reasoning, is often missing for agentic reasoning for many models. Based on the insight, we propose a tool-augmented test-time scaling method to improve LLM's agentic reasoning performance by adding tools to encourage models to revisit. We expect our benchmark and the agentic reasoning framework to aid future studies of understanding and pushing the boundaries of agentic reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
