arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.13151cs.AI
收藏

语言模型代理中的探索与利用误差是可测量的

Exploration and Exploitation Errors Are Measurable for Language Model Agents

University of Wisconsin–Madison(威斯康星大学麦迪逊分校);KRAFTON;Ludo Robotics

Jaden Park, Jungtaek Kim, Jongwon Jeong, Robert D. Nowak, Kangwook Lee, Yong Jae Lee

AI总结:

研究探讨了语言模型代理在复杂决策任务中探索与利用误差的测量方法,设计可控环境评估不同模型的表现,发现前沿模型在任务中表现不佳,推理模型更有效,通过少量工程优化可显著提升探索与利用能力。

摘要:

语言模型(LM)代理越来越多地用于复杂开放性决策任务,如AI编程和物理AI。在这些场景中,能够有效探索问题空间和利用已获取知识是核心要求。然而,系统区分和量化探索与利用,而无需访问代理内部策略,仍具挑战性。为此,我们设计了受实际具身AI场景启发的可控环境。每个环境由部分可观测的2D网格地图和未知任务的有向无环图(DAG)组成。地图生成可编程调整以强调探索或利用难度。为实现策略无关评估,我们设计了量化探索与利用误差的指标。我们评估了多种前沿LM代理,发现即使是最先进的模型在该任务上也表现不佳,不同模型表现出不同的失败模式。我们进一步发现,推理模型更有效地解决该任务,并且通过最小的工程优化可显著提升探索与利用能力。我们在此发布代码。

英文摘要:

Language Model (LM) agents are increasingly used in complex decision-making tasks, from AI coding to physical AI, where they must both explore a search space and exploit acquired knowledge effectively. Although exploration and exploitation play different roles in decision-making, it remains challenging to systematically distinguish exploration and exploitation without access to the agent's internal policy, and to determine when either constitutes an error. To address this, we design partially observable 2D grid maps paired with an unknown symbolic task directed acyclic graph (DAG), abstracting practical agentic AI scenarios. We then propose a policy-agnostic metric that detects unproductive traversals, while permitting necessary backtracking. We attribute each error to exploration or exploitation based on whether the current state admits productive exploration, exploitation, or both. Using generated maps and task DAGs with controllable difficulty, we evaluate 35 frontier LM agents and find that even state-of-the-art models struggle to reach the goal within the step budget. $\log$ exploration and exploitation errors exhibit strong relationships with task success ($R^2 = 0.885$ and $0.875$, respectively), providing empirical evidence that the metric captures behavior relevant to task success. Beyond success rate, the metric reveals distinct exploration and exploitation behaviors among models with similar success rates. We further find that a compact memory harness achieves higher task success than a reasoning baseline that retains its own reasoning traces. Finally, we extend our metric to an iconic video game, Minecraft Clone, beyond our synthetic environment. We release our code at https://github.com/jjj-madison/measurable-explore-exploit.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑