Language Model (LM) agents are increasingly used in complex decision-making tasks, from AI coding to physical AI, where they must both explore a search space and exploit acquired knowledge effectively. Although exploration and exploitation play different roles in decision-making, it remains challenging to systematically distinguish exploration and exploitation without access to the agent's internal policy, and to determine when either constitutes an error. To address this, we design partially observable 2D grid maps paired with an unknown symbolic task directed acyclic graph (DAG), abstracting practical agentic AI scenarios. We then propose a policy-agnostic metric that detects unproductive traversals, while permitting necessary backtracking. We attribute each error to exploration or exploitation based on whether the current state admits productive exploration, exploitation, or both. Using generated maps and task DAGs with controllable difficulty, we evaluate 35 frontier LM agents and find that even state-of-the-art models struggle to reach the goal within the step budget. $\log$ exploration and exploitation errors exhibit strong relationships with task success ($R^2 = 0.885$ and $0.875$, respectively), providing empirical evidence that the metric captures behavior relevant to task success. Beyond success rate, the metric reveals distinct exploration and exploitation behaviors among models with similar success rates. We further find that a compact memory harness achieves higher task success than a reasoning baseline that retains its own reasoning traces. Finally, we extend our metric to an iconic video game, Minecraft Clone, beyond our synthetic environment. We release our code at https://github.com/jjj-madison/measurable-explore-exploit.
Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shaping
University of Cambridge(剑桥大学);Tencent RoboticsX(腾讯机器人X实验室);Hong Kong University of Science and Technology(香港科技大学);Tsinghua University(清华大学);IDEA(粤港澳大湾区数字经济研究院);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
Department of Electrical-Engineering, Technion Institute of Technology, Israel(电气工程系,技术学院技术研究所,以色列);Nvidia Research(Nvidia研究);Technion (currently at Google Research)(技术学院(目前在谷歌研究))
HyPER: Bridging Exploration and Exploitation for Scalable LLM Reasoning with Hypothesis Path Expansion and Reduction
Institute for Artificial Intelligence, Peking University, Beijing(北京大学人工智能研究院);Huawei(华为);School of Integrated Circuits, Peking University, Beijing(北京大学集成电路学院)
An Improved FOX Optimization Algorithm Using Adaptive Exploration and Exploitation for Global Optimization
College of Computer Science, University of Technology(技术大学计算机科学学院);Department of Computer Science and Engineering(大学工程学院;人工智能研究所,库尔德大学);AIIC, University of Kurdistan
Controlling Exploration-Exploitation in GFlowNets via Markov Chain Perspectives
LUMIA Lab, School of Artificial Intelligence, SJTU(LUMIA实验室,人工智能学院,上海交通大学);School of Mathematical Sciences, SJTU(数学科学学院,上海交通大学);Shanghai Advanced Institute of Finance, SJTU(上海先进金融研究院,上海交通大学);School of Computer Science, SJTU(计算机科学学院,上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Innovation Institute(上海创新研究院);Shanghai Center for Data Science, NYU Shanghai(上海数据科学中心,纽约大学上海学院);NYU-ECNU Institute of Mathematical Sciences, NYU Shanghai(纽约大学-华东师范大学数学科学研究院,纽约大学上海学院)
E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoning
Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学(深圳));Huawei Technologies Co., Ltd.(华为技术有限公司);Hong Kong University of Science and Technology(香港科技大学)
HTPO: Towards Exploration-Exploitation Balanced Policy Optimization via Hierarchical Token-level Objective Control
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University(上海交通大学信息科学与电子工程学院);Xiaohongshu Inc.(小红书公司);MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University(上海交通大学人工智能研究院)
DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management
School of Mathematics and Statistics(数学与统计学学院);Beijing Jiaotong University(北京交通大学);Yau Mathematical Sciences Center(叶中科学中心);Tsinghua University(清华大学);Department of Mathematical Sciences(数学科学系);Beijing, China(中国北京);Zhongguancun Academy(中关村学院);Yanqi Lake Beijing Institute of Mathematical Sciences and Applications(燕琪湖数学科学与应用研究所);Academy of Mathematics and Systems Science(数学与系统科学研究院);Chinese Academy of Sciences(中国科学院);Microsoft Research AI4Science(微软研究院AI4Science)
$ϕ$-Decoding: Adaptive Foresight Sampling for Balanced Inference-Time Exploration and Exploitation
Shanghai AI Lab(上海人工智能实验室);Xi’an Jiaotong University(西安交通大学);The University of Hong Kong(香港大学);Peking University(北京大学);National University of Singapore(新加坡国立大学)