TowerMind: 一个用于LLM作为智能体的塔防游戏学习环境与基准
TowerMind: A Tower Defence Game Learning Environment and Benchmark for LLM as Agents
AI总结:
本文提出TowerMind,一个基于塔防子类型的轻量级、多模态游戏环境,用于评估大语言模型在长期规划和决策中的能力,并揭示其与人类专家的性能差距及关键局限性。
摘要:
近年来,大语言模型(LLM)的突破性进展使其成为智能体的一种有前景的范式,其中长期规划和决策作为适应不同场景和任务的核心通用能力逐渐凸显。实时策略(RTS)游戏因其固有的游戏玩法需要宏观战略规划和微观战术调整与行动执行,成为评估这两种能力的理想测试平台。现有的基于RTS游戏的环境要么计算需求较高,要么缺乏对文本观察的支持,这限制了RTS游戏在LLM评估中的应用。受此启发,我们提出了TowerMind,一种基于RTS游戏子类型——塔防(TD)的新型环境。TowerMind保留了RTS游戏评估LLM的关键优势,同时具有低计算需求和多模态观察空间,包括基于像素、文本和结构化游戏状态的表示。此外,TowerMind支持模型幻觉评估,并提供高度的可定制性。我们设计了五个基准关卡,以评估几种广泛使用的LLM在不同多模态输入设置下的表现。结果揭示了LLM与人类专家在能力和幻觉维度上的明显性能差距。实验进一步突出了LLM行为的关键局限性,例如规划验证不足、决策缺乏多终性以及行动使用效率低下。我们还评估了两种经典强化学习算法:Ape-X DQN和PPO。通过提供轻量级和多模态设计,TowerMind补充了现有的基于RTS游戏的环境格局,并为AI智能体领域引入了一个新的基准。源代码已在GitHub上公开(https://github.com/tb6147877/TowerMind)。
英文摘要:
Recent breakthroughs in Large Language Models (LLMs) have positioned them as a promising paradigm for agents, with long-term planning and decision-making emerging as core general-purpose capabilities for adapting to diverse scenarios and tasks. Real-time strategy (RTS) games serve as an ideal testbed for evaluating these two capabilities, as their inherent gameplay requires both macro-level strategic planning and micro-level tactical adaptation and action execution. Existing RTS game-based environments either suffer from relatively high computational demands or lack support for textual observations, which has constrained the use of RTS games for LLM evaluation. Motivated by this, we present TowerMind, a novel environment grounded in the tower defense (TD) subgenre of RTS games. TowerMind preserves the key evaluation strengths of RTS games for assessing LLMs, while featuring low computational demands and a multimodal observation space, including pixel-based, textual, and structured game-state representations. In addition, TowerMind supports the evaluation of model hallucination and provides a high degree of customizability. We design five benchmark levels to evaluate several widely used LLMs under different multimodal input settings. The results reveal a clear performance gap between LLMs and human experts across both capability and hallucination dimensions. The experiments further highlight key limitations in LLM behavior, such as inadequate planning validation, a lack of multifinality in decision-making, and inefficient action use. We also evaluate two classic reinforcement learning algorithms: Ape-X DQN and PPO. By offering a lightweight and multimodal design, TowerMind complements the existing RTS game-based environment landscape and introduces a new benchmark for the AI agent field. The source code is publicly available on GitHub(https://github.com/tb6147877/TowerMind).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。