APT：面向开放世界智能体的、使用大型语言模型进行建筑规划与文本到蓝图构建
APT: Architectural Planning and Text-to-Blueprint Construction Using Large Language Models for Open-World Agents
AI总结：
本文提出 LLM 驱动的 APT 框架，通过思维链、多模态输入、记忆与反思让 Minecraft 智能体规划并构建复杂建筑，并配套基准验证其性能与涌现能力。
摘要：
我们提出 APT，这是一种先进的、由大型语言模型（LLM）驱动的框架，使自主智能体能够在 Minecraft 环境中构建复杂且富有创造性的结构。以往方法主要聚焦于基于技能的开放世界任务，或依赖基于图像的扩散模型生成基于体素的结构；与这些方法不同，我们的方法利用 LLM 内在的空间推理能力。通过采用思维链分解以及多模态输入，该框架生成详细的建筑布局和蓝图，使智能体能够在零样本或少样本学习场景下执行。我们的智能体同时集成记忆模块与反思模块，以在整个建造过程中支持终身学习、自适应细化和纠错。为严格评估智能体在这一新兴研究领域中的表现，我们引入一个综合基准，其中包含多样化的建造任务，旨在测试创造性、空间推理、对游戏内规则的遵守，以及多模态指令的有效整合。使用多种基于 GPT 的 LLM 后端和智能体配置得到的实验结果表明，该智能体能够准确解释涉及大量物品及其位置和朝向的冗长指令。智能体成功生成复杂结构，并具备红石驱动系统等内部功能。A/B 测试表明，加入记忆模块会带来性能的显著提升，凸显其在实现持续学习和复用累积经验方面的作用。此外，智能体出人意料地涌现出脚手架行为，这凸显了未来由 LLM 驱动的智能体利用子程序规划、并借助 LLM 的涌现能力自主发展类人问题解决技巧的潜力。
英文摘要：
We present APT, an advanced Large Language Model (LLM)-driven framework that enables autonomous agents to construct complex and creative structures within the Minecraft environment. Unlike previous approaches that primarily concentrate on skill-based open-world tasks or rely on image-based diffusion models for generating voxel-based structures, our method leverages the intrinsic spatial reasoning capabilities of LLMs. By employing chain-of-thought decomposition along with multimodal inputs, the framework generates detailed architectural layouts and blueprints that the agent can execute under zero-shot or few-shot learning scenarios. Our agent incorporates both memory and reflection modules to facilitate lifelong learning, adaptive refinement, and error correction throughout the building process. To rigorously evaluate the agent's performance in this emerging research area, we introduce a comprehensive benchmark consisting of diverse construction tasks designed to test creativity, spatial reasoning, adherence to in-game rules, and the effective integration of multimodal instructions. Experimental results using various GPT-based LLM backends and agent configurations demonstrate the agent's capacity to accurately interpret extensive instructions involving numerous items, their positions, and orientations. The agent successfully produces complex structures complete with internal functionalities such as Redstone-powered systems. A/B testing indicates that the inclusion of a memory module leads to a significant increase in performance, emphasizing its role in enabling continuous learning and the reuse of accumulated experience. Additionally, the agent's unexpected emergence of scaffolding behavior highlights the potential of future LLM-driven agents to utilize subroutine planning and leverage the emergence ability of LLMs to autonomously develop human-like problem-solving techniques.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
