Optimus-2：基于目标-观测-动作条件策略的多模态 Minecraft 智能体
Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy
AI总结：
Optimus-2 提出结合多模态大语言模型高层规划与目标-观测-动作条件策略低层控制的 Minecraft 智能体，并构建含 3000 万目标-观测-动作对的 MGOA 数据集，在原子、长时程和开放式指令任务上均取得优越性能。
摘要：
构建一个能够模仿人类行为模式以完成各种开放世界任务的智能体是一个长期目标。为了使智能体能够有效地学习跨多样化任务的行为模式，一个关键挑战在于建模观测、动作和语言之间错综复杂的关系。为此，我们提出了 Optimus-2，一种新颖的 Minecraft 智能体，它结合了用于高层规划的多模态大语言模型（MLLM），以及用于低层控制的目标-观测-动作条件策略（GOAP）。GOAP 包含：（1）一个动作引导的行为编码器，它在每个时间步建模观测与动作之间的因果关系，然后与历史观测-动作序列动态交互，将其整合为固定长度的行为 token；（2）一个 MLLM，它将行为 token 与开放式语言指令对齐，以自回归方式预测动作。此外，我们引入了一个高质量的 Minecraft 目标-观测-动作（MGOA）数据集，该数据集包含 8 个原子任务中的 25,000 个视频，提供约 3000 万个目标-观测-动作对。该自动化构建方法以及 MGOA 数据集可以为社区训练 Minecraft 智能体的工作做出贡献。大量实验结果表明，Optimus-2 在 Minecraft 中的原子任务、长时程任务和开放式指令任务上均表现出卓越的性能。请参见项目页面 https://cybertronagent.github.io/Optimus-2.github.io/。
英文摘要：
Building an agent that can mimic human behavior patterns to accomplish various open-world tasks is a long-term goal. To enable agents to effectively learn behavioral patterns across diverse tasks, a key challenge lies in modeling the intricate relationships among observations, actions, and language. To this end, we propose Optimus-2, a novel Minecraft agent that incorporates a Multimodal Large Language Model (MLLM) for high-level planning, alongside a Goal-Observation-Action Conditioned Policy (GOAP) for low-level control. GOAP contains (1) an Action-guided Behavior Encoder that models causal relationships between observations and actions at each timestep, then dynamically interacts with the historical observation-action sequence, consolidating it into fixed-length behavior tokens, and (2) an MLLM that aligns behavior tokens with open-ended language instructions to predict actions auto-regressively. Moreover, we introduce a high-quality Minecraft Goal-Observation-Action (MGOA)} dataset, which contains 25,000 videos across 8 atomic tasks, providing about 30M goal-observation-action pairs. The automated construction method, along with the MGOA dataset, can contribute to the community's efforts to train Minecraft agents. Extensive experimental results demonstrate that Optimus-2 exhibits superior performance across atomic tasks, long-horizon tasks, and open-ended instruction tasks in Minecraft. Please see the project page at https://cybertronagent.github.io/Optimus-2.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
