ROCKET-1: 基于视觉-时序上下文提示掌握开放世界交互
ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting
AI总结：
针对视觉语言模型适配开放世界具身决策时语言难传空间信息的问题，提出视觉-时序上下文提示协议，结合SAM-2目标跟踪训练低层策略ROCKET-1，在Minecraft中实现76%的开放世界交互性能绝对提升。
摘要：
视觉语言模型（VLM）在多模态任务中表现出色，但将其适配到开放世界环境中的具身决策仍存在挑战。一个关键问题是弥合低层观测中的离散实体与有效规划所需的抽象概念之间的差距。常见解决方案是构建分层智能体，其中VLM作为高层推理器将任务分解为可执行子任务，通常用语言指定。然而，语言无法传递详细的空间信息。我们提出视觉-时序上下文提示，这是一种VLM与策略模型之间的新型通信协议。该协议利用过去观测中的目标分割来引导策略与环境的交互。通过该方法，我们训练了低层策略ROCKET-1，它基于拼接的视觉观测和分割掩码预测动作，并由SAM-2的实时目标跟踪提供支持。我们的方法释放了VLM的潜力，使其能够处理需要空间推理的复杂任务。在Minecraft中的实验表明，我们的方法使智能体能够完成此前无法实现的任务，开放世界交互性能绝对提升了76%。代码和演示现已在项目页面发布：https://craftjarvis.github.io/ROCKET-1。
英文摘要：
Vision-language models (VLMs) have excelled in multimodal tasks, but adapting them to embodied decision-making in open-world environments presents challenges. One critical issue is bridging the gap between discrete entities in low-level observations and the abstract concepts required for effective planning. A common solution is building hierarchical agents, where VLMs serve as high-level reasoners that break down tasks into executable sub-tasks, typically specified using language. However, language suffers from the inability to communicate detailed spatial information. We propose visual-temporal context prompting, a novel communication protocol between VLMs and policy models. This protocol leverages object segmentation from past observations to guide policy-environment interactions. Using this approach, we train ROCKET-1, a low-level policy that predicts actions based on concatenated visual observations and segmentation masks, supported by real-time object tracking from SAM-2. Our method unlocks the potential of VLMs, enabling them to tackle complex tasks that demand spatial reasoning. Experiments in Minecraft show that our approach enables agents to achieve previously unattainable tasks, with a $\mathbf{76}\%$ absolute improvement in open-world interaction performance. Codes and demos are now available on the project page: https://craftjarvis.github.io/ROCKET-1.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
