MEM：多尺度具身记忆用于视觉语言动作模型
MEM: Multi-Scale Embodied Memory for Vision Language Action Models
AI总结：
MEM通过结合多模态记忆实现长期视觉语言动作任务的高效执行与适应性策略调整。
摘要：
传统上，端到端机器人学习中的记忆涉及将一系列过去的观察输入到学习的策略中。然而，在复杂的多阶段现实任务中，机器人的记忆必须以多个粒度级别表示过去的事件：从长期记忆，它捕捉抽象的语义概念（例如，机器人做饭时应记住菜谱的哪些阶段已经完成），到短期记忆，它捕捉最近的事件并补偿遮挡（例如，机器人在手臂遮挡时记住它想要拿起的物体）。在本工作中，我们的主要见解是，一种有效的长期机器人控制记忆架构应结合多种模态来捕捉这些不同的抽象层次。我们引入了多尺度具身记忆（MEM），一种用于混合模态长期记忆的机器人策略方法。MEM结合基于视频的短期记忆（通过视频编码器压缩）和基于文本的长期记忆。共同作用，使机器人策略能够执行长达十五分钟的任务，如清理厨房或准备煎芝士三明治。此外，我们发现记忆使MEM策略能够智能地在上下文中适应操作策略。
英文摘要：
Conventionally, memory in end-to-end robotic learning involves inputting a sequence of past observations into the learned policy. However, in complex multi-stage real-world tasks, the robot's memory must represent past events at multiple levels of granularity: from long-term memory that captures abstracted semantic concepts (e.g., a robot cooking dinner should remember which stages of the recipe are already done) to short-term memory that captures recent events and compensates for occlusions (e.g., a robot remembering the object it wants to pick up once its arm occludes it). In this work, our main insight is that an effective memory architecture for long-horizon robotic control should combine multiple modalities to capture these different levels of abstraction. We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies. MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory. Together, they enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich. Additionally, we find that memory enables MEM policies to intelligently adapt manipulation strategies in-context.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
