MemoryVAM：将记忆融入视频动作模型以实现机器人操作
MemoryVAM: Integrating Memory into Video Action Model for Robot Manipulation
AI总结：
提出MemoryVAM，通过Recap-Cue模块将情景记忆注入视频世界模型策略，解决长时域操作中的非马尔可夫性问题，在LIBERO-Mem和真实机器人任务上显著提升成功率。
摘要：
视频世界模型策略通过预测未来观测来学习与动作相关的表示。然而，它们仅依赖于短观测窗口，当正确动作取决于不再可见的早期事件时，长时域操作变得非马尔可夫。我们提出MemoryVAM，一种用于视频世界模型策略的情景记忆机制。我们采用Recap-Cue（RC）模块，其中基于Perceiver的Recap压缩器将每帧CLIP嵌入映射为紧凑的记忆令牌，轻量级Cue门从记忆和语言估计任务完成度。这些令牌被注入视频骨干网络和动作解码器，使策略想象与情节进展对齐，并根据历史条件化动作。我们的模型通过视频预测、增量重建辅助损失和情节边界监督来训练记忆模块，无需每帧进度标签。同一机制通过仅更改交叉注意力注入接口适用于UNet和扩散变换器（DiT）骨干网络。在LIBERO-Mem上，我们的模型将平均成功率从5%提升至42.5%。在真实机器人上，计数任务成功率达78.3%，空间回忆达80.0%，顺序跟踪达75.0%。项目页面：此https URL
英文摘要：
Video-world-model policies learn action-relevant representations by predicting future observations. However, they condition on only a short observation window, which renders long-horizon manipulation non-Markovian when the correct action depends on earlier events that are no longer visible. We present MemoryVAM, an episodic memory mechanism for video-world-model policies. We employ a Recap-Cue (RC) module, in which a Perceiver-based Recap Compressor maps per-frame CLIP embeddings into compact memory tokens, and a lightweight Cue Gate estimates task completion from memory and language. These tokens are injected into both the video backbone and the action decoder, aligning policy imagination with episode progress and conditioning actions on history. Our model trains the memory module with video prediction, a delta-reconstruction auxiliary loss, and episode-boundary supervision, requiring no per-frame progress labels. The same mechanism applies to UNet and Diffusion Transformer (DiT) backbones by changing only the cross-attention injection interface. On LIBERO-Mem, our model improves average success from 5% to 42.5%. On real robots, it achieves 78.3% success on counting tasks, 80.0% on spatial recall, and 75.0% on sequential tracking. Project page: https://MemoryVAM.github.io/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
