arXiv 2607.076082026-07-09cs.ROcs.CV

用于机器人操作的视觉-语言-动作模型中的双潜记忆

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

Nanjing University of Science and Technology（南京理工大学）；Zhejiang University（浙江大学）；National University of Singapore（新加坡国立大学）

Hongyu Qu, Jianzhe Gao, Xiaobin Hu, Shaohuan Yang, Xinlei Yu, Rui Yan, Wenguan Wang, Xiangbo Shu, Shuicheng Yan

备注：: Project page: https://github.com/quhongyu/LaMem-VLA

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

研究针对主流VLA模型处理长期任务的不足，提出LaMem-VLA框架，通过四个协调组件将历史经验重构为潜记忆令牌并与VLA推理直接交织，实现在同一潜空间处理历史经验，经实验验证该框架具有优越性。

摘要：

主流视觉-语言-动作（VLA）模型在马尔可夫假设下主要根据当前观察预测动作，在处理长期、时间相关任务时存在困难。现有增强记忆的VLA要么扩大观察窗口，要么从记忆库检索历史作为辅助策略背景，但记忆处于VLA推理的原生潜嵌入空间之外。为此引入LaMem-VLA，一个潜记忆原生框架，将历史经验重构为潜记忆令牌并与VLA推理直接交织。其核心有四个协调组件：组织者、探索者、压缩器和编织器。通过在同一连续潜空间中表示、检索和使用历史经验，LaMem-VLA使记忆能直接参与VLA推理并在有限背景下指导动作生成。在SimplerEnv和LIBERO上的大量实验证明了LaMem-VLA的优越性。

英文摘要：

Mainstream Vision-Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing memory-augmented VLAs either expand the observation window or retrieve history from the memory bank as auxiliary policy-side context. However, they leave memory outside the native latent embedding space of VLA reasoning, preventing historical experience from being fluidly interleaved with multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term memory vaults; (ii) a seeker that queries both vaults using the multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent memory tokens; and (iv) a weaver that injects these memory tokens with the current observation and instruction into one continuous embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables memory to directly participate in VLA reasoning and guide action generation under a bounded context. Extensive experiments on SimplerEnv and LIBERO demonstrate the superiority of our LaMem-VLA.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。