发表机构
Tsinghua University; Microsoft Research; Fudan University; USTC（清华大学; 微软研究院; 复旦大学; 中国科学技术大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究针对视觉-语言-动作模型在接触丰富操作中的时间上下文推理问题，提出FM-VLA模型，通过基于力的记忆编码及投影，利用累积接触事件历史指导操作，在相关任务上评估，轻量级力记忆表现出色，显著优于基线方法。
AI 中文摘要
视觉-语言-动作（VLA）模型在机器人操作中实现了令人印象深刻的泛化，近期基于记忆增强的VLA通过以过去图像或语言摘要为条件放宽了马尔可夫假设。基于视觉的记忆方法通过对采样的过去图像帧进行条件设定来解决此问题，但在时间事件视觉模糊时计算成本高且有根本限制。我们提出FM-VLA，一种具有基于力的记忆的VLA模型，用于非马尔可夫、接触丰富操作的时间上下文推理。我们用变分自编码器（VAE）将力历史编码为紧凑的力记忆令牌，VAE通过力时间序列重建进行预训练。通过将力潜在表示和短状态历史投影为动作专家模块的额外条件令牌，使VLA能够利用累积的接触事件历史来指导操作。我们在三个依赖记忆的任务上评估FM-VLA，包括找到隐藏块、按按钮和擦拭盘子特定次数。我们的轻量级力记忆以最小推理开销实现了超过80%的成功率，显著优于基线方法。
英文摘要
Vision-language-action (VLA) models have achieved impressive generalization in robotic manipulation, and recent memory-augmented VLAs have relaxed the Markovian assumption by conditioning on past images or language summaries. Vision-based memory approaches address this by conditioning on sampled past image frames, but they are computationally expensive and fundamentally limited when temporal events are visually ambiguous, e.g., pushing a button multiple times with small movements. We propose FM-VLA, a VLA model with force-based memory, enabling temporal context reasoning for non-Markovian, contact-rich manipulation. We encode force histories into compact force memory tokens with a variational autoencoder (VAE) pretrained with force time series reconstruction. By projecting force latent representations and short state history as additional conditioning tokens to the action expert module, we enable VLAs to leverage accumulated contact event history to guide manipulation. We evaluate FM-VLA on three memory-dependent tasks, including finding a hidden block, pressing a button, and wiping a dish for a specific number of times. Our lightweight force memory achieves over 80% success rate with minimal inference overhead, significantly outperforming baseline approaches. Project page: https://qft-333.github.io/FM-VLA-Page/
