LaVA-Man：学习用于机器人操作的视觉动作表征
LaVA-Man: Learning Visual Action Representations for Robot Manipulation
AI总结：
本文提出 LaVA-Man，通过遮蔽目标图像重建的自监督任务学习视觉-动作表征，并构建 Omni-Object Pick-and-Place 数据集，在五个仿真与真实机器人基准上提升语言引导操作性能。
摘要：
视觉-文本理解对于语言引导的机器人操作至关重要。近期工作利用预训练视觉-语言模型来衡量编码后的视觉观测与文本指令之间的相似性，然后训练模型将这种相似性映射为机器人动作。然而，这种两步方法限制了模型捕捉视觉观测与文本指令之间关系的能力，导致操作任务精度下降。我们提出通过自监督代理任务来学习视觉-文本关联：在输入图像和文本指令的条件下重建被遮蔽的目标图像。该形式使模型无需机器人动作监督即可学习视觉-动作表征。随后，仅需少量示范即可对学到的表征进行微调，以用于操作任务。我们还推出了 Omni-Object Pick-and-Place 数据集，该数据集由带标注的机器人桌面操作片段组成，包含180个对象类别和3,200个实例，并配有相应的文本指令。该数据集使模型能够获得多样的对象先验，并可更全面地评估其跨对象实例的泛化能力。在五个基准上的实验结果（包括仿真和真实机器人验证）表明，我们的方法优于现有技术。
英文摘要：
Visual-textual understanding is essential for language-guided robot manipulation. Recent works leverage pre-trained vision-language models to measure the similarity between encoded visual observations and textual instructions, and then train a model to map this similarity to robot actions. However, this two-step approach limits the model to capture the relationship between visual observations and textual instructions, leading to reduced precision in manipulation tasks. We propose to learn visual-textual associations through a self-supervised pretext task: reconstructing a masked goal image conditioned on an input image and textual instructions. This formulation allows the model to learn visual-action representations without robot action supervision. The learned representations can then be fine-tuned for manipulation tasks with only a few demonstrations. We also introduce the \textit{Omni-Object Pick-and-Place} dataset, which consists of annotated robot tabletop manipulation episodes, including 180 object classes and 3,200 instances with corresponding textual instructions. This dataset enables the model to acquire diverse object priors and allows for a more comprehensive evaluation of its generalisation capability across object instances. Experimental results on the five benchmarks, including both simulated and real-robot validations, demonstrate that our method outperforms prior art.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
