机器人学习中隐式动作的关键影响因素
What Matters for Latent Actions in Robot Learning
浏览论文内容
中文总结 AI 辅助
本研究通过统一框架整合代表性隐式动作模型，系统探究41种设计选择，发现用隐式动作微调视觉语言模型主干可为下游机器人操纵策略学习提供更强初始化。
中文摘要 AI 辅助
隐式动作模型（LAMs）已成为一种极具前景的范式，可使机器人学习通过隐式动作利用大规模未标记视频，这些隐式动作是物理动作的紧凑替代表示。尽管进展迅速，但LAM的研究仍高度分散，现有方法在不一致的实验设置下孤立评估不同设计选择，导致难以确定真正决定下游机器人操纵性能的因素。本研究首次对机器人操纵的隐式动作学习开展全面实证研究，在统一的自动编码框架内整合代表性LAM方法，系统探究三个维度的41种LAM设计选择，包括隐式动作建模范式、学习目标与正则化方法、隐式动作整合策略。进一步考察评估隐式动作质量的四种代理指标，验证其可靠预测下游机器人操纵性能的能力。在三个广泛使用的基准上开展的大量实验提供了有力实证证据：通过隐式动作微调视觉语言模型（VLM）主干网络可为下游策略学习提供更强的初始化，该结论在真实世界机器人操纵任务中得到进一步验证。
英文摘要
Latent Action Models (LAMs) have emerged as a promising paradigm for enabling robot learning to leverage large-scale unlabeled videos through latent actions that serve as compact surrogates for physical actions. Despite rapid progress, research on LAM remains highly fragmented, with existing methods evaluating different design choices in isolation under inconsistent experimental settings, making it difficult to identify the factors that truly determine downstream robotic manipulation performance. In this work, we present the first comprehensive empirical study of latent action learning for robotic manipulation. We unify representative LAM methods within a common autoencoding framework and systematically investigate 41 LAM design choices across three dimensions, including latent action modeling paradigms, learning objectives and regularization methods, and latent action integration strategies. We further examine four proxy metrics for evaluating latent action quality and assess their ability to reliably predict downstream robotic manipulation performance. Extensive experiments on three widely used benchmarks provide strong empirical evidence that fine-tuning vision-language model (VLM) backbones with latent actions provides a stronger initialization for downstream policy learning, with further validation on real-world robot manipulation tasks.
发表机构
- College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）
- Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
- Guangdong Provincial Key Laboratory of Computility Microelectronics, Faculty of Computility Microelectronics, Shenzhen University of Advanced Technology（深圳先进技术研究院计算机微科学与技术学院、广东省计算微电子重点实验室）
- School of Aeronautics and Astronautics, Sichuan University（四川大学航空航天学院）
- Suzhou Evans Intelligent Technology Co., Ltd.（苏州伊文斯智能科技有限公司）
- Morphi Intelligence Technology Co., Ltd.（墨飞智能科技有限公司）
- School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
- Xiaomi EV（小米汽车）
机构由 AI 辅助整理，请以论文原文为准。
