arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.07582cs.RO
收藏

一次观看,随后行动:基于单次视频示范的任务学习视觉-语言-行动模型

See Once, Then Act: Vision-Language-Action Model with Task Learning from One-Shot Video Demonstrations

Beijing Institute of Technology(北京理工大学);LimX Dynamics

Guangyan Chen, Meiling Wang, Qi Shao, Zichen Zhou, Weixin Mao, Te Cui, Minzhao Zhu, Yinan Deng, Luojie Yang, Zhanqi Zhang, Yi Yang, Hua Chen, Yufeng Yue

AI总结:

ViVLA通过单次视频示范高效学习机器人操控任务,实现跨任务和跨身体的显著性能提升。

摘要:

开发稳健且通用的操控策略是机器人研究中的基本目标。尽管视觉-语言-行动(VLA)模型在端到端机器人控制中展示了有希望的能力,但现有方法仍然在任务超出训练分布时表现出有限的泛化能力。相比之下,人类在仅观察他人一次执行新技能后就能获得显著的技能。受这一能力的启发,我们提出了ViVLA,一种通用的机器人操控策略,能够在测试时从单个专家示范视频中高效地学习任务。我们的方法联合处理专家示范视频和机器人视觉观测,以预测演示的动作序列和随后的机器人动作,从而有效地从专家行为中提取细粒度的操控知识,并将其无缝转移到智能体。为了提高ViVLA的性能,我们开发了一种可扩展的专家-智能体配对数据生成管道,能够从容易获取的人类视频中合成配对轨迹,进一步通过公共数据集中的精心挑选配对进行增强。该管道生成了总共892,911个专家-智能体样本用于训练ViVLA。实验结果表明,我们的ViVLA能够在测试时仅从单个专家示范视频中获得新的操控技能。我们的方法在未见过的LIBERO任务上实现了超过30%的改进,并在跨身体视频中保持了超过35%的改进。现实世界实验表明,从人类视频中有效学习,导致在未见过的任务上改进超过38%。

英文摘要:

Developing robust and general-purpose manipulation policies represents a fundamental objective in robotics research. While Vision-Language-Action (VLA) models have demonstrated promising capabilities for end-to-end robot control, existing approaches still exhibit limited generalization to tasks beyond their training distributions. In contrast, humans possess remarkable proficiency in acquiring novel skills by simply observing others performing them once. Inspired by this capability, we propose ViVLA, a generalist robotic manipulation policy that achieves efficient task learning from a single expert demonstration video at test time. Our approach jointly processes an expert demonstration video alongside the robot's visual observations to predict both the demonstrated action sequences and subsequent robot actions, effectively distilling fine-grained manipulation knowledge from expert behavior and transferring it seamlessly to the agent. To enhance the performance of ViVLA, we develop a scalable expert-agent pair data generation pipeline capable of synthesizing paired trajectories from easily accessible human videos, further augmented by curated pairs from publicly available datasets. This pipeline produces a total of 892,911 expert-agent samples for training ViVLA. Experimental results demonstrate that our ViVLA is able to acquire novel manipulation skills from only a single expert demonstration video at test time. Our approach achieves over 30% improvement on unseen LIBERO tasks and maintains above 35% gains with cross-embodiment videos. Real-world experiments demonstrate effective learning from human videos, yielding more than 38% improvement on unseen tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑