arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.21798cs.ROcs.AI
收藏

从人类视频到机器人操作的跨 embodiments 因果迁移

Cross-Embodiment Causal Transfer from Human Videos to Robot Manipulation

University of Chinese Academy of Sciences(中国科学院大学);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);Meituan(美团)

Xingyu Zhang, Yingchao Li, Peizheng Guo, Jie Hu, Changwen Zheng, Wenwen Qiang

AI总结:

该研究针对机器人操作数据收集成本高的问题,提出CBST方法,通过因果束约束从人类视频中迁移操作知识,使机器人无需动作对应即可生成动作,提升了基准任务的成功率。

摘要:

扩展机器人操作模型需要大量交互数据,但收集多样化的真实机器人经验成本高昂。相比之下,人类操作视频提供了丰富多样的任务经验,可能有益于机器人学习。然而,形态、动作空间和控制机制的差异使得人类动作不适合直接作为机器人动作的监督,这引发了一个科学问题:人类视频中的哪些知识应跨 embodiments 迁移到机器人操作?为解决该问题,我们从因果束的角度分析子任务迁移,重点关注应保留哪些上下文条件和交互依赖以支持局部任务效应。基于此视角,我们提出因果束引导的子任务迁移(Causal-Beam-Guided Subtask Transfer,CBST)。CBST 从人类视频中学习与操作相关的表征,并利用机器人演示将其锚定到任务级语义变量。在预定义的操作模型内,因果束约束通过要求保留的赋值维持局部机制输出,同时对保留的输入保持非平凡响应性,来指导上下文信息选择。所选语义信息为世界-动作模型提供条件,使机器人能从自身观测生成动作,无需人类-机器人动作对应。在机器人操作基准上的实验表明,与对应的世界-动作模型基线相比,任务成功率有所提升。

英文摘要:

Scaling robot manipulation models requires large amounts of interaction data, yet collecting diverse real-robot experience remains costly. Human manipulation videos, in contrast, provide abundant and diverse task experience that could potentially benefit robot learning. However, differences in morphology, action spaces, and control mechanisms make human motions unsuitable as direct supervision for robot actions, raising a scientific question: what knowledge in human videos should be transferred across embodiments to robot manipulation? To address this question, we analyze subtask transfer through the lens of causal beams, focusing on which contextual conditions and interaction dependencies should be retained to support local task effects. Based on this perspective, we propose Causal-Beam-Guided Subtask Transfer (CBST). CBST learns manipulation-related representations from human videos and grounds them in task-level semantic variables using robot demonstrations. Within a predefined manipulation model, causal-beam constraints guide contextual information selection by requiring retained assignments to sustain local mechanism outputs while preserving nontrivial responsiveness to retained inputs. The selected semantic information conditions a world-action model, allowing the robot to generate actions from its own observations without requiring human-robot motion correspondence. Experiments on robot manipulation benchmarks show improvements in task success over the corresponding world-action model baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑