本文介绍了 GET-Zero,这是一种模型架构与训练流程,用于学习具备具身感知能力的控制策略,该策略能够在无需重新训练的情况下立即适应新的硬件变化。为此,我们提出了 Graph Embodiment Transformer(GET,图具身 Transformer),这是一种 Transformer 模型,它将具身图连接结构作为注意力机制中的一种可学习结构偏置加以利用。我们采用行为克隆,将来自具身专属专家策略的演示数据蒸馏到一个具备具身感知能力的 GET 模型中,该模型以机器人的硬件配置为条件来作出控制决策。我们在灵巧的手中物体旋转任务上进行了案例研究,使用四指机器人手的不同配置,包括移除关节以及延长连杆长度。将 GET 模型与自建模损失结合使用,使 GET-Zero 能够零样本泛化到未见的图结构和连杆长度变化,相比基线方法取得了 20% 的提升。所有代码和定性视频结果均位于 https://get-zero-paper.github.io。
英文摘要:
This paper introduces GET-Zero, a model architecture and training procedure for learning an embodiment-aware control policy that can immediately adapt to new hardware changes without retraining. To do so, we present Graph Embodiment Transformer (GET), a transformer model that leverages the embodiment graph connectivity as a learned structural bias in the attention mechanism. We use behavior cloning to distill demonstration data from embodiment-specific expert policies into an embodiment-aware GET model that conditions on the hardware configuration of the robot to make control decisions. We conduct a case study on a dexterous in-hand object rotation task using different configurations of a four-fingered robot hand with joints removed and with link length extensions. Using the GET model along with a self-modeling loss enables GET-Zero to zero-shot generalize to unseen variation in graph structure and link length, yielding a 20% improvement over baseline methods. All code and qualitative video results are on https://get-zero-paper.github.io
EmbodiSteer: Steering Embodiment-Agnostic Visuomotor Policies with Joint-Space Guidance for Zero-Shot Cross-Embodiment Deployment
Department of Automation, Tsinghua University(清华大学自动化系);Beijing Key Laboratory of Embodied Intelligence Systems(北京具身智能系统重点实验室);Institute for Embodied Intelligence and Robotics, Tsinghua University(清华大学具身智能与机器人研究所)
Demo-JEPA: Joint-Embedding Predictive Architecture for One-shot Cross-Embodiment Imitation
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(信息处理国家重点实验室,计算机学院,北京大学);University of Washington(华盛顿大学);Beijing Innovation Center of Humanoid Robotics(北京人形机器人创新中心)
SkelWAM: A Skeleton-Guided World-Action Model for Zero-Shot Cross-Embodiment Manipulation
School of Advanced Manufacturing and Robotics, Peking University(北京大学先进制造与机器人学院);Feagine;IIIS, Tsinghua University(清华大学智能产业研究院);Research Center for Robotics, Peking University(北京大学机器人研究中心)
Trajectory-based Diffusion from One-shot Human Video for Generalizable Manipulation
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Zerith Robotics(智臻机器人);University of Science and Technology of China(中国科学技术大学)
ZETA: A Controlled Study of Zero-Shot Cross-Embodiment VLA Transfer for Tabletop Manipulation
Galbot(智元机器人);CFCS, School of CS, Peking University(北京大学计算机学院CFCS);Peking University(北京大学);Renmin University of China(中国人民大学);Xiamen University Malaysia(马来西亚厦门大学);The University of Hong Kong(香港大学);Beihang University(北京航空航天大学)
AdaTracker: Learning Adaptive In-Context Policy for Cross-Embodiment Active Visual Tracking
Beihang University(北京航空航天大学);City University of Macau(澳门城市大学);School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院);Minzu University of China(民族大学);Center for Data Science in Clinical Medicine, Peking University Third Hospital(北京大学第三医院数据科学临床医学中心);School of Computer Science, Peking University(北京大学计算机学院)
Ego-Dynamics-Augmented World Model for Autonomous Driving with Zero-Shot Cross-Embodiment Adaptation
School of Mechanical and Aerospace Engineering, Nanyang Technological University(南洋理工大学机械与宇航工程学院);Collaborative Initiative, Interdisciplinary Graduate Programme, Nanyang Technological University(南洋理工大学跨学科研究生项目合作计划);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院)
Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));DeepCybo;ZGCA;ZGCI;Harbin Institute of Technology(哈尔滨工业大学);Huazhong University of Science and Technology(华中科技大学);Beihang University(北京航空航天大学)
New York University(纽约大学);University of California, Berkeley(加州大学伯克利分校);University of California, Los Angeles(加州大学洛杉矶分校);Hello Robot Inc.(Hello Robot公司);Ai2;University of Waterloo(滑铁卢大学)
Towards Embodiment Scaling Laws in Robot Locomotion
University of California San Diego(加州大学圣地亚哥分校);Hillbot Inc(Hillbot公司);Cornell University(康奈尔大学);Technical University of Darmstadt(达姆施塔特技术大学);German Research Center for AI (DFKI)(德国人工智能研究中心(DFKI);德国机器人研究所;hessian.AI);Robotics Institute Germany;hessian.AI
MachaGrasp: Morphology-Aware Cross-Embodiment Dexterous Hand Articulation Generation for Grasping
Robotics & Autonomous Systems Division, Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR-I 2 R)(机器人与自主系统 division,信息通信研究所,科技研究局);College of Computing and Data Science, Nanyang Technological University(计算与数据科学学院,南洋理工大学);Show Lab, National University of Singapore(Show Lab,国立新加坡大学)
One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion
Department of Computer Science, Technical University of Darmstadt(德累斯顿技术大学计算机科学系);Institute of Robotics and Machine Intelligence, Poznan University of Technology(波兹南技术大学机器人与机器智能研究所);German Research Center for AI (DFKI), Research Department: Systems AI for Robot Learning(德国人工智能研究中心(DFKI)系统AI for机器人学习研究部);IDEAS NCBR, Warsaw, Poland(华沙波兰IDEAS NCBR);Centre for Cognitive Science(认知科学中心)