Enabling robots to autonomously perform hybrid motions in diverse environments can be beneficial for long-horizon tasks such as material handling, household chores, and work assistance. This requires extensive exploitation of intrinsic motion capabilities, extraction of affordances from rich environmental information, and planning of physical interaction behaviors. Despite recent progress has demonstrated impressive humanoid whole-body control abilities, they struggle to achieve versatility and adaptability for new tasks. In this work, we propose HYPERmotion, a framework that learns, selects and plans behaviors based on tasks in different scenarios. We combine reinforcement learning with whole-body optimization to generate motion for 38 actuated joints and create a motion library to store the learned skills. We apply the planning and reasoning features of the large language models (LLMs) to complex loco-manipulation tasks, constructing a hierarchical task graph that comprises a series of primitive behaviors to bridge lower-level execution with higher-level planning. By leveraging the interaction of distilled spatial geometry and 2D observation with a visual language model (VLM) to ground knowledge into a robotic morphology selector to choose appropriate actions in single- or dual-arm, legged or wheeled locomotion. Experiments in simulation and real-world show that learned motions can efficiently adapt to new tasks, demonstrating high autonomy from free-text commands in unstructured scenes. Videos and website: hy-motion.github.io/
Humanoid Agent via Embodied Chain-of-Action Reasoning with Multimodal Foundation Models for Zero-Shot Loco-Manipulation
Embodied AI and Robotics (AIR) Lab, New York University, New York, USA and NYUAD Center for Artificial Intelligence and Robotics, New York University Abu Dhabi, Abu Dhabi, UAE(纽约大学Embodied AI和机器人实验室及纽约大学阿布扎赫尔人工智能与机器人中心);Harvard AI and Robotics Lab, Harvard University, Boston, USA(哈佛大学人工智能与机器人实验室);Department of Computer Science, University College London, London, UK(伦敦大学学院计算机科学系);Department of Computer Science, University of Liverpool, UK(利物浦大学计算机科学系);NYUAD Center for Artificial Intelligence and Robotics, New York University Abu Dhabi, Abu Dhabi, UAE(纽约大学阿布扎赫尔人工智能与机器人中心)
RoboDuet: Learning a Cooperative Policy for Whole-body Legged Loco-Manipulation
Tsinghua University(清华大学);University of San Diego(圣地亚哥大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Shanghai Qi Zhi Institute(上海期智研究院);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);Jianghuai Advance Technology Center(江淮先进技术中心)
X2-N: A Transformable Wheel-legged Humanoid Robot with Dual-mode Locomotion and Manipulation
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology(香港理工大学电子与计算机工程系);X-lab Department, Agibot Innovation Company(阿吉博创新公司X实验室)
MetaWorld-X: Hierarchical World Modeling via VLM-Orchestrated Experts for Humanoid Loco-Manipulation
University of Hamburg(汉堡大学);School of Information Science and Technology, Beijing University of Technology(信息科学与技术学院,北京理工大学);School of Information Science and Engineering, Fudan University(信息科学与工程学院,复旦大学);University of Alberta(阿尔伯塔大学)
Learning Visual Quadrupedal Loco-Manipulation from Demonstrations
Shanghai Qi Zhi Institute(上海期智研究院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));University of California, Berkeley(加州大学伯克利分校);Tsinghua University(清华大学)
Learning Whole-Body Loco-Manipulation for Omni-Directional Task Space Pose Tracking with a Wheeled-Quadrupedal-Manipulator
School of System Design and Intelligent Manufacturing (SDIM), Southern University of Science and Technology, Shenzhen, China(系统设计与智能制造学院,南方科技大学,深圳,中国);Zhejiang University-University of Illinois Urbana-Champaign Institute (ZJUI), Haining, China(浙江大学-伊利诺伊大学厄巴纳-香槟分校联合学院,海宁,中国);LimX Dynamics, Shenzhen, China(LimX Dynamics,深圳,中国)
RAMBO: RL-Augmented Model-Based Whole-Body Control for Loco-Manipulation
Computational Robotics Lab in the Department of Computer Science, ETH Zurich(苏黎世联邦理工学院计算机科学系计算机器人实验室);Robotics Institute and the School of Computer Science at Carnegie Mellon University(卡内基梅隆大学机器人研究所和计算机科学学院)
ULC: A Unified and Fine-Grained Controller for Humanoid Loco-Manipulation
State Key Laboratory of Robotics and Systems, Harbin Institute of Technology(机器人与系统国家重点实验室,哈尔滨工业大学);School of Engineering, Westlake University(工程学院,西湖大学)
PILOT: A Perceptive Integrated Low-level Controller for Loco-manipulation over Unstructured Scenes
School of Automation and Intelligent Sensing(自动化与智能感知学院);Shanghai Jiao Tong University(上海交通大学);Global College(全球学院);school of Computer Science(计算机科学学院);Shanghai Key Laboratory of Navigation and Location Based Services(导航与位置基于服务上海市重点实验室)
PAKE: Learning Whole-Body Loco-Manipulation with Partial Kinematic Embeddings
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Korea Advanced Institute of Science and Technology(韩国科学技术院);ETH Zurich(苏黎世联邦理工学院)
Integrating Learning-Based Manipulation and Physics-Based Locomotion for Whole-Body Badminton Robot Control
School of Computer Science and Technology, Shandong University(山东大学计算机科学与技术学院);Robotics Institute, Carnegie Mellon University(卡内基梅隆大学机器人研究所);DeepCode Robotics(DeepCode机器人公司);Department of Sports, Shanghai Jiao Tong University(上海交通大学体育学院)
ViLoMan: Learning Visual-Proprioceptive Whole-Body Loco-Manipulation Skills for Humanoid Robots
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS(中国科学院计算技术研究所人工智能安全国家重点实验室);University of Chinese Academy of Sciences (CAS)(中国科学院大学);Beijing Academy of Artificial Intelligence (BAAI)(北京智源人工智能研究院)